protoDC2 phoSim operations log

This issue will be a log/diary the operational progress of the protoDC2 phoSim image generation at NERSC.  This issue is not intended to be a venue for discussing phoSim configuration (see, for example,  #19 or #33) or results.
A few technical details about the workflow itself can be found here.

Production has started, although it is not unlikely a problem will be discovered to cause a halt and restart from the beginning.  A last minute decision: disable the phosim amplifier file output due to problems with those files.

The NERSC queues are not behaving nicely.  A single-node 24-hour job submitted on Monday morning is now scheduled to run tomorrow at the earliest.  To jump-start the process, a 10-node 8-hour KNL Pilot was submitted this afternoon and, lo!, it has started.  There are now >300 raytrace processes running.
A question arose earlier about how long it might take to complete the ~8000 visits in protoDC2.  Given that changes were made in the phoSim command file just yesterday, we won't have good data until sufficient sensor-visits have completed.  Hopefully tomorrow...
We're off and running!

Thursday 21 Dec Update
Some success over the past 12 hours: 1865 sensor-visits completed (mostly on KNL) using the latest phoSim command file (ref #63).  Based on these runtime performance statistics, a first estimate of total protoDC2 resource and time consumption may be made.

The mean wall-clock time for the KNL raytrace step is observed to be ~270 minutes (with a tail extending to 470 min).
The first 100 visits (r-filter) spawned a total of 7523 sensor-visits, or 75 sensors/visit on average (40% of the focal plane)
"protoDC2" contains  8080 visits per the following table.

band
#visits

u
534

g
776

r
1782

i
1795

z
1612

y
1581

Total
8080

Using average values:

8080 visits * 75 sensors/visit = 606,000 sensor-visits to simulate

Raytrace is run with 8-threads/instance, and 34 instances/KNL node, completely filling the available 272 hardware threads

Total  KNL node-hours = 606,000 sensor-visits * (250 min/sensor-visit) * (1 hr/60 m) * (8 threads) / (272 threads/node) = 74265 node-hours = 3094 node-days

Total NERSC-hours = 74265 node-hours * (96 NERSC-hours/node-hour) = 7.1M NERSC-hours (this assumes no wasted processing -- which will not be the case)

This amount of processing could, under unrealistically ideal conditions, be performed in less than one week using ~1000 nodes.  The trick will be keeping jobs running efficiently.  The main challenges include:

Getting SLURM jobs to run

There is some built-in inefficiency due to jobs terminating when the SLURM job times out (they must then be rerun from the beginning)

SLURM queue dwell times rise quickly with requested job length, thus making shorter jobs preferable from a dispatch perspective, but exacerbating the terminating job inefficiency

The instanceCatalog generation is delicate and must be done so as to not overload the UW server, but performed sufficiently quickly to keep the processing pipeline happy

The SLAC holiday power outage (26th-30th Dec) will impact production

I will be traveling and unable to focus 100% on this project over the coming 2 weeks.

Monitoring the workflow
The main r-filter workflow monitor is here.
An experimental Pilot job monitor is here.

Saturday 23 Dec 2017 Update:
As of 07:30 PST, over 19,000 sensor-visits have been completed (representing ~3% of the anticipated 606k total sensor-visits in the current visit lists).  The challenge has been keeping SLURM jobs running.  Short jobs (8-10 hours) seem to start up within a few hours, but they suffer from serious inefficiency when they end -- taking many partially run raytrace jobs with them.  Long jobs (24 hours) take many days in queue before they begin running.
When a block of cori-KNL nodes do begin to run, this is often accompanied by a set of failed raytrace jobs -- which fail when attempting to access one of the phoSim site or instrument files.  These jobs fail almost immediately (hence, a small impact on efficiency) and can be easily rolled back.  My guess is that the shock of starting hundreds of jobs simultaneously is putting a strain on the connection to the file system.  An annoyance, but not (yet) serious.
The plan going forward will be to attempt keeping a mix of short-running and long-running jobs in the cori queues in the hopes of improving overall utilization.  A plot of KNL usage vs. time is beginning to take form here:  https://portal.nersc.gov/project/lsst/glanzman/graph3.html

@TomGlanzman Starting hundreds of jobs should not be a "-- strain on the connection to the files system." In principle, hundreds of jobs is nothing to worry about -- you should file a ticket with NERSC about this. Something is not working correctly.

Friday 16 Feb 2018 Update:
Production for Run 1.1 (phosim) is imminent.  Initial test of revamped instanceCatalog generation, dynamic SEDs, updated visit lists, and other config changes and bug fixes has been completed for a single visit in the r-band (WFD).  Data products for this test are here:
/global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-3_WFD-r/output/000001

Note that this test is NOT the final configuration as a final tag of DC2_repo, and agreement on the phoSim "--fov" parameter is needed -- so these data will be overwritten with production data in the near future.
Run 1.1 consists of the the following visits.

Band
Survey
#Visits

u
WFD
67

g
WFD
91

r
WFD
245

i
WFD
223

z
WFD
247

y
WFD
252

u
uDDF
192

g
uDDF
138

r
uDDF
138

i
uDDF
137

z
uDDF
136

y
uDDF
135

-
TOTAL
2001

A new DC2 phoSim test visit has been completed, obsHistID=181866 with 63 sensors simulated.  Data products are here: /global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-3_WFD-r/output/000002 .  The biggest change since yesterday's test was the change in the --fov parameter from 0.5 to 2.1 (degrees) during instanceCatalog generation.  Please have a look.

Monday 19 Feb 2018 Update:
Ducks are lined up, so begin a production for Run 1.1, starting with r-band, WFD.

Tuesday 20 Feb 2018 Update:
Production is ramping up.  All 245 visits for r-band WFD have been submitted.  Twenty-five for each of the remaining 11 configurations have also been submitted.
There have been a few failures: 1) the expected failures when a Pilot terminates (normally, after it times-out); and, 2) the familiar issue of phoSim reading one of its many files during initialization.  This latter issue arises when many phoSim instances attempt to start (near) simultaneously.  A simple rollback solves the problem.
At 20:50 today, 425 sensor-visits have completed and 1904 are running.  I expect the number of running to increase as new KNL Pilot jobs start to run.
Here is a link to view all 12 workflows on one page.

For what is worth, on Fermi side Stephan Zimmer had cooked a trickleStream python cron job that (IIRC) sends automatically a controled flow of streams until it reaches allocation, and then goes on to stay at max allocation as much as possible. This might be handy in your case, if trickling leaves enough time discrepancy between starting jobs to avoid the multi-read issue.

(actually, if you need people with rollback permissions on the CET time zone to speed things up, we are available..... but of course we have no clue about the phosim slac/nersc pipeline, so it is only about pushing a green button while you are asleep :) )

Thanks @johannct, I actually provided tricklestream to Stephan some years ago so am quite familiar with its use.  In this case, we do not yet know enough to use this tool effectively -- we are still in the process of discovering the limits of the instanceCatalog generation step.  Also, given that DC2-phoSim consists of 12 independent pipeline tasks, trickleStream will need some work (and thought) to make it useful.

Wednesday 21 Feb 2018 Update:
Production continues to ramp up.  At the moment (09:00) there are 27 fully completed visits and 4128 completed sensor-visits (which includes both fully and partially completed visits).  1561 sensors are being simulated on 46 KNL nodes (about 34 phoSim instances/node).  Another 200 nodes have been requested but are not yet running.
A couple of operation issues arose in the last 12 hours.
#135 solved and closed
#136 open

oops I got the order of authorship wrong then! :)

Thursday 22 Feb 2018 Update:
As of 09:00 there are 66 fully completed visits (3% of Run 1.1's 2001 visits) with many more in progress.  Cori utilization has been rather low the past 24 hours: a max of 46 nodes dropping down to a mere 8 for several hours this morning.  About 1000 nodes have been requested, so we simply must wait for SLURM to let them run.
Good news is that both issues opened yesterday have been solved and closed.  In addition to the expected job terminations due to Pilot time-outs, a new failure is starting to appear: segmentation faults in phoSim's atmosphere creator, which seems correlated with many simultaneous instances (in the 30-50 range).  This problem acts as a transient, generally succeeding upon rollback, but bears watching.
Handy links (note some are updated only every few minutes):

Workflow summary page for all 12 configurations
Brief Pilot job summary
SLURM queue for waiting Pilot jobs
Plot of cori utilization since start (19 Feb 2018)

Friday 23 Feb 2018 Update:
As of 10:30 today, there are 76 fully completed visits representing 4% of total.  Production has ground to a virtual halt as the small set of Pilot jobs that started running 2 days ago have all completed.  Submitted Pilot jobs are just not running!  There are jobs submitted on the 19th and are still waiting for a chance to run.  SLURM indicates that up to 170 may come available in ~9.4 hours, but those estimates are largely unreliable.  Perhaps there will be a surge of activity starting this evening.  In the meantime, I am using various tricks to prepare the maximum number of waiting 'raytrace' steps as possible.

Thanks @TomGlanzman ! Sorry to hear we are not yet up to LSST data rates - but then I expect commissioning might be a bit like this too ;-)
While we are waiting for the conditions to improve, would you mind breaking down that 4% by band and survey please? Here's an extended version of the table you made further up the thread, is it easy for you to fill in (or autom-magically over-write) the table below? You can edit this message no problem, or just paste in your own version in a subsequent comment. I'm interested to see what kind of DM processing we could be doing. Thanks!

Sunday 25 Feb 2018 Update
More of the same (see yesterday's report).  The priorities at NERSC seem completely given over to huge MPI jobs - at the expense of the type we need for DC2-phoSim production... :(

We need to bug someone at NERSC; the queueing system was always something I had complained about. I will send an email to Richard Gerber.

Monday 26 Feb 2018 Update
The first half of the day was a repeat of yesterday -- no activity.  Then at 5 minutes 'til noon, the first large SLURM jobs began to run.  At first, two jobs (each with 20 nodes), followed by another job of 20 nodes.  With these 60 nodes, about 2000 single sensor-visits (raytrace) jobs are running.
The task summary page has been updated and is now a bit easier to read.  The tables indicate the overall progress of the DC2-phoSim Run 1.1.

The first table sums up the overall number of workflow batch jobs.

The second table breaks this down according to the process step (setupVisit = instanceCatalog generation and phoSim initialization; RunTrim is the phoSim catalog trimming step; RunRayTrace is the time-consuming ray tracing for a single sensor for a single visit; finishVisit runs when all sensors for a particular visit have completed).  Thus, Run 1.1 will be complete when there are 2001 successful 'finishVisit' steps.

The final set of tables gives the summary for each workflow task == each survey configuration, e.g., WFD r-band.

I sent an email to Richard Gerber yesterday -- let's see what he says.

Ok, Richard got back to me. He will get someone to talk to Tom; what they want is an estimate of the required throughput from us. You can provide that, right, Tom?

Tuesday 27 Feb 2018 Update

07:30   Finally!  After a week's wait, a significant amount of Cori resources are beginning to come online for DC2-phoSim.  At this time, 8  Pilot jobs, on  190  hosts ( 12160  cores) are running, which support ~6400 raytrace instances (sensor-visits).  There are currently 183 fully complete visits, or about 9% of total.

Wednesday 28 Feb 2018 Update

17:30 Today was a pretty good day, starting off with a continuation of 370 KNL nodes.  At the current time slightly more than 450 visits have been completed (~22% of total for Run 1.1), while many more are under way.  There is a considerable backlog of SLURM/Pilot jobs, some have been in queue for 7 days, so one hopes new jobs will start soon.  NOTE: due to Pilots running out of time, there will be a large burst of failed jobs -- that is to be expected; they will be rolled back and restarted.

An interesting conversation with a couple of NERSC folks this afternoon.  One reason for the poor SLURM response this past weekend was due to a massive 9,000 node MPI job that ran for several days.  (Cori-KNL has a total of about 9,800 nodes, so this had a serious impact on the rest of us.)  Other than that, they reaffirmed that the only strategies for better throughput they can offer include:

Limit [Pilot] jobs to run fewer hours (DC2 jobs request the maximum of 48 hours to be efficient)
Make a reservation (this requires one to specify an exact # nodes and run time, plus up to several days for nodes to drain to make room...if we do not see some job action by tomorrow mid-day, I may request a reservation, although this raises operational issues)

Neither of these two strategies is a silver bullet and will end up costing our allocation.
I also mentioned the lack of accuracy in the SLURM job dispatch estimate (in response to the 'sqs' command, for example), the burstiness of SLURM dispatch, the inability to execute a controlled ramp-up of production, and inability to control the rate at which jobs start.  Maybe a fresh look will help solve some of these problems.

Thursday 1 Mar 2018 Update

11:15 There are currently 548 visits complete (27% of total).  However, the number of sensor-visits complete is probably more like ~43% complete.  We are currently running with a mere 200 nodes.

While the number of Pilot jobs is decreasing after running out the 48-hour clock, SLURM has not been kind to us in replenishing this effort.  There remain dozens of jobs being held hostage in queue, some for over one week.  Perhaps Cori is preparing of another mega-job and not allowing our 48-hour jobs to backfill...  The question now is whether to request a 'reservation' - the request for which would take several days to process?  From today's standpoint, something like 300-400 nodes for 2-3 days would make a huge dent in the remaining work to be done.  But one cannot predict whether or how many of the existing queued jobs may start to run before a reservation could be put in place.

Friday 2 Mar 2018 Update
There are 550 complete visits, basically were we were yesterday.  NERSC has ramped us down to a mere 40 nodes.  There are 41 Pilots (880 nodes) waiting -- for up to 9 days -- to start running.  So, while essentially idle, we wait...........

But issues with phoSim have resulted in two actions being taken this morning: 1) hold all pending Pilot jobs; and, 2) cancel the 50-node/3-day 'reservation' due to start tomorrow morning.  It is unclear whether Run 1.1 will continue, but there may be a Run 1.2...

Monday 5 Mar 2018 Update
Run 1.1, although incomplete, is winding down.  713 visits are complete (36%).
Preparations for phoSim testing is underway in #140

Monday 2 Apr 2018 Update
Testing to resume production has gotten underway.  With new phoSim background parameters and updated catalog generation, this new project is dubbed "Run 1.2p".  A single visit with a non-production catalog generation is running here.
A NERSC 'reservation' has been requested (100 KNL nodes for 24-hours) to help jump-start this project.  If granted, this reservation will begin sometime Thursday, 5 Apr 2018.  Hopefully all production code will be in place by that time.

Friday 6 Apr 2018 Update
DC2 Run 1.2p began yesterday evening!  Five single-node 'premium' class SLURM Pilot jobs were submitted to get the ball rolling.
A fortuitous decision to "hold" rather than "cancel" a batch of SLURM jobs from Run 1.1p in February has lead to the discovery that once released, these jobs start very quickly.  There are 31 such "held" jobs and they collectively represent 680 nodes each for 48 hours.  Eleven of these 20-node jobs have been released and are now running.  A good way to start the production!
11:00 status

16 SLURM Pilot jobs running on 225 hosts (14,400 cores)
115 sensor-visits complete
7350 jobs running

Saturday 7 Apr 2018 Update
Thanks to those old SLURM jobs, production continues to run at a good pace.  Fresh jobs submitted yesterday are slowly moving up in the queue.  We have a 100-node 24-hour reservation coming up Monday morning at 10am.  With all of these factors, production may smoothly continue even as the old jobs terminate.
13:00 status:

48 visits (of 2001) fully complete (2.4%)
10,658 sensor-visits (raytrace) jobs active on 320 nodes (20,480 cores)
All "WFD" DC2 visits submitted (and about 10% of the uDDF visits submitted)
Ramp-up continues...

Monday 9 Apr 2018 Update
A productive weekend.  After a reasonably stable weekend, will begin again to ramp up the number of nodes.  A 100-node 24-hour reservation begins today at 10am.
08:00 status:

99 visits complete (5% of 2001)
18,794 complete sensor-visits (this will probably turn out to be ~10% of the total)
12,795 running sensor-visits (raytrace) on 280 nodes

Tuesday 10 Apr 2018 Update
A good 24 hours of normal running, increasing the number of nodes to 760.
07:50 status:

182 visits fully complete (9%)
28,638 sensor-visits complete
25,760 instances of raytrace running on 760 nodes
A number of sensor-visits requiring >24 hours have begun to appear.  In the WFD r-band, there are currently 54 sensor-visits (out of 13,352) so far.  My plan is to wait until the end of Run 1.2p, then request a 4-day reservation to take care of the outliers.

#Tuesday 11 Apr 2018 Update
The ramp-up of Run 1.2p has stalled and even lost momentum.  This is because NERSC is not running jobs that have now been in queue for over five (5) days.  At 09:45 I released the final set of jobs submitted in February (on the 22nd), so will see if those 3 jobs of 20 nodes each start running soon.
09:45 status:

231 visits fully complete (11.5% of 2001)
34,478 sensor-visits complete (about 22% of total)
1,658 sensor-visits have timed out after 48 hours of running
13,598 sensor-visits (raytrace) jobs currently running on 400 nodes

Wednesday 12 Apr 2018 Update
As of this morning the pipeline has completely run dry: there are zero jobs running at present.  Despite having >3000 nodes of batch jobs in the queues, some of which have been waiting for nearly a week, the NERSC scheduler is not providing computing resources, so production has ground to a full stop.
07:45 status:

245 visits fully complete (12% of 2001)
37,842 sensor-visits (raytrace) complete (~ 24% of total)
0 jobs running

Friday 13 Apr 2018 Update
After a 2-day ramp-down, a short (6-hour) outage and the false threat of a lengthier (27-hour) outage, production quickly ramped back up to just under 500 nodes where it has remained.
07:40 status:

245 visits fully complete (12% of 2001)
38895 sensor-visits complete (24%)
14,030 jobs running on 480 nodes

Note that ~4M NERSC hours have been spent on Run 1.2p thus far and it is nearly 25% complete.

Saturday 14 Apr 2018 Update
Something bad happened just after midnight this morning which caused 600 nodes to crash (at least that what the clues point to).  Further, whatever caused this to happen continued to plague new jobs all the way up to 08:00 this morning.  The end result is that >81,000 jobs were abnormally terminated.  It is a technical challenge simply to roll back such a large number of jobs :(
13:40 status:

255 visits fully complete (13% of 2001)
41,481 sensor-visits complete (26% of total)
16,997 jobs currently running on 500 nodes
~80,000 sensor-visit jobs to roll back

Sunday 15 April 2018 Update
Recovery from yesterday's mishap and, perhaps surprisingly, continued ramp up.
13:50 status:

315 visits fully complete (16%)
45,837 sensor-visits complete (29%)
30,934  sensor-visit jobs currently running on 910 hosts (!!)

Monday 16 Apr 2018 Update
A surprisingly good weekend -- after recovering from the Saturday morning meltdown.
07:45 status:

401 visits fully complete (20% of total)
54672 sensor-visits complete (34% of total)
35,612 sensor-visits currently running on 1060 nodes
~7800 sensor-visits timed-out (>48 hours)

Tuesday 17 Apr 2018 Update
Steady running the past 24 hours.  A large block of SLURM jobs timed out yesterday morning after 48 hours dropping production from over 1,000 to only ~320 nodes, but since then have ramped back up to >600 nodes.
07:45 status:

426 visits fully complete (21% of project total)
60,834 sensor-visits complete (38% of project total)
12,227 sensor-visits timed out after 48 hours (17% of raytrace attempts to date)
21,078 sensor-visits currently running on 620 nodes

Wednesday 18 Apr 2018 Update
Not a great 24 hours, but not due lack of resources.  Some hiccups due to script changes.  A recommended change to query machine memory unexpectedly caused a bash error which in turn caused a block of failures until the change was pulled out.    Yesterday's "maintenance day" appears to have been successful.  Cori memory is now routinely recorded and the transient read error of 'focalplanelayout.txt' will now be automatically retried.
09:45 status:

446 visits fully complete (22%)
63539 sensor-visits complete (40%)
18330 sensor-visits timed-out after 48 hours
13938 jobs running on 410 nodes
22,476,954 NERSC hours charged AY 2018

Friday 20 Apr 2018 Update
A smooth 24 hours, but a problem associated with the SLAC workflow engine has appeared: a large number of jobs (~30k) are stuck in a "QUEUED" state and are not being released for processing.  Experts notified.
13:40 status:

600 visits fully complete (30%)
77162 sensor-visits complete (49%)
18346 sensor-visits timed-out after 48 hours
23092 jobs running on 1010 nodes
23,840,497 NERSC hours charged AY 2018

Sunday 22 Apr 2018 Update
As expected, a large number of Pilot jobs timed out Saturday night, representing the large block of Cori nodes obtained Thursday evening which ran for 48 hours.  Interestingly, a large chunk of the terminated jobs are long-running, meaning they cannot be rolled back as they will simply run out of time yet again.  So they are being added to the ever growing collection that must be dealt with using a NERSC "reservation", once the shorter jobs have completed.  Given the fact that NERSC desires a 4-day lead time (or more) for a reservation and that the supply of shorter jobs is dwindling, I will be making a reservation request tomorrow after consulting with Wahid, our current NERSC contact.
09:25 status:

771 visits fully complete (39%)
97434 sensor-visits complete (61%)
31711 long-running sensor visits (>48 hours), est.
30,545,595 NERSC hours charged AY 2018

Yesterday, our NERSC usage passed a milestone of sorts: our usage now exceeds the "uniform usage" projection, i.e., the amount of allocation used if consumption distributed evenly over the allocation year. (Note that Run 1.2p began on 5 Apr 2018.)

Monday 23 Apr 2018 Update
NERSC is performing maintenance on the 'projecta' group disk storage system today from 0800-1400.  Because this space is explicitly called out as a requirement for all phoSim jobs, this explains why no new Pilots jobs were dispatched since Saturday morning.  As a result of this outage, Run 1.2p is temporarily halted.
NOTE: There are no more 'pending' sensor-visits; all 158,766 sensor-visits have now been attempted at least once.  There remains a large number of 'terminated' sensor-visits, some of which (>30k jobs) ran for >48 hours and the remainder terminated when their Pilot jobs ran out of time in the SLURM queue.
08:20 status:

850 visits fully complete (42%)
99916 sensor-visits complete (63%)
58766 terminated sensor-visits
30,781,410 billable NERSC hours used in AY 2018

Hi Tom.  Great; can you help me parse one thing?
Since you say that all visits have now been attempted and 63% are complete, that means that ~40% of the sensor visits needed to be repeated.  What fraction of the 30M hours were "wasted" in this way?  I assume it is a smaller number since many jobs never really started etc.  Do we know that that "CPU efficiency" number actually is?

Tuesday 24 Apr 2018 Update
After yesterday's storage outage, 500 nodes started up and they were able to accommodate all 15,590 remaining rolled back jobs (i.e., jobs that terminated due to Pilot time-out, but had not yet run for 48 hours).  The scale of the "extra long" jobs is now known:  ~44,000 (or about 28% of the total number of sensor-visits).  I have just started negotiating with Wahid at NERSC for a >48 hour Cori-KNL reservation to handle these jobs.
07:55 status:

888 visits fully complete (44%)
100042 sensor visits complete (63%)
43134 terminated visits (jobs that require >48 hours to complete)
15590 jobs currently running on 500 nodes
31,285,400 billable NERSC hours used in AY 2018

Thursday 26 Apr 2018 Update
DC2 Run 1.2p has reached a pause.  All sensor-visits that can complete within 48 hours (maximum queue time at NERSC) have completed.  The remaining sensor-visits will be attempted during a special 7-day 'reservation' which is scheduled to commence next Tuesday, 1 May 2018 at 10:00 Pacific.  With luck (depending on how long these sensor-visits actually take to complete), we could possibly complete Run 1.2p during this reservation.  If not, we will need to schedule a follow on mop-up reservation.
Here are the numbers:

1000 visits fully complete (50%)
104786 sensor-visits complete (66%)
53980 long-running sensor-visits that must be rolled back (34%)
There are zero jobs currently running
33,084,164 NERSC hours billed (33% of AY 2018)
11.3M NERSC hours to be consumed by next week's reservation

Next news update Tuesday, 1 May 2018.

Snapshot of the "blue" DC2-Run 1.2p status page, where statistics on the long-running sensor-visits may be found (timed-out jobs are tallied in the "TERMINATED" column):
   2018-04-26 08:30:03.770866  Pacific
  (Note: jobs in the "QUEUED" state are not tallied, and the "ALL" column is currently unreliable)

=======Task and (batch) Process-step Rollup==============================
+-----------+-----------+-----------+-----------+----------+--------------+-------+
|   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------+-----------+-----------+-----------+----------+--------------+-------|
|         0 |         0 |         0 |    151809 |        0 |        53980 |     0 |
+-----------+-----------+-----------+-----------+----------+--------------+-------+

=======Task Rollup===============================================
+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| setupVisit    |         0 |         0 |         0 |      2001 |        0 |            0 |     0 |
| RunTrim       |         0 |         0 |         0 |     44022 |        0 |            0 |     0 |
| RunRaytrace   |         0 |         0 |         0 |    104786 |        0 |        53980 |     0 |
| finishVisit   |         0 |         0 |         0 |      1000 |        0 |            0 |     0 |
+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+

======================================================

Total rolled back process steps =  249645

=======Detailed Task Summary======================================
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-u | setupVisit    |         0 |         0 |         0 |        67 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-u | RunTrim       |         0 |         0 |         0 |      1474 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-u | RunRaytrace   |         0 |         0 |         0 |      4473 |        0 |          348 |     0 |
| DC2-R1-2p-WFD-u | finishVisit   |         0 |         0 |         0 |        58 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-g | setupVisit    |         0 |         0 |         0 |        91 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | RunTrim       |         0 |         0 |         0 |      2002 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | RunRaytrace   |         0 |         0 |         0 |      6118 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | finishVisit   |         0 |         0 |         0 |        91 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-r | setupVisit    |         0 |         0 |         0 |       245 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-r | RunTrim       |         0 |         0 |         0 |      5390 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-r | RunRaytrace   |         0 |         0 |         0 |     15912 |        0 |         2562 |     0 |
| DC2-R1-2p-WFD-r | finishVisit   |         0 |         0 |         0 |       185 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-i | setupVisit    |         0 |         0 |         0 |       223 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-i | RunTrim       |         0 |         0 |         0 |      4906 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-i | RunRaytrace   |         0 |         0 |         0 |      9469 |        0 |         6774 |     0 |
| DC2-R1-2p-WFD-i | finishVisit   |         0 |         0 |         0 |       102 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-z | setupVisit    |         0 |         0 |         0 |       247 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-z | RunTrim       |         0 |         0 |         0 |      5434 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-z | RunRaytrace   |         0 |         0 |         0 |      7880 |        0 |        10219 |     0 |
| DC2-R1-2p-WFD-z | finishVisit   |         0 |         0 |         0 |        82 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-y | setupVisit    |         0 |         0 |         0 |       252 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-y | RunTrim       |         0 |         0 |         0 |      5544 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-y | RunRaytrace   |         0 |         0 |         0 |      2122 |        0 |        15903 |     0 |
| DC2-R1-2p-WFD-y | finishVisit   |         0 |         0 |         0 |        17 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-u | setupVisit    |         0 |         0 |         0 |       192 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-u | RunTrim       |         0 |         0 |         0 |      4224 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-u | RunRaytrace   |         0 |         0 |         0 |     16593 |        0 |          270 |     0 |
| DC2-R1-2p-uDDF-u | finishVisit   |         0 |         0 |         0 |       154 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-g | setupVisit    |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-g | RunTrim       |         0 |         0 |         0 |      3036 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-g | RunRaytrace   |         0 |         0 |         0 |      7901 |        0 |         4234 |     0 |
| DC2-R1-2p-uDDF-g | finishVisit   |         0 |         0 |         0 |        72 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-r | setupVisit    |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-r | RunTrim       |         0 |         0 |         0 |      3036 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-r | RunRaytrace   |         0 |         0 |         0 |      8227 |        0 |         3902 |     0 |
| DC2-R1-2p-uDDF-r | finishVisit   |         0 |         0 |         0 |        50 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-i | setupVisit    |         0 |         0 |         0 |       137 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-i | RunTrim       |         0 |         0 |         0 |      3014 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-i | RunRaytrace   |         0 |         0 |         0 |      8996 |        0 |         3047 |     0 |
| DC2-R1-2p-uDDF-i | finishVisit   |         0 |         0 |         0 |        63 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-z | setupVisit    |         0 |         0 |         0 |       136 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-z | RunTrim       |         0 |         0 |         0 |      2992 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-z | RunRaytrace   |         0 |         0 |         0 |      8896 |        0 |         3034 |     0 |
| DC2-R1-2p-uDDF-z | finishVisit   |         0 |         0 |         0 |        55 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-y | setupVisit    |         0 |         0 |         0 |       135 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-y | RunTrim       |         0 |         0 |         0 |      2970 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-y | RunRaytrace   |         0 |         0 |         0 |      8199 |        0 |         3687 |     0 |
| DC2-R1-2p-uDDF-y | finishVisit   |         0 |         0 |         0 |        71 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+

Elapsed time for workflow database queries =  25.947135  sec

A random data point.  Looking at a single (random) WFD y-band run that consumed ~41 hours clock time, here is the roll-up by photon source category of: number of photons and elapsed time to simulate them.

Type
Tot photons
Frac phot
Tot time(s)
$ time

Astro Objects
655,711,741
2.66E-04
3443.15
2.34%

Zodiacal
71,261,379,555
0.03
15020.15
10.21%

Moon-Mie
422,991,262,780
0.17
16034.5
10.90%

Moon-Rayleigh
1,342,127,786,118
0.54
28192.24
19.16%

Airglow
603,463,483,218
0.24
78302.19
53.22%

Cosmic Rays
23,524,903,800
0.01
6142.38
4.17%

Total
2,464,024,527,212
1.00
147134.61
100%

Tuesday 1 May 2018 Update:
A ~7-day, 720-node reservation on Cori-KNL begins today.  During this reservation, all remaining sensor-visits will be attempted.
 $ scontrol show res desc-run1.2p
ReservationName=desc-run1.2p
StartTime=2018-05-01T10:00:00
EndTime=2018-05-08T06:59:00
Duration=6-20:59:00
Nodes=nid[02688-02703,02708-02763,02768-02769,02771-02827,02836-02891,02896-02924,02926-02959,02964-03002,03004-03011,03013-03019,03024-03056,10743,12288-12456,12458-12599,12601-12671]
NodeCnt=720
CoreCnt=48960
Features=knl
PartitionName=regular
Flags=
TRES=cpu=195840
Users=glanzman,descpho,desc,descdm
Accounts=m1727
Licenses=(null)
State=INACTIVE
BurstBuffer=(null)
Watts=n/a

Thursday 3 May 2018 Update:
Jobs on all 720 nodes were running smoothly until this morning when a cluster of terminated jobs suddenly appeared.  After some investigation, the only clue from the Pilot is that 'srun' claimed an out-of-memory problem.  (Note, I have never seen an indication of a memory leak with phoSim and its memory foot print is very stable.)  This problem affected exactly 10 nodes, or 671 raytrace jobs (well, as of 11:30 PDT).  I have modified the Pilot parameters to allow slightly less memory to be allocated to see if that has any effect, and then resubmitted 10 1-node Pilot jobs.
11:40 status:

720 Pilots running on 720 nodes
23323 raytrace jobs running
101267 raytrace jobs complete
958 visits fully complete
approx 114 hours remaining in reservation

Friday 4 May 2018 Update:
The long-running jobs continue to make progress and we are now clearly seeing positive results of running within a 7-day reservation.
The problem of Pilots crashing is now suspected to be memory exhaustion on the KNL nodes.  Ongoing discussion as to whether this is memory consumption by user code (workflow Pilot and/or phoSim raytrace), or a system process, or some other problem on the afflicted node.  Some memory monitoring has been installed in the workflow.  To date, approx. 6% of the Pilot jobs have crashed.
A second problem arose when the 'descpho' pseudo-account's sub-allocation became exhausted, preventing any new jobs from being submitted.  Thanks to Richard for quickly fixing!
13:55 status:

1006 visits fully complete (50%)
117499 sensor-visits complete (74%)
24168 sensor-visits running on 720 nodes
88.85 hours remaining in current reservation
37376902 billable NERSC hours (through 5/3/2018), 37% of AY 2018

Sunday 6 May 2018 Update:
Smooth running since Friday.  We are now slightly less than two days away from completion of the 7-day reservation and it is possible to make a prediction.  Of the >60,000 sensor-visits in queue last Tuesday morning, approx 3900 have not yet started to run, therefore it is highly unlikely they will complete before  the upcoming Tuesday cut-off.  This represents a lower limit to the estimate of uncompleted jobs, as some fraction of the currently running jobs may not complete.
07:10 status:

1153 visits fully complete (58%)
130581 sensor-visits complete (82%)
32457 sensor-visits completed so far during the reservation
24335 sensor-visits currently running on 720 nodes
47.5 hours remaining in current reservation
40,694,883 billable NERSC hours used in AY 2018

Monday 7 May 2018 Update:
The 7-day/700-node reservation is beginning to wind down in preparation for tomorrow morning's completion.  A summary of the entire reservation along with a post mortem of the three primary hiccups encountered over the past week will be reported in this log tomorrow.
09:00 update:

1227 visits fully completed (61% of total)
140661 sensor-visits complete (89% of total)
42537 sensor-visits completed so far during this reservation (70% of remaining long-running jobs)
18084 jobs currently running on 714 nodes
21 hours remaining in reservation
42353743 billable NERSC hours used through Sunday

Tuesday 8 May 2018 Update:
The 7-day/700-node reservation ended this morning.
(This log entry will be updated later today.)
07:30 status:

1363 visits fully complete (68% of total)
148878 sensor visits complete (94% of total)
0 jobs running (NERSC maintenance day)
9888 sensor visits remaining
43,013,029 billable NERSC hours used in AY 2018

Tuesday 15 May 2018 Update:
Recent work by Adrian Pope and Tom Uram at Argonne motivated the release of phoSim v3.7.14, which has improved thread scaling.  Recall that DC2 production -- so far -- has been limited to 8-threads in the raytrace step due to internal code inefficiencies when using more threads [ref work last Fall by Glenn and myself].  As a performance and validation check, a single DC2 Run 1.2p visit has been run with the latest phoSim code.  This note summarizes the tests.
The executive summary is that the code changes do, indeed, allow for efficient running of many threads, thereby enabling us to attempt recovering the ~10,000 remaining sensor-visits in DC2 Run 1.2p.  There are some differences between the 8-thread (v3.7.9) and 54-thread (v3.7.14) images.  Although these differences do seem not qualitatively significant, it would be good if others could take a more quantitative look.

Performance
The phoSim v3.7.14 code was built at NERSC for KNL using the Intel compiler, exactly as was 3.7.9.  The phoSim command files and the instanceCatalog generation were identical with earlier production running.  The selected test visit was 7th in sequence for the WFD y-band survey.  This particular visit contains 26 sensor-visits, of which nine (9) were terminated due to running out of time. Whereas the original DC2 run was configured such that each KNL node would run 34 instances of 8-thread raytrace, the v3.7.14 test was configured to run 5 instances of 54-thread raytrace.  The visit (obsHistID) is 12449.  The following table shows the elapsed clock time for the new and old runs (in hours), followed by the final status of the job step ("TERMINATED" means the Pilot job ran out of time).  Finally, the ratio of elapsed times is presented.
Stream                        3.7.14                  v3.7.9            v3.7.9/v3.7.14
6.13.0   elapsedHours: test=  4.49 SUCCESS   ,  prod= 56.41 SUCCESS   ,  ratio= 12.56
6.13.1   elapsedHours: test=  4.87 SUCCESS   ,  prod= 60.72 SUCCESS   ,  ratio= 12.47
6.13.2   elapsedHours: test=  5.20 SUCCESS   ,  prod= 61.79 SUCCESS   ,  ratio= 11.89
6.14.0   elapsedHours: test=  6.87 SUCCESS   ,  prod= 64.34 SUCCESS   ,  ratio=  9.37
6.18.0   elapsedHours: test=  7.04 SUCCESS   ,  prod= 46.79 TERMINATED,  ratio=  6.65
6.18.1   elapsedHours: test=  7.73 SUCCESS   ,  prod= 64.88 SUCCESS   ,  ratio=  8.39
6.18.2   elapsedHours: test=  6.71 SUCCESS   ,  prod= 58.37 SUCCESS   ,  ratio=  8.70
6.18.3   elapsedHours: test=  6.62 SUCCESS   ,  prod= 39.50 TERMINATED,  ratio=  5.97
6.18.4   elapsedHours: test= 11.52 SUCCESS   ,  prod= 62.51 SUCCESS   ,  ratio=  5.43
6.18.5   elapsedHours: test=  6.79 SUCCESS   ,  prod= 65.69 SUCCESS   ,  ratio=  9.68
6.18.6   elapsedHours: test= 11.46 SUCCESS   ,  prod= 56.78 SUCCESS   ,  ratio=  4.95
6.18.7   elapsedHours: test=  7.21 SUCCESS   ,  prod= 42.28 TERMINATED,  ratio=  5.86
6.18.8   elapsedHours: test= 11.36 SUCCESS   ,  prod= 46.88 TERMINATED,  ratio=  4.13
6.19.0   elapsedHours: test=  5.52 SUCCESS   ,  prod= 65.65 SUCCESS   ,  ratio= 11.89
6.19.1   elapsedHours: test=  7.03 SUCCESS   ,  prod= 59.52 SUCCESS   ,  ratio=  8.47
6.19.2   elapsedHours: test=  6.69 SUCCESS   ,  prod= 64.18 SUCCESS   ,  ratio=  9.60
6.19.3   elapsedHours: test=  7.36 SUCCESS   ,  prod= 46.86 TERMINATED,  ratio=  6.37
6.19.4   elapsedHours: test=  6.73 SUCCESS   ,  prod= 46.88 TERMINATED,  ratio=  6.97
6.19.5   elapsedHours: test= 11.51 SUCCESS   ,  prod= 63.08 SUCCESS   ,  ratio=  5.48
6.19.6   elapsedHours: test=  6.60 SUCCESS   ,  prod= 60.41 SUCCESS   ,  ratio=  9.16
6.19.7   elapsedHours: test=  5.75 SUCCESS   ,  prod= 46.87 TERMINATED,  ratio=  8.16
6.20.0   elapsedHours: test= 11.52 SUCCESS   ,  prod= 60.94 SUCCESS   ,  ratio=  5.29
6.20.1   elapsedHours: test=  7.56 SUCCESS   ,  prod= 46.79 TERMINATED,  ratio=  6.19
6.20.2   elapsedHours: test=  6.46 SUCCESS   ,  prod= 46.43 TERMINATED,  ratio=  7.18
6.20.3   elapsedHours: test=  6.36 SUCCESS   ,  prod= 60.95 SUCCESS   ,  ratio=  9.59
6.20.4   elapsedHours: test=  4.71 SUCCESS   ,  prod= 47.29 SUCCESS   ,  ratio= 10.03

Clearly, the new code successfully allows a large number of threads to be used efficiently.  This should, if the images pass a final validation, allow us to complete the remaining ~10,000 or so sensor-visits within the normal NERSC 48-hour batch queue.
Interestingly, the elapsed time ratio is far from constant, ranging from 4 to >12.  It is known that random number sequences will be different when the threading changes (and, in fact, even if the threading remains the same, if >1 thread), but a factor of x3 seems excessive.  There is not yet an explanation for these differences.
Comparison of results
Two simple methods of comparison were carried out:

ds9 to visually compare the image files
the 'fdiff' tool to compare the "electron" files for both runs

These tests were applied to a single sensor-visit: R43_S22
The ds9 test involved setting the scale to be the same for the two images, fiddling with the color mapping, and then "blinking" them against each other.  The qualitative result is that the basic sky images appear identical (there are the occasional cosmic rays in one image but not the other due to random number differences).  The overall sky brightness is slightly different.  A view of the two images "tiled" is shown below.

The 'fdiff' tool showed the expected differences, but also some which may not be expected.  A full listing of the header differences is presented below.  Perhaps John or others can comment on the zernike parameter differences, how these differences come about and whether they are reasonable.
               FDIFF V1.1.4

lsst_e_12449_f5_R43_S22_E000.fits.gz[0] <     ../../../test-WFD-y/output/000006/lsst_e_12449_f5_R43_S22_E000.fits.gz[0] >

< card 65: AIRMASS  = 1.03033619727997
> card 65: AIRMASS  = 1.03033587642677
---
< card 71: AZIMUTH  =<PHONE_NUMBER>91532
> card 71: AZIMUTH  =<PHONE_NUMBER>03656
---
< card 446: BOD 0 3  = -2.733317E-05
> card 446: BOD 0 3  = 1.230041E-05
---
< card 447: BOD 0 4  = 1.69567E-06
> card 447: BOD 0 4  = 1.162446E-05
---
< card 448: BOD 0 5  = 1.481025E-05
> card 448: BOD 0 5  = 4.973081E-06
---
< card 452: BOD 1 3  = -3.077166E-05
> card 452: BOD 1 3  = -1.925894E-05
---
< card 453: BOD 1 4  = -4.57326E-05
> card 453: BOD 1 4  = 5.064447E-05
---
< card 454: BOD 1 5  = -9.247979E-05
> card 454: BOD 1 5  = 1.761588E-07
---
< card 458: BOD 2 3  = -2.733317E-05
> card 458: BOD 2 3  = 1.230041E-05
---
< card 459: BOD 2 4  = 1.69567E-06
> card 459: BOD 2 4  = 1.162446E-05
---
< card 460: BOD 2 5  = 1.481025E-05
> card 460: BOD 2 5  = 4.973081E-06
---
< card 633: BRANCH  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.9'
> card 633: BRANCH  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.14'
---
< card 15: CD1_1  = -5.29596186877207E-05
> card 15: CD1_1  = -5.29596138098411E-05
---
< card 16: CD1_2  = 1.67838774285813E-05
> card 16: CD1_2  = 1.67838928201687E-05
---
< card 17: CD2_1  = -1.67838774285813E-05
> card 17: CD2_1  = -1.67838928201687E-05
---
< card 18: CD2_2  = -5.29596186877207E-05
> card 18: CD2_2  = -5.29596138098411E-05
---
< card 611: FPFILE  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.9/data/l'
> card 611: FPFILE  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.14/data/'
---
< card 37: IMGDIR  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.9/data/i'
> card 37: IMGDIR  = '/global/common/software/lsst/cori-haswell-intel/phosim/v3.7.14/data/'
---
< card 68: ROTANGZ  =<PHONE_NUMBER>83143
> card 68: ROTANGZ  =<PHONE_NUMBER>34944
---
< card 36: SEDDIR  = '/global/cscratch1/sd/descpho/Pipeline-tasks/DC2-R1-2p-WFD-y/000006/S'
> card 36: SEDDIR  = '/global/cscratch1/sd/descpho/Pipeline-tasks/test-WFD-y/000006/SEDs'
---
< card 70: ZENITH  = 13.9381256984265
> card 70: ZENITH  = 13.9380538056427
---
< card 82: ZER 0 3  = -2.733317E-05
> card 82: ZER 0 3  = 1.230041E-05
---
< card 83: ZER 0 4  = 1.69567E-06
> card 83: ZER 0 4  = 1.162446E-05
---
< card 84: ZER 0 5  = 1.481025E-05
> card 84: ZER 0 5  = 4.973081E-06
---
< card 85: ZER 0 6  = 9.755061E-06
> card 85: ZER 0 6  = -1.523945E-05
---
< card 86: ZER 0 7  = 1.505774E-05
> card 86: ZER 0 7  = 6.204145E-06
---
< card 87: ZER 0 8  = 1.281214E-06
> card 87: ZER 0 8  = -1.903451E-05
---
< card 88: ZER 0 9  = -6.040004E-06
> card 88: ZER 0 9  = 1.590424E-06
---
< card 89: ZER 010  = 1.232056E-05
> card 89: ZER 010  = 6.291264E-06
---
< card 90: ZER 011  = 1.867872E-06
> card 90: ZER 011  = -5.081734E-07
---
< card 91: ZER 012  = 7.503871E-06
> card 91: ZER 012  = 7.349473E-06
---
< card 92: ZER 013  = -1.807692E-06
> card 92: ZER 013  = -1.243054E-06
---
< card 93: ZER 014  = 1.78905E-06
> card 93: ZER 014  = -1.022187E-05
---
< card 94: ZER 015  = 8.361695E-06
> card 94: ZER 015  = -2.125326E-06
---
< card 95: ZER 016  = -2.092621E-06
> card 95: ZER 016  = -6.186138E-06
---
< card 96: ZER 017  = -9.316427E-06
> card 96: ZER 017  = -1.349618E-07
---
< card 97: ZER 018  = 2.991919E-06
> card 97: ZER 018  = 1.25934E-05
---
< card 98: ZER 019  = 3.925723E-06
> card 98: ZER 019  = 1.487202E-06
---
< card 99: ZER 020  = -4.796312E-06
> card 99: ZER 020  = 2.770873E-06
---
< card 110: ZER 1 3  = -3.077166E-05
> card 110: ZER 1 3  = -1.925894E-05
---
< card 111: ZER 1 4  = -4.57326E-05
> card 111: ZER 1 4  = 5.064447E-05
---
< card 112: ZER 1 5  = -9.247979E-05
> card 112: ZER 1 5  = 1.761588E-07
---
< card 113: ZER 1 6  = 5.627657E-06
> card 113: ZER 1 6  = -3.831325E-06
---
< card 114: ZER 1 7  = -1.3229E-05
> card 114: ZER 1 7  = 5.332543E-07
---
< card 115: ZER 1 8  = -3.95235E-05
> card 115: ZER 1 8  = 1.436844E-06
---
< card 116: ZER 1 9  = 2.477007E-05
> card 116: ZER 1 9  = -1.273148E-05
---
< card 117: ZER 110  = -1.249233E-05
> card 117: ZER 110  = 9.565704E-06
---
< card 118: ZER 111  = 2.938097E-06
> card 118: ZER 111  = -2.556876E-06
---
< card 119: ZER 112  = -1.971055E-05
> card 119: ZER 112  = 9.10336E-06
---
< card 120: ZER 113  = 1.15437E-06
> card 120: ZER 113  = 1.93246E-06
---
< card 121: ZER 114  = 5.489848E-06
> card 121: ZER 114  = -4.240682E-06
---
< card 122: ZER 115  = -6.689914E-06
> card 122: ZER 115  = -7.73816E-07
---
< card 123: ZER 116  = -4.706865E-06
> card 123: ZER 116  = -3.688771E-06
---
< card 124: ZER 117  = -1.715983E-06
> card 124: ZER 117  = 4.391131E-06
---
< card 125: ZER 118  = 9.166808E-06
> card 125: ZER 118  = -1.177108E-05
---
< card 126: ZER 119  = -1.678203E-06
> card 126: ZER 119  = 4.99118E-06
---
< card 127: ZER 120  = 1.333635E-05
> card 127: ZER 120  = 1.473233E-05
---
< card 138: ZER 2 3  = -2.733317E-05
> card 138: ZER 2 3  = 1.230041E-05
---
< card 139: ZER 2 4  = 1.69567E-06
> card 139: ZER 2 4  = 1.162446E-05
---
< card 140: ZER 2 5  = 1.481025E-05
> card 140: ZER 2 5  = 4.973081E-06
---
< card 141: ZER 2 6  = 9.755061E-06
> card 141: ZER 2 6  = -1.523945E-05
---
< card 142: ZER 2 7  = 1.505774E-05
> card 142: ZER 2 7  = 6.204145E-06
---
< card 143: ZER 2 8  = 1.281214E-06
> card 143: ZER 2 8  = -1.903451E-05
---
< card 144: ZER 2 9  = -6.040004E-06
> card 144: ZER 2 9  = 1.590424E-06
---
< card 145: ZER 210  = 1.232056E-05
> card 145: ZER 210  = 6.291264E-06
---
< card 146: ZER 211  = 1.867872E-06
> card 146: ZER 211  = -5.081734E-07
---
< card 147: ZER 212  = 7.503871E-06
> card 147: ZER 212  = 7.349473E-06
---
< card 148: ZER 213  = -1.807692E-06
> card 148: ZER 213  = -1.243054E-06
---
< card 149: ZER 214  = 1.78905E-06
> card 149: ZER 214  = -1.022187E-05
---
< card 150: ZER 215  = 8.361695E-06
> card 150: ZER 215  = -2.125326E-06
---
< card 151: ZER 216  = -2.092621E-06
> card 151: ZER 216  = -6.186138E-06
---
< card 152: ZER 217  = -9.316427E-06
> card 152: ZER 217  = -1.349618E-07
---
< card 153: ZER 218  = 2.991919E-06
> card 153: ZER 218  = 1.25934E-05
---
< card 154: ZER 219  = 3.925723E-06
> card 154: ZER 219  = 1.487202E-06
---
< card 155: ZER 220  = -4.796312E-06
> card 155: ZER 220  = 2.770873E-06
---
These two data units are not identical.

For reference, the command file and instanceCatalog files are located here for the production and test runs:
/global/cscratch1/sd/descpho/Pipeline-tasks/DC2-R1-2p-WFD-y/000006
/global/cscratch1/sd/descpho/Pipeline-tasks/test-WFD-y/000006

The image files (both electron and amplifier) are located here:
/global/projecta/projectdirs/lsst/production/DC2/DC2-R1-2p-WFD-y/output/000006
/global/projecta/projectdirs/lsst/production/DC2/test-WFD-y/output/000006

In the test of v3.7.14, were there always 5 instances of phosim running per KNL node, or would there possibly have been fewer than 5 instances running on the node toward the end when trying to finish up the last jobs? I was wondering if having fewer than 5 active phosims at some point in a sensor visit could make it look artificially fast.

Adrian, yes, there were times times when there were fewer than five instances per node.  The times quoted are elapsed clock time per instance.  Are you suggesting that the presence of other instances running on the same node might slow down the processing?

I think there are several effects that could change the wall clock of a particular phosim instance depending on how much total work is active on a node.
When a compute node is doing more total work, there will be less opportunity to use turbo mode and boost the clock frequency of compute cores. This is probably a ~20% effect.
I'm not exactly sure how pthreads work with the compute node kernel, slurm, and affinity settings, but it is possible that pthreads are not "pinned" to a particular set of cores/HW-threads, and they may be allowed to migrate around the hardware resources. In this case, if there are fewer than 5 phosim instances running on a node, the pthreads from the remaining instances might be allowed to "spread out" and run a bit more quickly (though possibly less efficiently overall). I don't know as much about the runtime rules for pthreads on XC40/KNL as I do about OpenMP threads, so I'm trying to chat with Intel folks to learn more about how to control and measure this.
I'm also not exactly sure how the workflow works, but I had assumed that it tried to keep the same number of phosim instances active on a compute node until hitting the wall clock limit, eg. when one phosim instance on a node finishes, the workflow finds the next sensor visit from a task queue and launches it on the compute node, so very little time is spent with fewer than the expected number of phosim instances active.
In my (simple) performance testing, I've been replicating the same phosim sensor visit N times on a single KNL node. So I'm trying to keep N phosim instances active for the same amount of wall clock, and trying to replicate roughly the same average throughput as my understanding of the workflow. I suspect there will be some difference in effective throughput during the single-threaded regions, but I'm hoping that difference is relatively small. I would expect the workflow to be slightly more efficient when the single-threaded region of one phosim instance overlaps with multi-threaded regions of the other phosim instances, as the multi-threaded regions of the other instances might get slightly more done when there are some extra resources available.

yes, but there a dozens of other things too.  when you switch to a new versions its as if you went into a brand new universe.   the random parts will get re-realized.
On May 16, 2018, at 7:36 AM, Adrian Pope <notifications<EMAIL_ADDRESS>wrote:
John - I'm curious about the possible slight sky brightness variation. Would different realizations have slightly different shutter opening times? Slightly different realizations of the atmospheric effects that affect brightness?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/DC2_Repo/issues/65#issuecomment-389542184, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8gCMhQxpqX3ogh94c6w-ppuONRbSks5tzDl8gaJpZM4RHrHX.

Tuesday 22 May 2018 Update
Final preparations are being made to begin reprocessing the 9,888 long-running (and timed-out) phosim (raytrace) jobs.  This mid-course correction contains two changes:

phoSim version 3.7.9 -> 3.7.14
# threads 8 -> 54

Because the number of threads is deeply embedded into the workflow, several steps were required to update the workflow.   For posterity, those steps are summarized here.

[SLAC] Brian changed the NTHREADS pipeline variable in all 12 workflow tasks associated with Run 1.2p
[SLAC] a script (longRuns.{sh,py}) was developed to generate a list of the 9888 'terminated' workflow jobs that timed-out; a copy of this list was transferred to NERSC.
[NERSC] a second script (genLongRunList.py) was developed to harvest obsHistID/sensorID from log files and appended to list of terminated jobs.  A reference copy is stored here:
/global/projecta/projectdirs/lsst/production/DC2/Run1.2/DC2-longRunList.ref
This file is a simple text file containing one line per terminated job and in the following format. Note that this file is a record of exactly which sensors were processed with phoSim v3.7.14 should any questions arise in the future.  Example entry in the long runs file:
DC2-R1-2p-uDDF-g 091<PHONE_NUMBER>508 R14_S11

col 1: workflow (task) name
col 2: visit index (top-level stream)
col 3: raft index (2nd level stream)
col 4: sensor index (3rd level stream)
col 5: obsHistID
col 6: sensorID

[NERSC] a final script (modPARSfiles.py), guided by the terminated runs list, was developed to modify 9888 raytrace*.pars files in the phoSim work directories changing the requested number of threads from 8 to 54.  The original raytrace*.pars files were first copied to raytrace*.pars.backup  prior to any modification.  It also worth reminding everyone that the entire set of phoSim $SCRATCH/work directories will disappear in a few weeks and it was never our intent to keep them.  This includes instanceCatalogs, SED files and phoSim work files.  Other files, e-image, amplifier files, centroid files, log files, etc., however, are all preserved permanently in the project area at NERSC.

Tom-
if you switch to a different version, you’ll have to run all the jobs of the same visit over again.
john
On May 22, 2018, at 6:28 PM, Tom Glanzman <notifications<EMAIL_ADDRESS>wrote:
Tuesday 22 May 2018 Update
Final preparations are being made to begin reprocessing the 9,888 long-running (and timed-out) phosim (raytrace) jobs. This mid-course correction contains two changes:

phoSim version 3.7.9 -> 3.7.14

threads 8 -> 54

Because the number of threads is deeply embedded into the workflow, several steps were required to update the workflow. For posterity, those steps are summarized here.

[SLAC] Brian changed the NTHREADS pipeline variable in all 12 workflow tasks associated with Run 1.2p
[SLAC] a script (longRuns.{sh,py}) was developed to generate a list of the 9888 'terminated' workflow jobs that timed-out; a copy of this list was transferred to NERSC.
[NERSC] a second script (genLongRunList.py) was developed to harvest obsHistID/sensorID from log files and appended to list of terminated jobs. A reference copy is stored here:
/global/projecta/projectdirs/lsst/production/DC2/Run1.2/DC2-longRunList.ref
This file is a simple text file containing one line per terminated job and in the following format. Note that this file is a record of exactly which sensors were processed with phoSim v3.7.14 should any questions arise in the future. Example entry in the long runs file:
DC2-R1-2p-uDDF-g 091<PHONE_NUMBER>508 R14_S11
*   col 1: workflow (task) name
*   col 2: visit index (top-level stream)
*   col 3: raft index (2nd level stream)
*   col 4: sensor index (3rd level stream)
*   col 5: obsHistID
*   col 6: sensorID
[NERSC] a final script (modPARSfiles.py), guided by the terminated runs list, was developed to modify 9888 raytrace*.pars files in the phoSim work directories changing the requested number of threads from 8 to 54. The original raytrace*.pars files were first copied to raytrace*.pars.backup prior to any modification. It also worth reminding everyone that the entire set of phoSim $SCRATCH/work directories will disappear in a few weeks and it was never our intent to keep them. This includes instanceCatalogs, SED files and phoSim work files. Other files, e-image, amplifier files, centroid files, log files, etc., however, are all preserved permanently in the project area at NERSC.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/DC2_Repo/issues/65#issuecomment-391160859, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8ozORJVxFCz0aXHUhghDhqOggSxRks5t1JETgaJpZM4RHrHX.

Wednesday 23 May 2018 Update
Production of the remaining 9888 long-running sensor-visits started up earlier today, using phoSim 3.7.14 and 54 threads/instance (and 5 instances/node).  This is something of an experiment - to see how well a fully-loaded KNL node handles this configuration.  CPU utilization looks great and memory consumption is <15 GB (out of 95 GB).  But there have already been about a dozen mysterious failures (sig4 and sig11) which I'm keeping an eye on.  More news tomorrow.

Thursday 24 May 2018 Update
Processing continues...  Of the 9,888 long-running jobs, here is the status as of 14:35

Jobs successfully completed: 3736 (38% of target)
Jobs currently running: 1752 (18%)
Jobs remaining to start: 4400 (44%)

There continue to be rather bizarre crashes at the few % level.  It turns out that all of these oddball failures occur on exactly four KNL nodes (nid05377, nid05594, nid02566, nid02620) so I've submitted a ticket with the NERSC folks

In addition, there was a rash of 23 crashes at 11:13am this morning -- all associated with the perennial problem of reading in the (phosim config) focalplanelayout.txt file.  This is being treated as a transient file system issue, so these have been rolled back and I expect they may well succeed next time around.

Friday 25 May 2018 Update
(On the road, so this will be brief.)  The first round of 48-hour jobs completed this morning and the result if this first attempt was >50% of the remaining sensor-visits completed successfully.  Status as of 15:20

Jobs successfully completed: 5571 (56% of target)
jobs currently running: 260 (that's because I waited to submit more Pilots to assess how the first set did)
Jobs remaining to start: 4057

Saturday 26 May 2018 Update
Production continues - and with a refreshing lack of failures!  Perhaps the reported machines have been taken off line...  Status as of 20:50:

Jobs successfully completed: 7569 (77% of target)
Jobs running: 2319
Jobs remaining to start: 0

WIth only 2319 sensor-visits remaining (1.5% of the total in Run 1.2p), production may complete tomorrow (Sunday).

Tuesday 29 May 2018 Update
The current round of DC2 Run 1.2p processing, the "long-running job campaign", is nearly over, with some mop-up and a relatively small number of extra-long-running jobs to deal with.  Here is the status of the long-running job campaign as of 07:00 today:

Number of long-running jobs at start of campaign: 9888
Jobs successfully completed: 9287 (94% of target)
Jobs running: 0
Jobs remaining to start: 25
Extra long running jobs that exceeded 48-hours running on 54 cores: 576 (6% of target)

During the long-running job campaign, there were, as described previously, a number of abnormal crashes.  The 25 remaining jobs then got restarted on Pilots that had <<48 hours remaining, and they did not complete.  A set of fresh Pilots has now been submitted/running to mop up these few jobs.
Overall, the status of Run 1.2p now looks like this:

Total number of visits: 2001
Number of fully complete visits: 1984 (99% of entire campaign)

Recovery of the remaining 17 visits will require additional workflow tweaks to allow for a larger number of threads.  Not knowing how much longer these jobs need, my plan is to raise the number of threads from 54 (1/5 of a KNL node) to either 90 (1/3 of a KNL node) or to 135 (1/2 a KNL node), in both cases leaving a thread or two for the OS.

Thursday 31 May Update:
All "long-running" jobs are now complete.  Only 588 "extra long-running" jobs (sensor-visits) remain as indicated in the table.  These jobs exceeded 48-hours and 54 threads on Cori KNL.

Survey/Band
# sensor-visits remaining
# visits affected

WFD u-band
334
4

WFD r-band
37
1

WFD y-band
216
3

uDDF g-band
1
1

TOTAL
588
9

Status as of 09:30:

Number of fully complete visits: 1992 (99.6% of Run 1.2p)
Number of long-running visits completed in last campaign: 9300 (94% of target)

Saturday 2 June 2018 Update:
Shockingly, of the 588 extra-long running phosim (raytrace) steps started two days ago, exactly one successfully completed, and that one was likely a shorter-running job that sneaked onto the list.  So what happened?  As the 48-hour time limit approached and with no sign of jobs completing, I logged into a handful of KNL nodes to see what what was happening.  In all cases, instances of raytrace were running in a single thread.  I had previously confirmed that raytrace was running with all 90 threads early on.  This result means that a single bright source likely caused the extended run times, and that increasing the number of threads will likely have no impact - other than to waste even more core hours.
It would be good to know exactly what catalog items are triggering these anomalously long run times.  The log files are not much help - they all stall at the point prior to reporting the source summary.  The phosim input data (instance catalog, command file, SEDs) are all preserved.  Perhaps @johnrpeterson  could make a suggestion?

In other news (and this is very preliminary), I've resurrected my old checkpoint test scripts and tried to use the officially-installed (by NERSC) version of dmtcp (v3.0.0) on phoSim v3.7.14.  What I am seeing is that when phosim is still in its single-thread operation, all works as it should, including the creation of checkpoint files when requested.  Restarting from the checkpoint files also seems to work, although current tests only check the mechanics of the system, not the detailed output.  However, once phoSim enters the multi-threaded phase of operation, requests to create a checkpoint appear to be totally ignored; no checkpoint files are created and phoSim continues merrily along its way.  Bah!  This symptom holds true for my own builds of dmtcp as well, so it is unlikely a bad build.  This is being investigated...

Tom-
i’d just check the brightest magnitude in the catalog.  i thought we were using a magnitude cut (10?)?
john
On Jun 2, 2018, at 5:40 PM, Tom Glanzman <notifications<EMAIL_ADDRESS>wrote:
Saturday 2 June 2018 Update:
Shockingly, of the 588 extra-long running phosim (raytrace) steps started two days ago, exactly one successfully completed, and that one was likely a shorter-running job that sneaked onto the list. So what happened? As the 48-hour time limit approached and with no sign of jobs completing, I logged into a handful of KNL nodes to see what what was happening. In all cases, instances of raytrace were running in a single thread. I had previously confirmed that raytrace was running with all 90 threads early on. This result means that a single bright source likely caused the extended run times, and that increasing the number of threads will likely have no impact - other than to waste even more core hours.
It would be good to know exactly what catalog items are triggering these anomalously long run times. The log files are not much help - they all stall at the point prior to reporting the source summary. The phosim input data (instance catalog, command file, SEDs) are all preserved. Perhaps @johnrpetersonhttps://github.com/johnrpeterson could make a suggestion?

In other news (and this is very preliminary), I've resurrected my old checkpoint test scripts and tried to use the officially-installed (by NERSC) version of dmtcp (v3.0.0) on phoSim v3.7.14. What I am seeing is that when phosim is still in its single-thread operation, all works as it should, including the creation of checkpoint files when requested. Restarting from the checkpoint files also seems to work, although current tests only check the mechanics of the system, not the detailed output. However, once phoSim enters the multi-threaded phase of operation, requests to create a checkpoint appear to be totally ignored; no checkpoint files are created and phoSim continues merrily along its way. Bah! This symptom holds true for my own builds of dmtcp as well, so it is unlikely a bad build. This is being investigated...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/DC2_Repo/issues/65#issuecomment-394119148, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8pAjpA6Jr5sAjSFo1f6wCv7ALQNCks5t4wZggaJpZM4RHrHX.

Monday 4 June 2018 Final Update:
The long-running jobs noted over the weekend were discovered to contain a very bright (M4.5) galaxy.  How this happened is being studied by the experts and will hopefully not happen again in Run 2.0.  Fixing the problem, either by removing the offending sources completely or manually resetting their magnitudes to "10" seemed risky given how that might affect downstream processing.  Thus, it was decided DC2 Run 1.2p phoSim production would be officially deemed complete.
The design goal for Run 1.2p is shown in the table at the very top of this issue.  The "missing" visits are tabulated below:

Survey/Band
# sensor-visits unfinished
# visits affected

WFD u-band
334
4

WFD r-band
37
1

WFD y-band
216
3

TOTAL
587
8

After a thrilling 2-month phoSim production run, for Run 1.2p this is The End.

To be sure, this is THE BEGINNING for the analysis working groups. We hope that you will be able to use the results to design and test data access patterns and pipelines and will help with very important validation tasks. THANKS @TomGlanzman!

good job!  and this makes sense now.
On Jun 4, 2018, at 5:04 PM, Tom Glanzman <notifications<EMAIL_ADDRESS>wrote:
Monday 4 June 2018 Final Update:
The long-running jobs noted over the weekend were discovered to contain a very bright (M4.5) galaxy. How this happened is being studied by the experts and will hopefully not happen again in Run 2.0. Fixing the problem, either by removing the offending sources completely or manually resetting their magnitudes to "10" seemed risky given how that might affect downstream processing. Thus, it was decided DC2 Run 1.2p phoSim production would be officially deemed complete.
The design goal for Run 1.2p is shown in the table at the very top of this issue. The "missing" visits are tabulated below:
Survey/Band     # sensor-visits unfinished      # visits affected
WFD u-band      334     4
WFD r-band      37      1
WFD y-band      216     3
TOTAL   587     8

After a thrilling 2-month phoSim production run, for Run 1.2p this is The Endhttps://www.youtube.com/watch?v=JSUIQgEVDM4.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/LSSTDESC/DC2_Repo/issues/65#issuecomment-394498941, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8oZqysjqbaeJrtw6N7e2RlrBOHtdks5t5aDogaJpZM4RHrHX.

Post-postscript:
For the record, here is a snapshot of the final workflow status:
	DC2 Run 1.2p summary
	====================
   2018-06-14 09:00:11.115332  Pacific
  (Note: jobs in the "QUEUED" state are not tallied, and the "ALL" column is currently unreliable)

=======Task and (batch) Process-step Rollup==============================
+-----------+-----------+-----------+-----------+----------+--------------+-------+
|   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------+-----------+-----------+-----------+----------+--------------+-------|
|         0 |         0 |         0 |    206195 |        0 |          587 |     0 |
+-----------+-----------+-----------+-----------+----------+--------------+-------+

=======Task Rollup===============================================
+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| setupVisit    |         0 |         0 |         0 |      2001 |        0 |            0 |     0 |
| RunTrim       |         0 |         0 |         0 |     44022 |        0 |            0 |     0 |
| RunRaytrace   |         0 |         0 |         0 |    158179 |        0 |          587 |     0 |
| finishVisit   |         0 |         0 |         0 |      1993 |        0 |            0 |     0 |
+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+

======================================================

Total rolled back process steps =  368861

=======Detailed Task Summary======================================
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-u | setupVisit    |         0 |         0 |         0 |        67 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-u | RunTrim       |         0 |         0 |         0 |      1474 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-u | RunRaytrace   |         0 |         0 |         0 |      4487 |        0 |          334 |     0 |
| DC2-R1-2p-WFD-u | finishVisit   |         0 |         0 |         0 |        63 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-g | setupVisit    |         0 |         0 |         0 |        91 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | RunTrim       |         0 |         0 |         0 |      2002 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | RunRaytrace   |         0 |         0 |         0 |      6118 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-g | finishVisit   |         0 |         0 |         0 |        91 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-r | setupVisit    |         0 |         0 |         0 |       245 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-r | RunTrim       |         0 |         0 |         0 |      5390 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-r | RunRaytrace   |         0 |         0 |         0 |     18437 |        0 |           37 |     0 |
| DC2-R1-2p-WFD-r | finishVisit   |         0 |         0 |         0 |       244 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-i | setupVisit    |         0 |         0 |         0 |       223 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-i | RunTrim       |         0 |         0 |         0 |      4906 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-i | RunRaytrace   |         0 |         0 |         0 |     16243 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-i | finishVisit   |         0 |         0 |         0 |       223 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-z | setupVisit    |         0 |         0 |         0 |       247 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-z | RunTrim       |         0 |         0 |         0 |      5434 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-z | RunRaytrace   |         0 |         0 |         0 |     18099 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-z | finishVisit   |         0 |         0 |         0 |       247 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task            | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-WFD-y | setupVisit    |         0 |         0 |         0 |       252 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-y | RunTrim       |         0 |         0 |         0 |      5544 |        0 |            0 |     0 |
| DC2-R1-2p-WFD-y | RunRaytrace   |         0 |         0 |         0 |     17809 |        0 |          216 |     0 |
| DC2-R1-2p-WFD-y | finishVisit   |         0 |         0 |         0 |       249 |        0 |            0 |     0 |
+-----------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-u | setupVisit    |         0 |         0 |         0 |       192 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-u | RunTrim       |         0 |         0 |         0 |      4224 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-u | RunRaytrace   |         0 |         0 |         0 |     16863 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-u | finishVisit   |         0 |         0 |         0 |       192 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-g | setupVisit    |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-g | RunTrim       |         0 |         0 |         0 |      3036 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-g | RunRaytrace   |         0 |         0 |         0 |     12135 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-g | finishVisit   |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-r | setupVisit    |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-r | RunTrim       |         0 |         0 |         0 |      3036 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-r | RunRaytrace   |         0 |         0 |         0 |     12129 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-r | finishVisit   |         0 |         0 |         0 |       138 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-i | setupVisit    |         0 |         0 |         0 |       137 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-i | RunTrim       |         0 |         0 |         0 |      3014 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-i | RunRaytrace   |         0 |         0 |         0 |     12043 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-i | finishVisit   |         0 |         0 |         0 |       137 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-z | setupVisit    |         0 |         0 |         0 |       136 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-z | RunTrim       |         0 |         0 |         0 |      2992 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-z | RunRaytrace   |         0 |         0 |         0 |     11930 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-z | finishVisit   |         0 |         0 |         0 |       136 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+
| Task             | ProcessStep   |   WAITING |   PENDING |   RUNNING |   SUCCESS |   FAILED |   TERMINATED |   ALL |
|------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------|
| DC2-R1-2p-uDDF-y | setupVisit    |         0 |         0 |         0 |       135 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-y | RunTrim       |         0 |         0 |         0 |      2970 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-y | RunRaytrace   |         0 |         0 |         0 |     11886 |        0 |            0 |     0 |
| DC2-R1-2p-uDDF-y | finishVisit   |         0 |         0 |         0 |       135 |        0 |            0 |     0 |
+------------------+---------------+-----------+-----------+-----------+-----------+----------+--------------+-------+

Elapsed time for workflow database queries =  33.522016  sec

And last, a final plot of Cori utilization (# of Cori-KNL nodes as a function of time) during the entire Run 1.2p period.  Remember that a single Cori-KNL node contains 68 cores / 272 hardware threads, thus each node could support up to ~34 instances of 8-threaded 'raytrace'.


XCSkip equivalents in swift-testing

Description
The old XCTest based test case:
func testDemo() throws {
    let value: Int? = ...
    guard let value else {
        throw XCTSkip("Some skip message here")

    }
    XCTAssertEqual(value, 3)
}

The closest migrating to swift-testing I could think of.
func testDemo() throws {
    let value: Int? = ...
    guard let value else {
        withKnownIssue {
            throw "Some skip message here"
        }
        return
    }
    #expect(BloomFilter(type: type).value == expectedValue)
}
extension String: Error {}

Expected behavior
XCSkip have an equivalent in swift-testing
Actual behavior
XCSkip have no equivalent in swift-testing
Steps to reproduce
No response
swift-testing version/commit hash
No response
Swift & OS version (output of swift --version ; uname -a)
No response

Use the .enabled(if:), .disabled(if:), and .disabled() traits to control if a test will run. See: https://swiftpackageindex.com/apple/swift-testing/main/documentation/testing/enablinganddisabling

See also: https://swiftpackageindex.com/apple/swift-testing/main/documentation/testing/migratingfromxctest#Converting-XCTSkip-XCTSkipIf-and-XCTSkipUnless-calls


No Longer any Way to Open Recently Used in a New Tab.

Control-T pops up the folder to select a file.  I don't see any way to create a new tab and then open a recently used file in there.
I'll add that despite having "Restore Tabs" checked, it lost mine since the last update.

It looks like you may have reported an issue against the wrong package. This qpdf is a command-line tool and C++ library and doesn't have a graphical UI. I think there are other packages with similar names that might have tabs and so forth. Since this issue doesn't apply to this package, I'm closing it. I hope you will be able to report it to the right place since it's helpful to projects when people report issues. Thanks.

Oops.  Yes, thanks.  qpdfview


Step fails to install sudo

Hey folks,
thanks for providing this step! 💪  Unfortunately I'm unable getting my wercker.yml to work with this step running since the script tries to install sudo. This is the output I'm receiving:
source "/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run.sh" < /dev/null
/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run_terraform: line 1673: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run_terraform: line 1674: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run_terraform: line 1675: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run_terraform: line 1677: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
/pipeline/run-terraform-69fd16f6-82bb-4a8d-a4c7-7b2d3e1b2834/run_terraform: line 1678: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
run_terraform_src: WARN: Installing missing packages 'sudo gawk curl p7zip-full'
run_terraform_src: FAIL: Essential dependency 'sudo' not found on the PATH (with arguments '-p

This is my wercker.yml with some anonymized data:
build:
    box: ubuntu
    steps:
    -<EMAIL_ADDRESS>        action: "plan"
        out_file: "output_file"
        state_stor_bucket: "fancy-bucket-123"
        state_stor_bucket_region: "eu-central-1"
        state_stor_key: "terraform.tfstate"
    -<EMAIL_ADDRESS>        action: "apply"
        in_file: "output_file"
        state_stor_bucket: "fancy-bucket-123"
        state_stor_bucket_region: "eu-central-1"
        state_stor_key: "terraform.tfstate"
    after-steps:
    - slack-notifier:
        url: $SLACK_URL
        channel: builds
        username: Wercker - Terraform

Do I need to use a special box for this step? Thanks in advance for any help! :)

Hello @apinnecke, this step hasn't gotten the appropriate attention lately. I should test it myself again, and update for TF 0.7 (and the upcoming 0.8).

Hey @ringods,
thanks a lot for the super quick response! So do you find some time to do this within the next week? Since this isn't urgent at all I'd just need to know weather some near-future work is planned to shift to a different solution otherwise. :)

I'm also watching this issue to be resolved: https://github.com/hashicorp/terraform/issues/7927
With that in place, one could terraform plan in Wercker build phase and terraform apply with the plan file in the Wercker deploy phase.

And as a current workaround, you can also this box inside a custom pipeline:
box:
  id: hashicorp/terraform:0.7.12
  # Set a shell as entrypoint for wercker. The image defaults to running terraform immediately.
  entrypoint: /bin/bash -c

Unfortunately the provided solution does not work for me. This is what Wercker says about it:
Guest command failed with exit code -1: mkdir -p "/pipeline"

This is the log, including the latest wercker.yml:
Running wercker version: 1.0.649 (Compiled at: 2016-10-11T01:24:24Z, Git commit: 046824c5b5234c15f99639d44cd95fceb151e731)
Using config:
build:
    box:
      id: hashicorp/terraform:0.7.12
      # Set a shell as entrypoint for wercker. The image defaults to running terraform immediately.
      entrypoint: /bin/bash -c
    steps:
    -<EMAIL_ADDRESS>        action: "plan"
        out_file: "output_file"
        state_stor_bucket: "fancy-tf-state"
        state_stor_bucket_region: "eu-central-1"
        state_stor_key: "terraform.tfstate"
    -<EMAIL_ADDRESS>        action: "apply"
        in_file: "output_file"
        state_stor_bucket: "fancy-tf-state"
        state_stor_bucket_region: "eu-central-1"
        state_stor_key: "terraform.tfstate"
    after-steps:
    - slack-notifier:
        url: $SLACK_URL
        channel: builds
        username: Wercker - Terraform

Pulling from hashicorp/terraform: 0.7.12
Already exists: 3690ec4760f9
Pulling fs layer: f8af16d0d0f5
Verifying Checksum: f8af16d0d0f5
Download complete: f8af16d0d0f5
Pull complete: f8af16d0d0f5
Digest: sha256:4f77a51c66e4406c54323fc781aa79f3a0737d67c073ea36a722449d552210f4
Status: Downloaded newer image for hashicorp/terraform:0.7.12

Any update on this issue?


Font position changes on moving text cursor in integrated terminal

VSCode Version: 1.31.0
OS Version: Mac OSX 10.14.3

Steps to Reproduce:

Open integrated terminal
Type something and move the text cursor left right

Does this issue occur when all extensions are disabled?: Yes

Appears to be related to the terminal.integrated.lineHeight setting. I normally have it set to 1.3 and I see this issue, but I tried changing it back to 1 and it doesn't happen.

Upstream issue: https://github.com/xtermjs/xterm.js/issues/1937


make the repo name and package name consistent

package name is 'leaflet-groupedlayercontrol' and repo name is 'Leaflet.groupedlayercontrol'
it's somewhat misleading and cost me about 30 seconds of time

The dot in the name comes from the Leaflet plugin convention, and the dasherized name was a requirement of the npm repo. I think the latter is more important and consistent in the javascript ecosystem, so I can rename the github repo accordingly.

Resolved in e628cee.
I haven't published a new version of the package, as github seems to be redirecting from the old address to the new one.

Thanks!

@ismyrnow is there value in renaming the files to follow the '-' scheme as well? it would technically be a breaking change

I'd rather leave it the way it is, since once you've found the package, file names don't matter as much. If it wasn't a breaking change, I'd probably just do it and publish a new patch version.


[BUG] Unknown collation: 'utf8mb4_0900_ai_ci'

I run docker in latest version
Error log
flowise-flowise-1  | yarn run v1.22.19
flowise-flowise-1  | $ run-script-os
flowise-flowise-1  | $ cd packages/server/bin && ./run start
flowise-flowise-1  | 2023-09-18 04:11:09 [INFO]: Starting Flowise...
flowise-flowise-1  | 2023-09-18 04:11:09 [INFO]: ⚡ [server]: Flowise Server is listening at 3000
flowise-flowise-1  | 2023-09-18 04:11:09 [INFO]: 📦 [server]: Data Source has been initialized!
flowise-flowise-1  | 2023-09-18 04:11:09 [ERROR]: ❌ [server]: Error during Data Source initialization: ER_UNKNOWN_COLLATION: Unknown collation: 'utf8mb4_0900_ai_ci'
flowise-flowise-1  | QueryFailedError: ER_UNKNOWN_COLLATION: Unknown collation: 'utf8mb4_0900_ai_ci'
flowise-flowise-1  |     at Query. (/usr/src/packages/node_modules/typeorm/driver/mysql/MysqlQueryRunner.js:158:37)
flowise-flowise-1  |     at Query. (/usr/src/packages/node_modules/mysql/lib/Connection.js:526:10)
flowise-flowise-1  |     at Query._callback (/usr/src/packages/node_modules/mysql/lib/Connection.js:488:16)
flowise-flowise-1  |     at Sequence.end (/usr/src/packages/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
flowise-flowise-1  |     at Query.ErrorPacket (/usr/src/packages/node_modules/mysql/lib/protocol/sequences/Query.js:92:8)
flowise-flowise-1  |     at Protocol._parsePacket (/usr/src/packages/node_modules/mysql/lib/protocol/Protocol.js:291:23)
flowise-flowise-1  |     at Parser._parsePacket (/usr/src/packages/node_modules/mysql/lib/protocol/Parser.js:433:10)
flowise-flowise-1  |     at Parser.write (/usr/src/packages/node_modules/mysql/lib/protocol/Parser.js:43:10)
flowise-flowise-1  |     at Protocol.write (/usr/src/packages/node_modules/mysql/lib/protocol/Protocol.js:38:16)
flowise-flowise-1  |     at Socket. (/usr/src/packages/node_modules/mysql/lib/Connection.js:88:28)

It work fine after I changed mysql migration code to:
ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;`
instead of  CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci


Automated cherry pick of #2007: Merge DaemonSet and Deployment template metadata as well

Cherry pick of #2007 on release-v1.27.
#2007: Merge DaemonSet and Deployment template metadata as well

/merge-when-ready delete-branch

OK, I will merge the pull request when it's ready, leave the commits as is when I merge it, and delete the branch after I've merged it.

I'm sorry but I failed delete the branch after merging the pull request.