  This work will be done at SLAC using existing Pipeline workflow tools (see issue #69 ).

After discussions with @rbiswas4 and @johnrpeterson a scheme has been established to merge the SNe SED files with the 'standard' SED file set (involving a local sym-link tree with the addition of a spectra_files/ directory).  Thanks to @jchiang87 for updating the SLAC DMstack to refresh the 'standard' SED file set.
At this moment, I have 17 instance catalog, SEDs pairs, each pair corresponding to a single phoSim invocation.  Additional instance catalog, SEDs pairs should soon be forthcoming via NERSC (LSST project directory).

phoSim is not happy running the Twinkles configuration.  The log file shows:

Atmosphere Creator
Creating layer 0.
Creating layer 1.
Creating layer 2.
Creating layer 3.
Creating layer 4.
Creating layer 5.
Creating layer 6.
Instrument Configuration
and then quietly exits (with return code of 0).  I have checked that each of the SED files referenced in the instance catalog exist and are readable.  However, I fear there are some limits within phoSim that are causing this problem.  For the instance catalog named phosim_input_484933.txt.gz,
Total lines      =  326110
Total SEDs       =  326090
Total good SEDs  =  326090
Total bad SEDs   =  0
Perhaps @johnrpeterson knows if these numbers look problematic?

@rbiswas4 have you seen this before?
On Wed, Feb 24, 2016 at 2:21 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

phoSim is not happy running the Twinkles configuration. The log file shows:
Atmosphere Creator
Creating layer 0.
Creating layer 1.
Creating layer 2.
Creating layer 3.
Creating layer 4.
Creating layer 5.
Creating layer 6. Instrument Configuration
and then quietly exits (with return code of 0). I have checked that each
of the SED files referenced in the instance catalog exist and are readable.
However, I fear there are some limits within phoSim that are causing this
problem. For the instance catalog named phosim_input_484933.txt.gz,
Total lines = 326110
Total SEDs = 326090
Total good SEDs = 326090
Total bad SEDs = 0
Perhaps @johnrpeterson https://github.com/johnrpeterson knows if these
numbers look problematic?
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188480970
.

@TomGlanzman Can you provide the path to the instance catalog and output files for that run?

An example command that demonstrated the problem is:
time /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/phoSim/3.4.2/phosim.py /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_input/20160219/phosim_input_484933.txt.gz -s R22_S11 --sed /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/SEDs/2016.01.26 -w /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/work -o /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output
The full log file is here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/log.jsp?pi=40601577
These failed jobs did not produce output.

Are these from the nersc InstanceCatalogs?
Thanks,
Rahul
On Wed, Feb 24, 2016 at 3:08 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

An example command that demonstrated the problem is:
time
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/phoSim/3.4.2/phosim.py
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_input/20160219/phosim_input_484933.txt.gz
-s R22_S11 --sed
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/SEDs/2016.01.26 -w
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/work
-o
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output
The full log file is here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/log.jsp?pi=40601577
These failed jobs did not produce output.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188504132
.

Hi @rbiswas4 , no, these are the 17 obtained from your UW server via 'crawl'.
Are new data available at NERSC?  If so, please send me the coordinates and I can give them a try.  Are there changes between the NERSC and other data?

If you can run and reproduce this manually:
I think this error is happening in raytrace.
I vaguely remember something like this happening to me before and tracking it down this way. I believe in my case it was related to some files not being there or some output path not existing but I'm not sure.

There should not be any changes from what I was putting out earlier and the
NERSC ones (just more).  These are in the project space I discussed, but I
am still pushing things that don't seem to be there:
/project/projectdirs/m1727/edison/twinkles_InstanceCatalogs/InstanceCatalogs
Thanks,
Rahul
On Wed, Feb 24, 2016 at 3:23 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Hi @rbiswas4 https://github.com/rbiswas4 , no, these are the 17
obtained from your UW server via 'crawl'.
Are new data available at NERSC? If so, please send me the coordinates and
I can give them a try. Are there changes between the NERSC and other data?
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188512362
.

That run stops right after it tries to run instrument.  However, if I run on an ungzipped instance catalog file, raytrace executes seemingly without any problem, so I think that's the issue.  I don't see the physics override file being used in the log of that run, if so, then that should also be updated.

Hi Rahul,
That directory is read protected:
drwx------ 3 rbiswas rbiswas 1048576 Feb 24 14:35 InstanceCatalogs/
You will also need to change the group ownership of the directory (and contents).
Thanks,

Tom

On 02/24/2016 03:42 PM, rbiswas4 wrote:

There should not be any changes from what I was putting out earlier and the
NERSC ones (just more). These are in the project space I discussed, but I
am still pushing things that don't seem to be there:
/project/projectdirs/m1727/edison/twinkles_InstanceCatalogs/InstanceCatalogs
Thanks,
Rahul
On Wed, Feb 24, 2016 at 3:23 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Hi @rbiswas4 https://github.com/rbiswas4 , no, these are the 17
obtained from your UW server via 'crawl'.
Are new data available at NERSC? If so, please send me the coordinates and
I can give them a try. Are there changes between the NERSC and other data?
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188512362
.

—
Reply to this email directly or view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188520732.

@TomGlanzman Sorry about that. fixed this: it is now world readable and is owned by the lsst group.

Thanks to @jchiang87 and @cwwalter for your suggestions.  An unzipped but stripped down instance catalog has been running for the past 2+ hours so this looks promising.  A second test is also running with the full, uncompressed instance catalog.
The first test with the stripped down instance catalog looks like this:
Found 1 source(s) for chip R22_S10.
Found 2 source(s) for chip R22_S11.
Found 1 source(s) for chip R22_S21.
While the 2nd (full) test has:
Found 18020 source(s) for chip R22_S00.
Found 48237 source(s) for chip R22_S01.
Found 19171 source(s) for chip R22_S02.
Found 47788 source(s) for chip R22_S10.
Found 65551 source(s) for chip R22_S11.
Found 46510 source(s) for chip R22_S12.
Found 18199 source(s) for chip R22_S20.
Found 46620 source(s) for chip R22_S21.
Found 17623 source(s) for chip R22_S22.
The run is limited to producing output only for R22_S11.  It may be a bit curious that the first run, with only two sources, has already been running for >2 hours?

@TomGlanzman The time will really be dominated by what sort of sky background model you are using.  It can be really long if you are using full non-optimized background and even longer if the moon is in the sky.
For the "The run is limited to producing output only for R22_S11", I found a subtle bug which I reported a while ago.  If you simply comment out sources in the instance catalog (as opposed to deleting them) PhoSim will think you want those chips that the sources are on. As I recall you won't get sources on those chips, but you will still get them with sky background.

On Feb 25, 2016, at 2:04 PM, Chris Walter <notifications<EMAIL_ADDRESS>wrote:
@TomGlanzmanhttps://github.com/TomGlanzman The time will really be dominated by what sort of sky background model you are using. It can be really long if you are using full non-optimized background and even longer if the moon is in the sky.
For the "The run is limited to producing output only for R22_S11", I found a subtle bug which I reported a while ago. If you simply comment out sources in the instance catalog (as opposed to deleting them) PhoSim will think you want those chips that the sources are on. As I recall you won't get sources on those chips, but you will still get them with sky background.
but not if you do -s R22_S11, which is what Tom is doing.
—
Reply to this email directly or view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-188933014.

The second full test generated the following non-fatal messages:
[...]
Installing Universe.
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Error in SED file; 0 value at 500 nm
Creating Air.
Generating Turbulence.
Building Optics.
Placing Obstructions.
Perturbing Design.
Electrifying Devices.
Contaminating Surfaces.
Diffracting.
[...]
How may one discover which SED file(s) are at fault?

Hi @rbiswas4 .  Can you change the NERSC directory permission to include 'x' both for group and world?  This will allow one to 'cd' into the directory.

OK Done.

Five test runs are now complete and the data are available via the data catalog.  Navigate to http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-DESC then to /LSST-DESC/Twinkles/Twinkles-phoSim/singleVisit.  The five complete runs include task streams 0,1,2,5,13.  The remaining 12 jobs are still running at this time.  Note that stream 0 (corresponding to Opsim_obhistid 484933) had its instance catalog stripped down to four sources; at some point, I should rerun that one with the full catalog.
For example, the full list of phoSim data products for stream 0 include:
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/centroid_lsst_e_484933_R22_S11_E000.txt
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C00_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C01_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C02_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C03_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C04_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C05_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C06_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C07_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C10_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C11_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C12_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C13_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C14_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C15_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C16_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C17_E000.fits.gz
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_e_484933_f4_R22_S11_E000.fits.gz
Highest priorities now are: 1) pushing more visits through phoSim; and, 2) devising a new way to handle the instance catalogs (uncompressed) and the associated ~100 MB of SED files.

Progress on the phoSim runs may be monitored using the Pipeline workflow web page:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40601107

OK so this is (slightly) off topic but given all of the "I can't see your directory stuff" above, I noticed at NERSC the umask is set by default to 27.
That gives you something that looks like this (in your home directories)
-rw-r----- 1 cwalter cwalter
This means that no one else in the lsst group can look at the files since by default your "user" group is used here.
On
/global/project/projectdirs/lsst
others can look at it as the the directory is set to to group lsst and the permissions are
drwxrwsr-x 29 nugget lsst
The "s" means that all of the files below will be set to group lsst.  So you get things like this:
-rw-r----- 1 cwalter lsst
So other lsst group members should see it.  I'm not sure why the edison space was having a problem.. I assume it is setup the same way.
But, my question is shouldn't we all have our umask set to say "2" so we get something that looks like this in our home directories?
002	rw-rw-r–
So we can see and copy each others work?  Or is there some NERSC prohibition against this?  Perhaps @nugent68 can comment?

Thanks @cwwalter, great if we can get that cleared up quickly: group readability is pretty important.
Nice going, @TomGlanzman! I was able to download both lsst_e_495735_f2_R22_S11_E000.fits.gz and centroid_lsst_e_495735_R22_S11_E000.txt.  Here's a pretty picture:

This is great! I might missing it, but is there a way to easily get either
the obsHIstID or the MJD/ filter of the files that are being completed?
That might help us to figure out what is missing.
Thanks,
Rahul
On Fri, Feb 26, 2016 at 9:22 AM, Phil Marshall<EMAIL_ADDRESS>wrote:

Thanks @cwwalter https://github.com/cwwalter, great if we can get that
cleared up quickly: group readability is pretty important.
Nice going, @TomGlanzman https://github.com/TomGlanzman! I was able to
download both lsst_e_495735_f2_R22_S11_E000.fits.gz and
centroid_lsst_e_495735_R22_S11_E000.txt. Here's a pretty picture:
[image: lsst_e_495735_f2_r22_s11_e000_logscale]
https://cloud.githubusercontent.com/assets/710903/13359716/6a9b7c54-dc6a-11e5-986b-53d924a1211a.jpg
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-189380130
.

Hi @rbiswas4 .  This is what we discussed yesterday: the obshistid is part of the file names, while the filter number/letter is not.  For the moment, Tony will used your CSV file to extract the filter given the obshistid.  I am thinking it would be good to define {obshistid,MJD,filter} as stream (data catalog folder) metadata.  If agreed, I'll put that on my to-do list.

That sounds good. And to remind myself of how Tom was suggesting we design future runs:
1.the phosim input catalog (phosim instance : PS_input_obsHistID_mjd_filtername.txt
- obsHistID is an integer formatted to string without padding
- mjd is the MJD formatted as a float with four places of decimal

filtername is a string 'u', 'g'. 'r', 'i', 'z'

The phosim input catalog contains the filename of the spectra as spectra/obsHistID/spec_obsHistID_sourceID_mjd_filtername.dat
where the sourceID is a unique identifier of the SN.

I will assign myself such an issue and ensure this is easily done next time.

Just in case people don't know (I can't tell where the CSV file is coming from) the finer, obhistID, mjd etc are all recorded in the fitsheaders of the output file.
Here are a few lines from running 'fitsheader' on one of my output files:
OBSID   =              1000000 / Opsim observation ID
DATASET = '1000000 '           / Opsim observation ID
TAI     =     49552.2999131944 / International Atomic Time scale
MJD-OBS =     49552.2999131944 / Modified Julian date (also TAI)
..
FILTNM  =                    2 / Filter (0=u, 1=g, 2=r, 3=i, 4=z, 5=y)
FILTER  = 'r       '           / Filter

Thanks @rbiswas4 that looks fine.
@cwwalter Right, but Tony needs that data to find the phoSim output FITS files in the dataCatalog.  Eventually, there will be metadata for a proper query.  Rahul's csv file is a stop-gap.

Hi @SimonKrughoff @jchiang87, I am trying to feed Tom's recently produced phosim images through my DM "cookbook" workflow, but I am running into problems at the first step (ingestImages.py). I am getting the
RuntimeError: No mapper provided and no _mapper available
I notice that the files which I was using before (from Simon) have a _mapper file in the input directory, containing:
lsst.obs.lsstSim.LsstSimMapper
which is (of course) not there in Tom's output directory. Is there any way that the mapper can just be specified on the command line as an option to the ingestImages.py script? Or some other workaround better than writing the _mapper into Tom's output or copying/linking all of the files?

Hi @SimonKrughoff @jchiang87, another question, if I work around the _mapper problem by just adding a _mapper file to each of Tom's output directories, then I can process each of Tom's visits separately through processEImage. When I do this in the natural way I end up with one output directory for each visit, containing:
lrwxrwxrwx 1 tonyj ey 46 Feb 27 16:01 _parent -> /nfs/farm/g/lsst/u1/users/tonyj/work/999/input
drwxr-xr-x 3 tonyj ey  3 Feb 27 16:01 calexp
drwxr-xr-x 2 tonyj ey  4 Feb 27 16:01 config
drwxr-xr-x 3 tonyj ey  4 Feb 27 16:02 deepCoadd
drwxr-xr-x 3 tonyj ey  3 Feb 27 16:02 deep_makeCoaddTempExp_metadata
drwxr-xr-x 3 tonyj ey  3 Feb 27 16:01 icSrc
drwxr-xr-x 3 tonyj ey  3 Feb 27 16:01 processEimage_metadata
drwxr-xr-x 2 tonyj ey  4 Feb 27 16:01 schema
drwxr-xr-x 3 tonyj ey  3 Feb 27 16:01 src
Now if I want to run the assembleCoadd stage I assume I have to run it simultaneously on all of the (separate) output directories -- but I do not know how to do that? Alternatively can I run all of the individual visits through processEImage separately and have it write all of the output to the same output directory (I assume not since that would presumably clobber the _parent link)? Yet again I could try running ingestImages on each visit separately and write the ingested images to a single output directory, but again the existence of a _parent link in that directory suggests this may not be the way to go?
Any suggestions?

I think adding the _mapper file in each directory is unavoidable, but @SimonKrughoff would know better.
I would run ingestImages.py and have the processed phosim files directed to the same input_data directory.  Then subsequent processEImage.py runs would all take that same input_data directory as its argument, if you are running a bunch in parallel batch jobs.   The _parent file in input_data shouldn't be needed after ingestImages.py (at least that's my guess).

A Mapper is necessary for the Butler in order to find the proper datasets, more or less. The Butler is generic get/put framework for data and has no default notion of data, files, or file types, so you have to provide it with some necessary can either provide the Butler a mapper object or it will look for a way to construct that object through the _mapper file. In this case, I believe it's the lsst.obs.lsstSim.LsstSimMapper class.
_parent is a way of telling the Butler this is a child repository and it may look for an _mapper higher in the parent

Right. The initial question was whether there was a way of specifying the mapper class by some means other than via that _mapper file, e.g., a command line option.  I'm fairly certain the answer is no, but it would be useful if there were some other way.

I am following @jchiang87 suggestion, and ingesting all files into the same directory, after copying a _mapper file into the phosim output directory. This actually fails with:
RuntimeError: Output repository path '/nfs/farm/g/lsst/u1/users/tonyj/work/999/input/_parent' already exists and differs from input repository path '/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000002/output'
but I am going to try just deleting the old _parent link and see how that goes.

Thanks all for helping Tony out.  Just to condense into a single post.  A data repository is simply a directory with a _mapper file in it containing the class name to use for mapping data to/from disk.
In the case of ingest, the repository is the directory specified as the first argument to ingestImages.py.  The files to ingest are specified as a glob or list of files on the command line.
Tasks (of which ingestImage.py is one) try not to overwrite data if it can be helped, so by default if the repository differs from what is expected (the configs don't match or a _parent repository is different) it will complain and fail.  If you know what you are doing, you can add the --clobber-config option to essentially ignore these checks.

Hi @SimonKrughoff, my trick of deleting the _parent directory if it exists does not work well, because although the file may not exist when the job starts it may have been created when the job goes to write its output (if multiple jobs are running at once). Needless to say I do not know what i am doing, but I tried the --clobber-config option anyway but it did not appear to help:
 ++ ingestImages.py /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C00_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C01_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C02_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C03_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C04_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C05_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C06_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C07_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C10_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C11_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C12_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C13_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C14_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C15_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C16_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_a_484933_f4_R22_S11_C17_E000.fits.gz /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/lsst_e_484933_f4_R22_S11_E000.fits.gz --mode link --output /nfs/farm/g/lsst/u1/users/tonyj/twinkles/run1/ingest/ --doraise --clobber-config : Loading config overrride file '/nfs/farm/g/desc/u1/Twinkles/obs_lsstSim/config/ingest.py' : Config override file does not exist: '/nfs/farm/g/desc/u1/Twinkles/obs_lsstSim/config/lsstSim/ingest.py' : input=/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output : calib=None : output=/nfs/farm/g/lsst/u1/users/tonyj/twinkles/run1/ingest Traceback (most recent call last):   File "/nfs/farm/g/desc/u1/Twinkles/obs_lsstSim/bin/ingestImages.py", line 29, in <module>     SimIngestTask.parseAndRun()   File "/nfs/farm/g/desc/u1/LSST_Stack/lsstsw/stack/Linux64/pipe_tasks/2015_10.0-12-g9f70a69/python/lsst/pipe/tasks/ingest.py", line 363, in parseAndRun     args = parser.parse_args(config)   File "/nfs/farm/g/desc/u1/LSST_Stack/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a+2/python/lsst/pipe/base/argumentParser.py", line 397, in parse_args     outputRoot = namespace.output,   File "/nfs/farm/g/desc/u1/LSST_Stack/lsstsw/stack/Linux64/daf_persistence/2015_10.0-5-g4063539/python/lsst/daf/persistence/butler.py", line 138, in __init__     self.mapper = cls(root=root, **mapperArgs)   File "/nfs/farm/g/desc/u1/Twinkles/obs_lsstSim/python/lsst/obs/lsstSim/lsstSimMapper.py", line 53, in __init__     super(LsstSimMapper, self).__init__(policy, policyFile.getRepositoryPath(), **kwargs)   File "/nfs/farm/g/desc/u1/LSST_Stack/lsstsw/stack/Linux64/daf_butlerUtils/2015_10.0+11/python/lsst/daf/butlerUtils/cameraMapper.py", line 203, in __init__     "input repository path '%s'" % (dst, src) RuntimeError: Output repository path '/nfs/farm/g/lsst/u1/users/tonyj/twinkles/run1/ingest/_parent' already exists and differs from input repository path '/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output'

@tony-johnson I don't think the ingestImages script is intended to be thread safe.  I had assumed we'd just ingest in one serial job.
Why are you using two different repositories?  I think it would be much better to use the output repository as the only repository.  I.e.
$> ingestImages.py /nfs/farm/g/lsst/u1/users/tonyj/twinkles/run1/ingest/ /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_output/000000/output/*.fits.gz --mode link

I think that may get around the _parent problem too.

@SimonKrughoff -- OK, I can change the ingest images to run serially, it was just convenient to ingest each visit in a separate job since I already had one job per visit for running processEimage. I don't think it will help with the _parent issues, since each visit still comes from a different phosim directory.
I had always been using a separate directory for the output of the ingest and the output of the rest of the processing, probably for no good reason,  I can change it to use the same directory.
If you have time for a quick bluejeans meeting tomorrow let me know, I have several jobs which are failing, such as:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/run.jsp?pi=40602293
and
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/run.jsp?pi=40602312
I am sure you can help me fix whatever I am doing wrong.

@tony-johnson I'm free at 1PM Pacific if you want to chat.  Regarding the _parent issue, I think that will be solved by not using the phosim output directory as the input repository to ingestImages.py.  I think I can explain on the phone.

1pm is good, we can use the same bluejeans number that we use for the Friday meetings (https://bluejeans.com/256850342/).

I'd like to listen in on this discussion.

Everyone is welcome!

@tony-johnson I am so sorry that I missed this meeting.  I will likely be free in about 30 min if you are still willing to meet  today.  What do you think?

Hi Simon, no problem, yes OK, should we say 3pm to be specific?
Tony
On 02/29/2016 02:09 PM, SimonKrughoff wrote:

@tony-johnson https://github.com/tony-johnson I am so sorry that I
missed this meeting. I will likely be free in about 30 min if you are
still willing to meet today. What do you think?
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190419553.

Can we try 2:30?  I have another thing at 3.
As an aside, I may have given you the wrong astrometric reference catalog.  Let me look around.

I have new astrometric reference files for you to use.  Can you try these out?
https://lsst-web.ncsa.illinois.edu/~krughoff/data/and.tar.gz

Just saw  this, will call in to Bluejeans now.
Tony
On 02/29/2016 02:12 PM, SimonKrughoff wrote:

Can we try 2:30? I have another thing at 3.
As an aside, I may have given you the wrong astrometric reference
catalog. Let me look around.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190421541.

With new astrometric files, and hints on how to run the ingest of the phosim output from @SimonKrughoff I have now been able to process the visits that were complete as of last night -- except the forced photometry which I will do today.

Hooray Tony! Nice going. @jbkalmbach, your wish for light curves for The
Monitor is about to come true :-)
On Tue, Mar 1, 2016 at 8:29 AM, Tony Johnson<EMAIL_ADDRESS>wrote:

With new astrometric files, and hints on how to run the ingest of the
phosim output from @SimonKrughoff https://github.com/SimonKrughoff I
have now been able to process the visits that were complete as of last
night -- except the forced photometry which I will do today.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190796488
.

Two quick updates.

Of the original 17 phoSim runs submitted five days ago, five were still running this morning.  The longest public queue at SLAC is 7200 min (5 days), and the first time-out has just occurred.  I expect the remaining 4 to fail soon.  This does not bode well for the 1600 needed runs.  The instance catalog specifies 30s exposures (if I interpret recent emails correctly).  Would reducing that to 15s be a reasonable approach?  I am assuming it would also significantly reduce the needed CPU time.  View real-time status here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40601107

A 2nd generation phoSim task, Twinkles-phoSimII, is being commissioned which handles the mass of instance catalogs and SED files as delivered by Rahul, et al at UW.  One hundred jobs have been submitted to test the new mechanism and I think they are good.  Of course, the above issue remains in force so many of these are destined for the 5-day ash bin.  I will await suggestions on reducing execution time before submitting more visits.  View real-time status here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40602489

Excellent! Let's diagnose these week-long epics. @danielsf @rbiswas
@SimonKrughoff what sky brightness and filter is being requested of the
problem visits? Does the view at
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40601107
give
you enough information to track down the visit information?
On Tue, Mar 1, 2016 at 2:11 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Two quick updates.

Of the original 17 phoSim runs submitted five days ago, five were
still running this morning. The longest public queue at SLAC is 7200 min (5
days), and the first time-out has just occurred. I expect the remaining 4
to fail soon. This does not bode well for the 1600 needed runs. The
instance catalog specifies 30s exposures (if I interpret recent emails
correctly). Would reducing that to 15s be a reasonable approach? I am
assuming it would also significantly reduce the needed CPU time. View
real-time status here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40601107
2.
A 2nd generation phoSim task, Twinkles-phoSimII, is being commissioned
which handles the mass of instance catalogs and SED files as delivered by
Rahul, et al at UW. One hundred jobs have been submitted to test the new
mechanism and I think they are good. Of course, the above issue remains in
force so many of these are destined for the 5-day ash bin. I will await
suggestions on reducing execution time before submitting more visits. View
real-time status here:
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40602489
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190929641
.

Jim pointed out that the Moon is quite high in some of these runs.  Here are the values of moonalt in the 17 instance catalogs that Tom started with.
phosim_input_484933.txt:Opsim_moonalt -47.1439987
phosim_input_488076.txt:Opsim_moonalt 8.99858865
phosim_input_488756.txt:Opsim_moonalt 20.9496861
phosim_input_490132.txt:Opsim_moonalt 36.4522052
phosim_input_490142.txt:Opsim_moonalt 34.8007116
phosim_input_490162.txt:Opsim_moonalt 31.888539
phosim_input_490187.txt:Opsim_moonalt 28.3278928
phosim_input_490189.txt:Opsim_moonalt 23.2562423
phosim_input_490810.txt:Opsim_moonalt 44.2148093
phosim_input_490820.txt:Opsim_moonalt 43.4705944
phosim_input_490840.txt:Opsim_moonalt 42.0164275
phosim_input_490844.txt:Opsim_moonalt 41.5249443
phosim_input_490867.txt:Opsim_moonalt 36.7402884
phosim_input_495735.txt:Opsim_moonalt -0.241387119
phosim_input_495743.txt:Opsim_moonalt 4.74368947
phosim_input_495753.txt:Opsim_moonalt 6.14067517
phosim_input_495799.txt:Opsim_moonalt 11.4969138
I'd guess that a quick speed up would be to put the Moon well below the horizon for each run.  This would presumably override what is in the Opsim database for the selected observations, but especially for Run 1, I don't think that is important.

Sounds good to me - what is the Moon altitude in the 5 epics?
I am reminded of this verse:
"The stars are not wanted now: put out every one;
Pack up the moon and dismantle the sun;
Pour away the ocean and sweep up the wood.
For nothing now can ever come to any good."
Sigh.
On Tue, Mar 1, 2016 at 3:55 PM, Seth Digel<EMAIL_ADDRESS>wrote:

Jim pointed out that the Moon is quite high in some of these runs. Here
are the values of moonalt in the 17 instance catalogs that Tom started with.
phosim_input_484933.txt:Opsim_moonalt -47.1439987
phosim_input_488076.txt:Opsim_moonalt 8.99858865
phosim_input_488756.txt:Opsim_moonalt 20.9496861
phosim_input_490132.txt:Opsim_moonalt 36.4522052
phosim_input_490142.txt:Opsim_moonalt 34.8007116
phosim_input_490162.txt:Opsim_moonalt 31.888539
phosim_input_490187.txt:Opsim_moonalt 28.3278928
phosim_input_490189.txt:Opsim_moonalt 23.2562423
phosim_input_490810.txt:Opsim_moonalt 44.2148093
phosim_input_490820.txt:Opsim_moonalt 43.4705944
phosim_input_490840.txt:Opsim_moonalt 42.0164275
phosim_input_490844.txt:Opsim_moonalt 41.5249443
phosim_input_490867.txt:Opsim_moonalt 36.7402884
phosim_input_495735.txt:Opsim_moonalt -0.241387119
phosim_input_495743.txt:Opsim_moonalt 4.74368947
phosim_input_495753.txt:Opsim_moonalt 6.14067517
phosim_input_495799.txt:Opsim_moonalt 11.4969138
I'd guess that a quick speed up would be to put the Moon well below the
horizon for each run. This would presumably override what is in the Opsim
database for the selected observations, but especially for Run 1, I don't
think that is important.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190974016
.

Well put.  The run time will also depend on how far the Moon is from the particular pointing direction and on the filter.  I was just advocating packing up the Moon entirely.
The information below is from the instance catalogs in /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_input/20160219
I'll look into correlating these with run times unless someone beats me to it.

phosim_input
moon_alt (deg)
dist2moon (deg)
filter

484933
-47.1439987
134.069431
4

488076
8.99858865
92.0526026
0

488756
20.9496861
82.1848942
0

490132
36.4522052
63.6410579
2

490142
34.8007116
63.6124673
1

490162
31.888539
63.5607865
3

490187
28.3278928
63.4951829
4

490189
23.2562423
63.3966341
5

490810
44.2148093
47.2872955
2

490820
43.4705944
47.2921083
1

490840
42.0164275
47.2996713
3

490844
41.5249443
47.3018486
4

490867
36.7402884
47.3157715
5

495735
-0.241387119
86.9740193
2

495743
4.74368947
87.1317546
1

495753
6.14067517
87.1752421
3

495799
11.4969138
87.3394518
5

Go for it. If you're right, the epics are
490810,  490820,  490840,  490844,  490867
I wonder if a dist2moon of 47 is particularly extreme?
On Tue, Mar 1, 2016 at 4:26 PM, Seth Digel<EMAIL_ADDRESS>wrote:

Well put. The run time will also depend on how far the Moon is from the
particular pointing direction and on the filter. I was just advocating
packing up the Moon entirely.
The information below is from the instance catalogs in
/nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim/phosim_input/20160219
I'll look into correlating these with run times unless someone beats me to
it.
phosim_input moon_alt (deg) dist2moon (deg) filter
484933 -47.1439987 134.069431 4
488076 8.99858865 92.0526026 0
488756 20.9496861 82.1848942 0
490132 36.4522052 63.6410579 2
490142 34.8007116 63.6124673 1
490162 31.888539 63.5607865 3
490187 28.3278928 63.4951829 4
490189 23.2562423 63.3966341 5
490810 44.2148093 47.2872955 2
490820 43.4705944 47.2921083 1
490840 42.0164275 47.2996713 3
490844 41.5249443 47.3018486 4
490867 36.7402884 47.3157715 5
495735 -0.241387119 86.9740193 2
495743 4.74368947 87.1317546 1
495753 6.14067517 87.1752421 3
495799 11.4969138 87.3394518 5
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190986600
.

@sethdigel ,  The bulk of instance catalogs (used with the new Twinkles-phoSimII task) are located here:  /nfs/farm/g/desc/u1/data/Twinkles/phoSim/Run1_InstCats_SEDs
There are 1227 {instance catalog, SED tarball} pairs.

The three jobs still running at this moment are using phosim instance catalogs 490189, 490810, and 490840.  Others may have timed out.
Yes, for Run 1 we'd need to move the Moon in a lot of instance catalogs, if that is what we want to do.  I do not know if phosim requires consistency (the catalog files also include the RA and Dec of the Moon, in addition to moon_alt and dist2moon) or whether setting moon_alt to -90 would do it.

To be safe, we could choose a safe (Ra,Dec,Moonalt,dist2moon) tuple from
the list we have. The only thing inconsistent then would be the MJD, and I
bet that wouldn't cause problems.
Another approach is just to skip them and move on - we have 1227 visits to
get through, and I doubt we'll reach the end of the list by next week
anyway. We could just keep track of the epics, and chalk them up to
experience.
On Tue, Mar 1, 2016 at 4:51 PM, Seth Digel<EMAIL_ADDRESS>wrote:

The three jobs still running at this moment are using phosim instance
catalogs 490189, 490810, and 490840. Others may have timed out.
Yes, for Run 1 we'd need to move the Moon in a lot of instance catalogs,
if that is what we want to do. I do not know if phosim requires consistency
(the catalog files also include the RA and Dec of the Moon, in addition to
moon_alt and dist2moon) or whether setting moon_alt to -90 would do it.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-190996441
.

Just to comment on this quickly.  There is also a moon phase input, but setting the moon well below the horizon should deal with this.  Typically when I set the moon down, I set moon_alt to -90 and dist2moon to 180.  As far as I know, phosim does no checking of consistency of observation parameters, so I don't think you need to worry about that if you decide to set the moon to be below the horizon.

Well, dist2moon depends on the pointing direction, but yes, we could find something that works.  I've updated the table to include the phase of the moon and the CPU time.  The tough cases seem to be in g band.
Tonight I could look through the big set of instance catalogs to see what the distributions of dist2moon, moonalt, etc. are.

What I'm saying is that phosim doesn't care if the input parameters are self consistent so you can set the moon parameters to whatever you want.  The safest (and maybe most self documenting thing) is to put the moon as far down and as far away from your observation as you can.

Here is the distribution of moon altitudes in the 1227 instance catalogs for Twinkles-phoSimII.  The distribution is very similar for filters 1-5.  For filter 0 (u band), the distribution is skewed to negative moon altitudes.

Here also, for what it is worth, is a plot of moon phase vs. moon altitude.

I'm not sure whether to trust the dist2moon values, but the minimum is a bit more than 45 deg.  I don't know what rule Opsim was following, but at least it apparently was not letting the moon get any closer than that (which seems awfully close).
My vote would be for just putting the moon at -90 deg altitude, and not, say, trying to cherry pick the instance catalogs with the most favorably placed moon locations.

I would suggest also setting dist2moon to 180.  I'm not sure how those two values are used in the sky background calculation.

Sure, and maybe set moonphase to 0 too for good measure.  I do not think that rewriting the files with these changes (only) would be difficult.  I'd volunteer to make new versions, unless Rahul prefers to regenerate them, or maybe Tom has a slick way to do it in bulk.  The compressed instance catalog files take up about 20 Gbyte, not counting the SED tar files, and the instance catalog and SED files may need to all be in the same directory, so it could be a bit fiddly.

all you have to do is set the moon below the horizon (say -90).  and the lunar flux goes to 0.  you don’t have to worry about consistency, if you don’t want to.
BTW, it is approximately true in real life that the lunar flux drops very rapidly once it goes below the horizon.
john
On Mar 2, 2016, at 3:27 AM, Seth Digel <notifications<EMAIL_ADDRESS>wrote:
Sure, and maybe set moonphase to 0 too for good measure. I do not think that rewriting the files with these changes (only) would be difficult. I'd volunteer to make new versions, unless Rahul prefers to regenerate them, or maybe Tom has a slick way to do it in bulk. The compressed instance catalog files take up about 20 Gbyte, not counting the SED tar files, and the instance catalog and SED files may need to all be in the same directory, so it could be a bit fiddly.
—
Reply to this email directly or view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191125884.

Wednesday morning update:
I went ahead and submitted a total of 160 phoSim jobs to the workflow and as of 07:50 this morning, 91 had completed successfully.  It could be that jobs with the new task are running faster?  If so, what is different?  The main changes to structure include: 1) instance catalog is uncompressed and stored in lustre (possibly faster disk access); 2) visit-specific SED files are presented only for the individual visit being processed - rather than a huge pool of all SED files being presented to all jobs.
There were eight phoSim failures, but these were all due to two batch hosts being rebooted.  There have, so far, been no phoSim crashes.
Data is accumulating in the dataCatalog folder:
/LSST-DESC/Twinkles/Twinkles-phoSimII,
http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-DESC
Not certain this attachment will show, but below is a plot of execution times.  These represent, of course, the short end of the distribution so will have to see if any stragglers persist past five days.

Stay tuned...

Another status update, I finished processing Tom's original 10 runs through forced photometry. The output is on SLAC unix in:
/nfs/farm/g/lsst/u1/users/tonyj/twinkles/run1/output
@jbkalmbach -- if you want to look at this data and SLAC is not a good location I can tar up this directory and put it somewhere you can download it. Uncompressed it is 2.5GB.
Do we have any simple diagnostics that can be run on the output as a sanity check?
I will now move on to processing the phosimII output as it appears.

Thanks @tony-johnson! I don't have a SLAC account so somewhere I can download it would be great. I do have access at NERSC if that's easier.

Let's try hard to use the Data Catalog web interface: we'll need it for the
Hack Day if we are to attract people to come play with our data. We have an
open issue on providing documentation and (fingers crossed) a python API...
Who has an example of this in action? Is it you, @jchiang87?
Love the embedded plot, Tom. I used my handheld calculator to find that
1000 minutes is just 17 hours so that's great! Just to check: did all 160
jobs have the Moon in the correct place?
John: I hope there are citizens following this thread and feeling reassured
that we too have to remind ourselves that it gets dark when the Moon goes
down :-)
On Wed, Mar 2, 2016 at 8:34 AM, Bryce Kalmbach<EMAIL_ADDRESS>wrote:

Thanks @tony-johnson https://github.com/tony-johnson! I don't have a
SLAC account so somewhere I can download it would be great. I do have
access at NERSC if that's easier.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191316669
.

@tony-johnson Yes.  I've working working exactly on questions of validating data sets (as has @SimonKrughoff in related but different ways) and would be happy to take a look at these.
I'm transferring the images and catalogs now and hope to have some results by this afternoon.

Great, thanks Michael! Can you please issue this validation task over on
the Monitor repo? Thanks!
On Wed, Mar 2, 2016 at 8:42 AM, Michael Wood-Vasey<EMAIL_ADDRESS>
wrote:

@tony-johnson https://github.com/tony-johnson Yes. I've working working
exactly on questions of validating data sets (as has @SimonKrughoff
https://github.com/SimonKrughoff in related but different ways) and
would be happy to take a look at these.
I'm transferring the images and catalogs now and hope to have some results
by this afternoon.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191319497
.

@drphilmarshall  I did not edit the instance catalogs: the moon is wherever Rahul, et al placed it.

@jbkalmbach -- for a quick solution I put the data here:
ftp://ftp.slac.stanford.edu/outgoing/tonyj/

Out of the 160 runs that Tom started, 97 have finished as of a few minutes ago.  Of these, 90 have negative moonalt and the average moonalt is -24 deg.
Of the 63 that are still running, 16 have negative moonalt, and the overall average moonalt is +17 deg.

Good, that makes sense. I guess we'll end up taking delivery of the dark
time first! :-)
On Wed, Mar 2, 2016 at 10:12 AM, Seth Digel<EMAIL_ADDRESS>wrote:

Out of the 160 runs that Tom started, 97 have finished as of a few minutes
ago. Of these, 90 have negative moonalt and the average moonalt is -24 deg.
Of the 63 that are still running, 16 have negative moonalt, and the
overall average moonalt is +17 deg.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191355411
.

Thanks @tony-johnson, I got the files.
Everybody else, stay tuned for some monitor updates.

In the hope that the current phoSim project configuration is good, I am starting to ramp up production with a goal of 1227 visits, but will monitor this issue for concerns if/when they arise.

Here is a plot of the CPU time vs. moon altitude for the 17 runs in Tom's first set plus the 97 that had finished as of earlier this morning.  (One of the original 17 was still running but must be on the verge of hitting the run limit, and I've counted it that way.)  The points are color coded by filter.  My impression is that there's no particular speed up in the second batch, but that set of observations in the second set included many with the moon below the horizon and it looks like a number of u band runs as well.  These are not representative of the Run 1 sample overall.

The dotted line is the run time limit.  Of the 1227 runs for Run 1 I'd estimate that 100-150 will hit the run limit.  The overall total CPU time will be dominated by the runs with the moon above about 10 deg.
I guess the sense of Twinkles is to go ahead.  I think that the runs that will hit the limit probably will be clustered together in (simuluated) MJD, so there should be reasonably long stretches with good DDF cadence.  Tom, if by any chance you want a list of the instance catalogs ordered by moon altitude, please let me know.

Agreed. Let's press on, and work with the dark time that we're given. At
the meeting, we'll need to discuss what to do about simulating bright time.
On Wed, Mar 2, 2016 at 12:11 PM, Seth Digel<EMAIL_ADDRESS>wrote:

Here is a plot of the CPU time vs. moon altitude for the 17 runs in Tom's
first set plus the 97 that had finished as of earlier this morning. (One of
the original 17 was still running but must be on the verge of hitting the
run limit, and I've counted it that way.) The points are color coded by
filter. My impression is that there's no particular speed up in the second
batch, but that set of observations in the second set included many with
the moon below the horizon and it looks like a number of u band runs as
well. These are not representative of the Run 1 sample overall.
[image: cpu_moonalt]
https://cloud.githubusercontent.com/assets/6035835/13473614/eee36cda-e06e-11e5-8524-4042acd84829.png
The dotted line is the run time limit. Of the 1227 runs for Run 1 I'd
estimate that 100-150 will hit the run limit. The overall total CPU time
will be dominated by the runs with the moon above about 10 deg.
I guess the sense of Twinkles is to go ahead. I think that the runs that
will hit the limit probably will be clustered together in (simuluated) MJD,
so there should be reasonably long stretches with good DDF cadence. Tom, if
by any chance you want a list of the instance catalogs ordered by moon
altitude, please let me know.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191405803
.

Thanks @sethdigel , nice plot!   Sure, if the moon-ordered list is handy, give me a pointer, it may be useful at some point.

~digel/lsst/obshistid_moonalt.txt has the instance catalogs obshistid values sorted by moonalt, which is the second number in each row.

@drphilmarshall   I'm not actually planning to look at lightcurves.  I'm just looking at photometric and astrometric repeatability of the output source catalogs using http://github.com/lsst/validate_drp
Would you like for the processCcd - first level processing to be discussed in a new Issue in the Twinkles repo or to have that discussion continued here?

@tony-johnson I've done some quick analysis of those 10 images and the output catalogs look sane.  More details tomorrow.

Got it. I'd say make a new issue in Twinkles - this one is not far from
being closed (since it was only to generate the images, not validate
them. Thanks Michael - looking fwd to seeing your results!
On Wed, Mar 2, 2016 at 7:13 PM, Michael Wood-Vasey<EMAIL_ADDRESS>
wrote:

@tony-johnson https://github.com/tony-johnson I've done some quick
analysis of those 10 images and the output catalogs look sane. More details
tomorrow.
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191558137
.

Morning update:
All 1227 phoSim runs of Twinkles Run 1 were submitted by last night.
On Thu, Mar 3, 2016 at 8:22 AM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Morning update:
All 1227 phoSim runs of Twinkles Run 1 were submitted by last night. At
present, 681 are complete and 546 still running.
http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=40602489
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191836045
.

As of a short time ago a total of 754 runs were complete (or had reached the run limit, counting Tom's original 17 runs, which are the only ones that have been running long enough to reach the 5-day limit).  Here's an updated plot of CPU time vs. moon altitude.  I removed duplicated runs (the 17 are a subset of the 1227).  The symbol size is scaled according to the phase of the moon.  The histogram at the bottom shows the distribution of moon altitudes among the jobs that are still running.  The scaling for the histogram is linear, the bin size is 4 deg, and the peak is at about 40.

Love this. I guess in bright time, dist2moon will come into play in
determining CPU time. For those visits with moonalt > 0, it'd be fun to
see the joint distribution of log CPU time, dist2moon and moonalt
together, per filter. If we really wanted to predict CPU time accurately,
we'd model that PDF I think.
There's still interesting structure in the dark time too though: each
filter seems to have "short" and "long" runs, with CPU times different by a
factor of a bit less than 2 in each case.
A fun hack for next Friday would be to put up all our visit metadata and
try and predict CPU time with a machine learning model.
On Thu, Mar 3, 2016 at 12:01 PM, Seth Digel<EMAIL_ADDRESS>wrote:

As of a short time ago a total of 754 runs were complete (or had reached
the run limit, counting Tom's original 17 runs, which are the only ones
that have been running long enough to reach the 5-day limit). Here's an
updated plot of CPU time vs. moon altitude. I removed duplicated runs (the
17 are a subset of the 1227). The symbol size is scaled according to the
phase of the moon. The histogram at the bottom shows the distribution of
moon altitudes among the jobs that are still running. The scaling for the
histogram is linear, the bin size is 4 deg, and the peak is at about 40.
[image: cpu_moonalt2]
https://cloud.githubusercontent.com/assets/6035835/13507417/569f54c4-e137-11e5-8af4-77616c674519.png
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191938375
.

Yes.  I forgot to mention that the CPU time did not actually seem to depend on dist2moon.  Here's the plot.

A dependence is more clear if you look only at runs with moonalt > 0:

Hi,
Kindly remove my email id from the list. It's not relevant to me. Seems
it's coming by mistake.
On Fri, Mar 4, 2016 at 2:50 AM, Seth Digel<EMAIL_ADDRESS>wrote:

Yes. I forgot to mention that the CPU time did not actually seem to depend
on dist2moon. Here's the plot.
[image: cpu_dist2moon2]
https://cloud.githubusercontent.com/assets/6035835/13509724/c90950d2-e141-11e5-9420-974b38196e5c.png
A dependence is more clear if you look only at runs with moonalt > 0:
[image: cpu_dist2moon2_posmoonalt]
https://cloud.githubusercontent.com/assets/6035835/13509845/87e40c22-e142-11e5-98f2-5c9da34747f0.png
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191969454
.

--
Thanks & Regards,
============X=============
Rajib Biswas.
Bangalore.
===========X===============

Apologies, Rajib - we have a team member whose GitHub username is
@rbiswas4, I guess we mis-spelled it at some point. Unfortunately I cannot
remove you from this issue thread, but you can  remove yourself by going to
the link below and clicking "unsubscribe". Apologies again!
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-192933301
Others: we need to close this issue as soon as we possibly can, to avoid
spamming Rajib even more. Tom, can you please close it with a very brief
comment as soon as the current PhoSim run is finished, and open new issues
to discuss any remaining problems? Thanks!
On Sun, Mar 6, 2016 at 8:59 AM, Rajib Biswas<EMAIL_ADDRESS>wrote:

Hi,
Kindly remove my email id from the list. It's not relevant to me. Seems
it's coming by mistake.
On Fri, Mar 4, 2016 at 2:50 AM, Seth Digel<EMAIL_ADDRESS>wrote:

Yes. I forgot to mention that the CPU time did not actually seem to
depend
on dist2moon. Here's the plot.
[image: cpu_dist2moon2]
<
https://cloud.githubusercontent.com/assets/6035835/13509724/c90950d2-e141-11e5-9420-974b38196e5c.png
A dependence is more clear if you look only at runs with moonalt > 0:
[image: cpu_dist2moon2_posmoonalt]
<
https://cloud.githubusercontent.com/assets/6035835/13509845/87e40c22-e142-11e5-98f2-5c9da34747f0.png
—
Reply to this email directly or view it on GitHub
<
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-191969454
.

--
Thanks & Regards,
============X=============
Rajib Biswas.
Bangalore.
===========X===============
—
Reply to this email directly or view it on GitHub
https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/137#issuecomment-192933301
.

I also changed the one comment, where my id was misspelt. I am hoping github will stop bugging him if it does not find his id on the thread?

This issue is complete (as of 3/13/2016).  Final numbers: 1227 phoSim runs attempted, 1124 completed successfully and 103 ran out of time.


Hacky FlashAttn 2 support

Enables flash attention v2 for model training.
TODOs that I definitely won't get to pre-ICML:

Do we want to retain support for Flash attn version 1?
retain support for KV-packed flash attention for inference (haven't checked what this entails yet)
retain Triton support (possibly check that the Triton updated FA2 kernel works with NeoX?)

Do we want to retain support for Flash attn version 1?
If I read the paper correctly, Flash V2 is comprised of two components.
One is further optimization of the Flash V2 attention and the second is improved work partitioning between warps and different parallelism (sequence parallelism if i'm correct)

While the former parts regard more fundamental aspects around memory access and attention calculations, the latter might be more hardware specific and difficult to support across different architectures or generations.
I haven't done a deep dive into flash v2 code, (and tbh, I will likely never) but if these two are neatly separated, it might be a good idea to fully move forward with the first part while leaving the second part for later implementation or selection.

I think you're right! It probably makes sense to retain Flash v1 support and use that if a user has installed a pre-2.0 version of the package, because Flash v2 still doesn't support Turing GPUs.

Using this PR caused a training run to fail when computing validation loss. I successfully completed a run with the same config before merging this change and upgrading flash-attn to 2.0.0.post1. This is the stack trace:
 samples/sec: 13.340 | iteration     1000/   10000 | elapsed time per iteration (ms): 9595.6 | learning rate: 2.989E-05 | approx flops per GPU: 193.1TFLOPS | lm_loss: 8.667910E-01 | loss scale: 256.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 2277.35 | backward: 7152.80 | backward-backward: 7152.69 | backward-allreduce: 0.00 | optimizer: 163.86 | batch generator: 3.39
Traceback (most recent call last):
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
Traceback (most recent call last):
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
Traceback (most recent call last):
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
            iteration = train(    iteration = train(    iteration = train(
iteration = train(
iteration = train(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
        evaluate_and_print_results(evaluate_and_print_results(

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
      File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
        evaluate_and_print_results(evaluate_and_print_results(evaluate_and_print_results(

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
        total_loss_dict = evaluate(total_loss_dict = evaluate(

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
            total_loss_dict = evaluate(total_loss_dict = evaluate(total_loss_dict = evaluate(

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
            iteration = train(loss = forward_step_fn(loss = forward_step_fn(

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
loss = forward_step_fn(loss = forward_step_fn(loss = forward_step_fn(  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
        outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)

              File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)    outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)

evaluate_and_print_results(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl

  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
    total_loss_dict = evaluate(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
    loss = forward_step_fn(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
    outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
                return forward_call(*input, **kwargs)        return forward_call(*input, **kwargs)return forward_call(*input, **kwargs)
return forward_call(*input, **kwargs)return forward_call(*input, **kwargs)

  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn

  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
return forward_call(*input, **kwargs)
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
                ret_val = func(*args, **kwargs)    ret_val = func(*args, **kwargs)    ret_val = func(*args, **kwargs)ret_val = func(*args, **kwargs)
ret_val = func(*args, **kwargs)
ret_val = func(*args, **kwargs)

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
        loss = self.module(*inputs, **kwargs)        loss = self.module(*inputs, **kwargs)
        loss = self.module(*inputs, **kwargs)loss = self.module(*inputs, **kwargs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
loss = self.module(*inputs, **kwargs)loss = self.module(*inputs, **kwargs)

  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl

  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
        return forward_call(*input, **kwargs)return forward_call(*input, **kwargs)

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
        return forward_call(*input, **kwargs)return forward_call(*input, **kwargs)

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
    iteration = train(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
    evaluate_and_print_results(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
    total_loss_dict = evaluate(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
    loss = forward_step_fn(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
        x = self.activation_checkpoint_func(x = self.activation_checkpoint_func(

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
    CheckpointFunction.apply(function, all_outputs, *args)
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
    CheckpointFunction.apply(function, all_outputs, *args)
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
        x = self.activation_checkpoint_func(x = self.activation_checkpoint_func(

    x = self.activation_checkpoint_func(  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
        x = self.activation_checkpoint_func(
inputs = layer(inputs)x = self.activation_checkpoint_func(
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
    CheckpointFunction.apply(function, all_outputs, *args)
      File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
CheckpointFunction.apply(function, all_outputs, *args)
    CheckpointFunction.apply(function, all_outputs, *args)  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
return forward_call(*input, **kwargs)

CheckpointFunction.apply(function, all_outputs, *args)  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
CheckpointFunction.apply(function, all_outputs, *args)  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward

  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    outputs = run_function(*inputs_cuda)
      File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    return super().forward(hidden_states, attention_mask), attention_mask
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
        return super().forward(hidden_states, attention_mask), attention_maskreturn super().forward(hidden_states, attention_mask), attention_mask

return super().forward(hidden_states, attention_mask), attention_mask    return super().forward(hidden_states, attention_mask), attention_mask  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
return super().forward(hidden_states, attention_mask), attention_mask  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward

Traceback (most recent call last):
return super().forward(hidden_states, attention_mask), attention_mask

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward

  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/train.py", line 27, in <module>
    pretrain(neox_args=neox_args)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 226, in pretrain
    iteration = train(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 849, in train
    evaluate_and_print_results(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 964, in evaluate_and_print_results
    total_loss_dict = evaluate(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 910, in evaluate
    loss = forward_step_fn(
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/training.py", line 371, in forward_step
    outputs = model((tokens, position_ids, attention_mask), neox_args=neox_args)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/za2514/DeeperSpeed/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/engine.py", line 1731, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 186, in forward
    x = self.activation_checkpoint_func(
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 713, in checkpoint
    CheckpointFunction.apply(function, all_outputs, *args)
  File "/home/za2514/DeeperSpeed/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 555, in forward
    outputs = run_function(*inputs_cuda)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/utils.py", line 163, in exec_func
    inputs = layer(inputs)
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 914, in forward
    return super().forward(hidden_states, attention_mask), attention_mask
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 858, in forward
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
wandb: Waiting for W&B process to finish... (failed 1).
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    attention_output, attention_bias = self.attention(
  File "/home/hailey81/miniconda3/envs/llmath_flashv2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 685, in forward
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
    context_layer = self.flash_attention(query_layer, key_layer, value_layer)
  File "/nobackup/scratch/usr/za2514/math-lm/gpt-neox/megatron/model/transformer.py", line 537, in flash_attention
    output = self.flash_kv_fn(
TypeError: 'NoneType' object is not callable
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: Run history:
wandb: runtime/flops_per_sec_per_gpu ▁▄▄▄▅▄▅▄▄▅▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▇▇▇▇██▇█▇███
wandb:        runtime/iteration_time █▅▅▅▄▄▄▅▄▄▅▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▃▂▂▂▂▁▁▂▁▂▁▁▁
wandb:       runtime/samples_per_sec ▁▄▄▄▅▄▅▄▄▅▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▇▇▇▇██▇█▇███
wandb:               timers/backward ▄██▇▇▇▇▇▇▇▇▇▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▅▃▄▃▁▂▂▁▂▂▂▁▁
wandb:     timers/backward-allreduce ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:      timers/backward-backward ▄██▇▇▇▇▇▇▇▇▇▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▅▃▄▃▁▂▂▁▂▂▂▁▁
wandb:        timers/batch generator █▄▂▃▃▃▃▄▄▁▄▄▂▂▂▃▂▃▅▄▅▃▂▁▂▂▂▂▂▃▃▂▃▃▂▅▄▂▂▂
wandb:                timers/forward █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:              timers/optimizer █▇▇▇▇▇▇▇▇▇▇▂▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▁▇▇▁▇▇▇▇▇▇▆▇▆
wandb:           train/learning_rate ▁▁▂▂▂▃▃▃▄▄▄▅▅▆▆▆▇▇▇█████████████████████
wandb:                 train/lm_loss █▄▆▆▅▅▅▄▄▄▅▃▃▄▄▃▄▄▃▅▂▂▄▅▁▃▄▄▃▅▆▂▂▃▃▃▃▄▄▃
wandb:              train/loss_scale ███████████▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▂▁▁▁▁▁▁▁▁▁▁▁▁
wandb:
wandb: Run summary:
wandb: runtime/flops_per_sec_per_gpu<PHONE_NUMBER>26595.9
wandb:        runtime/iteration_time 9.59555
wandb:       runtime/samples_per_sec 13.33951
wandb:               timers/backward 7.1528
wandb:     timers/backward-allreduce 0.0
wandb:      timers/backward-backward 7.15269
wandb:        timers/batch generator 0.00339
wandb:                timers/forward 2.27735
wandb:              timers/optimizer 0.16386
wandb:           train/learning_rate 3e-05
wandb:                 train/lm_loss 0.82912
wandb:              train/loss_scale 256.0

Ah oof oops. I for some reason thought that we only use the KV-packed flash attn in generation, making flash_kv_fn unused for our current use.
If I have time I can try and go fix this this week. A quick workaround I think should be to set self.flash_kv_func = flash_attn_varlen_kvpacked_func .

Yes, this can now be closed!


According to RFC 7159, application/json should not have charset

RFC 7159:

Added a note to the "IANA Considerations" clarifying the absence of a "charset" parameter for the application/json media type.

I know this effect can be achieved by flag "shouldAppendDefaultContentCharsetToContentTypeIfUndefined". My intention is change default behavior for application/json.
EncoderConfig: if application/json is removed from DEFAULT_CHARSET_FOR_CONTENT_TYPE, only text/json will remain. text/json is non-standard, undefined content type, so is't behavior is also undefined. I removed text/json and whole DEFAULT_CHARSET_FOR_CONTENT_TYPE. (text/json is abandoned concept, replaced by application/json)
UnicodeITest: I removed failing test. I debug the case and I found that response content type is set to "application/json; charset=ISO-8591-1". This content type is undefined, therefore expectation is undefined. I decide remove the test.

While I don't like to change default behavior, I know that it's wrong to supply a charset in application/json. So with that in mind I think it would be good to accept this.


Tolgee: Cannot read properties of undefined (reading 'scopes')

With the current Tolgee version 3.8.2 it is not possible to fetch the translations
I get the following messages:
In the notification popup:
Tolgee: Cannot read properties of undefined (reading 'scopes')

In the debug console:
Error loading translations
TypeError: Cannot read properties of undefined (reading 'scopes')
en

Do you have an idea what here happens?

Either way I will close this issue, since this repo is probably so outdated, that fixing this specific error wouldn't even make things work 🤫


DNNpackager ReadKey error in CI pipelines

If you try running DNNpackager in a CI pipeline it will throw an error at the console.ReadKey.
The pipeline environment fails here as there is no one there to enter any input.
I think we need to swallow this error in a try catch or find a better way...??  Goal being to enable it's usage in automated builds.

I do see if you pass more than 1 argument the ReadKey call gets bypassed...   I will try adding more than just the project as a param.

It looks like this issue only happens if you run a single param.  If you always include the a 2nd param it works perfectly.


Polish things up

I looked at the upstream library and I didn't see a keyword arg named input or input_ (or output either).

It is here: https://github.com/prompt-toolkit/python-prompt-toolkit/blob/5b3dd6dbbd72ac8323e749cc12e6aa2d7c7411af/src/prompt_toolkit/shortcuts/prompt.py#L409

@tannewt aaaahhh -- too bad they did that. :)


RESTEASY-2009

Restore / migrate to Undertow excluded tests in resteasy-spring

work in progress, don't merge yet.

this branch needs rebase, I'll create a new PR.


Create .gitignore

Gets rid of useless directories (dist and egg-info). These are managed by PyPi and should not be part of the git repository.

Cloning the repository also takes longer than expected because https://github.com/dsdanielpark/Bard-API/blob/main/assets/bardapi.gif is not stored using lfs

I intentionally left out gitignore, but I'm willing to sacrifice some completeness for brevity. Thank you for your contribution.


Update to android plugin beta 7; build tools to 26.0.2

The GithubBrowserSample is already at 7.
Feel free to update all of them to 7 and we can merge it.

@florina-muntenescu Done.

Everything has been updated and a version file was created that will be used by all projects.
PR is obsolete.


wip: multiasset treasury pallet

WIP: nothing to see yet

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2503542

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2503543

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2512790

With some exploration, it seems a simpler solution is making the treasury more generic to the payment approach. So, I'm closing this in favour of a different PR: https://github.com/paritytech/substrate/pull/13607/


Delete sub-chart value key from parent-chart

It seems that it is not possible to delete value key of sub-chart from parent-chart.
./values.yaml
name: parent
test.resources: null

./charts/test/value.yaml
name: child
resurces:
  requests:
    cpu: 500m
    memory: 2Gi

I would expect that after the merge the result will be:
./charts/test/value.yaml
name: child

Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:29:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"6e5b33a290a99c067003632e0fd6be0ead48b233", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}
GKE

There is a typo in "resurces" in charts/test/values.yaml. If this is really your file's content, that would explain the problem.


Directly load configuration values from environment variables instead of relying on Django settings

In Open Inwoner we are implementing setup_configuration and we were wondering if it is really necessary for every configuration variable to be defined as a setting.
From my understanding it was originally implemented using settings, because there could be certain settings that have defaults (which should also be used when running setup_configuration), but in the case of Open Inwoner we have several steps that don't use any defaults, leading to a lot of duplication because we have to define settings for everything: https://github.com/maykinmedia/open-inwoner/blob/develop/src/open_inwoner/conf/app/setup_configuration.py#L17
@annashamray is directly loading variables from the environment maybe something that could be possible next to using settings?

Sergei also proposed to remove settings layer cause it's unnecessary.
Having them both could be a solution. We can first check the setting and if it doesn't exist we can check the environment.
I don't mind this approach.

@sjoerdie I think this issue is what you mentioned would be preferable, right?


Split into logical components

Fixes #3

Suggested scope is complete: split the 3 actors (Initiate, TokenVerifier and TokenGenerator) and use them in App, TokenVerification API and TokenGeneration API. Purpose of the 3 is to demonstrate what responsibility lies in the respective components.
Code now illustrates a security-improvement more clearly which is that App (Client.Console) needs to know the public key, TokenVerification API needs the private key and the TokenGeneration API needs both private and public keys.


Cloudflare Pages Deploy Error

14:18:50.472	(For a CapTP with native promises, see @endo/eventual-send and @endo/captp)
14:18:51.210	➤ YN0032: │ evp_bytestokey@npm:1.0.3: Implicit dependencies on node-gyp are discouraged
14:18:51.731	➤ YN0002: │ starflare@workspace:. doesn't provide webpack (pcc690), requested by sass-loader
14:18:51.731	➤ YN0002: │ svg-sprite-loader@npm:4.3.0 doesn't provide webpack (p605ee), requested by html-webpack-plugin
14:18:51.731	➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
14:18:51.734	➤ YN0000: └ Completed in 6s 228ms
14:18:51.802	➤ YN0000: ┌ Post-resolution validation
14:18:51.803	➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
14:18:51.803	➤ YN0000: └ Completed
14:18:51.803	➤ YN0000: Failed with errors in 6s 299ms
14:18:51.844	Error: Exit with error code: 1
14:18:51.845	    at ChildProcess.<anonymous> (/snapshot/dist/run-build.js)
14:18:51.845	    at Object.onceWrapper (node:events:652:26)
14:18:51.845	    at ChildProcess.emit (node:events:537:28)
14:18:51.845	    at ChildProcess._handle.onexit (node:internal/child_process:291:12)
14:18:51.853	Failed: build command exited with code: 1
14:18:52.704	Failed: error occurred while running build command

是不是需要额外配置什么信息？

cloudflare 默认的 build 环境node js 版本太高，需要指定使用低版本node js. 同时还需配置其他环境变量：
CLIENT_ID  # github app client id
CLIENT_SECRET # github app client secret
NODE_VERSION 16.20.0
YARN_VERSION 1.22.19

感谢解惑，部署成功后，我发现点击了登录按钮，还是会跳转到 starflare.app
在 Homepage URL 和 Authorization callback URL 我填写的都是我的自定义域名。
https://gitstar.payloads.online/

你要改下 .env 文件 https://github.com/nieheyong/starflare/blob/main/.env
callback url 也需要更新一下，参照我的：

可以了，Callback URL需要增加 /#/login，感谢解惑！


Docker ignore nrcan_etl

If etl is in the .dockerignore file, then the new ETL nrcan_etl likely should be as well.

After talking to @szinck1 last week, this shouldn't be needed.


Bootstrap 4 support

This is a WIP PR for adding Bootstrap 4 support.

@kingcody - would love to get your input and feedback on this. Adding Bootstrap 4 support in the way I wanted to ended up being a bit of an undertaking.

@third774 do you have any idea of the release of v3.0.0 ? days, weeks, months, ... thanks 😗