phoSim sky background tests

The ongoing Run 1.1 has revealed a background issue with phoSim.  A summary from Rachael Mandelbaum in the slack desc-dc2 channel (2 Mar 2018, 09:21am Pacific):
For those who did not see all this traffic, a comparison between HSC data and the PhoSim run 1.1
images showed quite different levels of structure in the sky background.  In particular, PhoSim images
seem to have an excess of small-scale structure.  The reason this is a problem is that if you have to
ratchet up the complexity of the background model in DM, then it can complicate measurements related
to individual objects (e.g., galaxy photometry).  In other words: if the sky does not have strong
fluctuations on small scales, the combined background + object modeling problem is quite different from
the case where it does have strong fluctuations on small scales.  So -- there is a concern that the
PhoSim sky background may be turning this into an unrealistically complex problem.  This then leads to
the question of what to do about the rest of run 1.1, and what to do about actual DC2.

This issue will define the phoSim tests to be done (scope and configuration) and their results.  These tests will aim to change phoSim configuration settings in a controlled way to observe the differences in resultant images.

Table of tests (all WFD, r-band, obsHistID=417009, central raft (R22), otherwise identical with Run1.1 ) indicating relevant phoSim physics override changes.
Note:  "n/s" = not set in the command file, hence a default value is used. (see below)

Test(stream)
backalpha
backbeta
backgamma
backdelta
activebuffer
airglowvariation
clearclouds
note

Run1.1
0.1
4.0
1000
1.0
600
n/s
no

000000
0.1
4.0
1000
1.0
600
0.0
no

000001
0.05
4.0
1000
1.0
600
n/s
no

000002
0.01
4.0
1000
1.0
600
0.0
no

000003
0.1
4.0
1000
1.0
600
0.2
no

000004
0.05
4.0
1000
1.0
600
0.2
no

000005
0.01
4.0
1000
1.0
600
0.2
no

000006
0.05
4.0
1000
1.0
600
0.0
no

000007
0.01
4.0
1000
1.0
600
0.0
yes

000008
0.01
4.0
10
1.0
600
0.0
yes

000009
n/s
n/s
n/s
n/s
n/s
0.0
yes
full backgrounds

000010
0.01
5.0
1000
1.0
600
0.0
yes
oops!

000011
0.01
6.0
1000
1.0
600
0.0
yes
oops!

000012
0.1
5.0
1000
1.0
600
0.0
yes

000013
0.1
6.0
1000
1.0
600
0.0
yes

Note: for reference, all phoSim command files used in these tests are stored in /global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-test_WFD-r with names like commands-000001.txt (for Test 000001), etc.

Location of images (at NERSC):
/global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-test_WFD-r/output/<stream>

Brief description of the background parameters in phoSim taken from https://bitbucket.org/phosim/phosim_release/wiki/Physics Commands
Optimization Commands (use with caution):

- backalpha X (X is the signal to noise of a groups of background
photons above the background itself, default = 0.1, set to 0 for
single photon background (very slow))

- backbeta X (X is the size factor (times 10") of the background groups
above the detector, default = 4.0 which means 40" gaussian radius)

- backgamma X (X is the number of photons to simulate at a time (and
therefore smooth over detector physics) when a photon hits the sensor,
default = 2.0, for quickbackground use 1000.0)

- backdelta X (X is the size factor to divide by to get the size of the
grouping in the sensor, default = 80.0 which means a 0.5" gaussian
radius, for quick background use 1.0)

The default value for 'activebuffer' is 10 (for quickbackground use 600).  The default of 'backBuffer' is 100.

As a practical matter, some thoughts on the scope of the initial tests would be appreciated.  Remember that running phoSim for a single sensor is quite easy (interactively on haswell), but if the number of sensor-visits grows beyond 2-3, e.g., an entire raft, then we're talking batch jobs - and the lengthy delays in queue.
To start, @johnrpeterson has suggested two tests:

like run1.1 but adding "airglowvariation 0.0"
like run1.1 but adding "backalpha 0.05"

Is there discussion on this?  There were some hints that 'quickbackground' should be disabled entirely, is that also of interest (at the expense of a huge increase of execution time)?
Open questions:

which band(s)?
which visit(s)? [This determines the random number seed - but note that with multi-threaded operation, exact reproducibility is not guaranteed]
which sensor(s)?

@jchiang87 @johnrpeterson @sethdigel If I understood correctly from the DC2 channel discussion, starting with a couple of single sensors would be valuable, right? Since most images that were shown in the Slack channel for in r-band, that's probably the best one to look at? Is there an easy way to decide the visit and sensors so that we don't simulate some of them that don't have much on them? And then I guess start with the two options that Tom points out above? Thanks!

Maybe the image in this slack message: https://lsstc.slack.com/files/U06DGJCTB/F9H6VRYTA/v417009-fr-focalplane-summary.png

Who would be able to give Tom all the information he needs to do this
one? Thanks!
On 3/5/18 12:02 PM, Rachel Mandelbaum wrote:

Maybe the image in this slack message:
https://lsstc.slack.com/files/U06DGJCTB/F9H6VRYTA/v417009-fr-focalplane-summary.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370507860,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jJj-DxZI5rD-HETAipDGSBUsRvL2ks5tbX2-gaJpZM4ScjMo.

Can @johnrpeterson give us the validation suite that he runs internally, and point us at the tests that he used to validate this part of phoSim?  Otherwise we are reinventing wheels.

@katrinheitmann yes, that is sufficient to start setting up the tests (WFD, r-band, stream 157 = visitID 417009, #sensors = 132)

Looks like raft R:2,2 (the center guy) would be sufficient, i.e., we only need the central 9 sensors.

Is 9 small enough for doing this through interactive jobs? Thanks!

No, will need definitely need batch.  Although for only nine sensors, that means a single node for, say, 24 hours (I do not really know the execution time consequence of the config changes), so will hopefully run relatively quickly.

Sorry, I couldn't answer before.  Yes.. I agree with those tests to start.  I might also consider turning clouds off to start since that can also change the sky background.

(if you haven't already started)

Sorry Chris, the first two test jobs already started.  Unfortunately, Cori is in 'degraded' mode (i.e., DOWN) so progress may be slow.  Also, the first two jobs are testing my changes.  Should they crash, I could add in the cloud disabling.  However, I would like to get some consensus on what configurations are desired before committing to these runs.
At the moment, the two defined and (partially) executed test runs are described below.  All tests are being run using the DC2-phoSim-test_WFD-r workflow.  Each instance (stream) of the workflow corresponds to a different test.  Output will be stored in a directory indexed by Stream number, /global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-test_WFD-r/output/

Test(stream)
Changes from Run 1.1

000000
airglowvariation 0.0

000001
backalpha 0.05

Well we might as well just go with that.  Let's see what we find out after those two cases and then evaluate the appropriate next test depending on what we see.

How long should we expect the airglowvariation run to take?

Here is the wall-clock execution time for Run1.1 (WFD, r-band):

Sorry, just getting back to this after a bout of flu. It's essential that every run must have the same seed otherwise we may run into difficulties comparing runs. My understanding is that PhoSim is not strictly reproducible on the KNLs but I am not sure under what conditions this non-reproducibility appears. So the absolutely first thing to do is run PhoSim with the standard default parameters that were used to get the "wrong" answer on a single raft and check that the results are reproducible with the same seed and same (KNL-threading) run settings. If this test has been done already, I would like to see the results posted. Thanks!

The first two tests are complete (central raft, R22), per details above.  Please take a look.

I've made mosaic images.
airglowvariation 0.0:

backalpha 0.05:

It seems clear that the one with the airglow variation on has much more structure.  I'm worried about the artifacts from the optimization too, but at a lower level than the airglow variation.

And here is the original Run 1.1.  It seems like the optimization actually smooths out the structure rather than the other way around:

Thanks Simon! Curious to hear @johnrpeterson take on these results.

Note I updated my previous post as it was scaled for the whole focal plane which smoothed out the structure for the central raft.

Very interesting @SimonKrughoff .  Looks like the airglow variation is responsible to me, but curious to hear others.

For dummies: does that simply mean the model for the airglow variation
was too extreme and needs to be adjusted (how)? Are there other tests we
have to do now to see if the structures can now be handled by DM without
any major problems? Thanks all!
On 3/6/18 8:20 AM, Rachel Mandelbaum wrote:

Very interesting @SimonKrughoff https://github.com/simonkrughoff .
Looks like the airglow variation is responsible to me, but curious to
hear others.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370795684,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jEvU0awC4EkaZVKXICLNMbGGuQZRks5tbpstgaJpZM4ScjMo.

For dummies: does that simply mean the model for the airglow variation
was too extreme and needs to be adjusted (how)? Are there other tests we
have to do now to see if the structures can now be handled by DM without
any major problems? Thanks all!

I think if this is the airglow then we probably need to adjust PhoSim's model (or just turn off airglow for now).  It looks like to much power is just extended to small scales. John can you tell us what the form of the power law actually is?  What kind of cut-off are you applying to to the k**-0.3?
It is in some sense not surprising that the optimization could smooth out the structure since as John has explained, it does that to other features too.
But it's a bit hard to for me to see in the scaled down images: Simon, do you not see the structure in the airglow=0 case? I see some structure but the images are scaled and that one is smaller on each sensor so it is hard for me to tell.
BTW: The reason I asked about the clouds the other day was I wasn't sure if they could block the light in some pattern like this.

BTW: I can't find airglowvariation on the PhoSim command page but I assume this is suppose to be
adjusting the amount of variance not whether there is sky glow at all (https://bitbucket.org/phosim/phosim_release/wiki/Physics Commands).  So this is why we still see something in the airglowvariation=0 case and the question is if the structure is still there.

[Too stop confusion: I see Simon edited his post and the comment about optimization smoothing the structure is now gone.  So, I am also going to take that part out of my response.]

Hi Karin,

Are there other tests we have to do now to see if the structures can now be handled by DM without any major problems?

Sorry... I realized I didn't answer this part before.  What Simon told us what that DM can handle the structure no problem.  The problem is by making this structure part of the sky background fit by increasing the order of the polynomial, it will introduce  biases into our galaxy photometry which is something we want to check so that he and Robert thought we shouldn't do it.  Simon should say more.

@cwwalter sorry for the confusion, and you are right.
We can fit out any background, it's just a question of how that background interacts with other algorithms.  We have been assuming the background would be smooth in r-band on scales of a chip, but if that's not true it has impacts on e.g. galaxy photometry.  We want realistic complexity so we can get a realistic estimate of the performance of our measurement pipeline as a whole.

Looking at it more, I'm really confused by the images with no airglow.  They seem to fall off at the edges.  That is also bad for background modeling because sharp features generally cause ringing.

@SimonKrughoff - I share your confusion about the roll-off at the edge of each chip, but I wouldn't expect it to impact the IQR, since it's only a tiny fraction of the pixels that are affected and the IQR should be robust to such outliers.  I apologize for adding to the noise in this issue, but where are the IQR values tabulated?  I cannot find them.  If IQR is not a useful statistic for diagnosing / understanding this issue then it's good to know that, but I am very surprised.

I share your confusion about the roll-off at the edge of each chip, but I wouldn't expect it to impact the IQR, since it's only a tiny fraction of the pixels that are affected and the IQR should be robust to such outliers.

Good point.

Why is there any background variation in this first image?  What's going on at the edge?

I assume this is from the "optimization", but John should say more.

I don't think we can accept phoSim optimisations that generate artefacts visible to the naked eye.  If they lead to subtle correlations that's bad (but maybe OK);  this sort of structure is not.
DM can handle "data" like this, but it is unphysical and will impact the results.  For example, we are planning to estimate the sky on super-CCD levels and this sort of bug will have serious consequences.  This particular issue isn't a problem for DC2, but I do not guarantee that DM will correctly handle data that is not representative of real sky data even for DC2.

Well, this edge effect is clearly unacceptable and will no doubt have to be fixed, DM or no DM. I don't see why it should come from an optimization either, so some rational explanation is needed.

Here are the IQR values and medians for the e-images.  For Run 1.1, the phoSim configuration parameters are listed here; I think that they correspond to airglow variations enabled and backalpha = 0.1.  The units in the table are electrons/pixel.
0 -> airglowvariation = 0
1 -> backalpha = 0.05
2 -> Run 1.1 version

CCD  IQR0  IRQ1  IRQ2  Median0  Median1  Median2
S00 37  38  38  703  699  700
S01 38  38  38  704  697  701
S02 38  38  38  703  703  704
S10 37  38  38  703  693  694
S11 38  38  38  705  703  704
S12 37  38  38  701  705  706
S20 37  37  38  705  687  688
S21 37  38  39  703  699  700
S22 38  38  38  704  703  705```

Ok, just to make this clear -- we are running 1.1 to identify any issues
-- this is one now we found. So no need to panic here yet ;-)
On 3/6/18 11:50 AM, Salman Habib wrote:

Well, this edge effect is clearly unacceptable and will no doubt have
to be fixed, DM or no DM. I don't see why it should come from an
optimization either, so some rational explanation is needed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370868226,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jCzGC-BfyI_B7U5TnyAZ3IW000Mdks5tbsxigaJpZM4ScjMo.

Since the summary statistics of the pixel values don't seem very informative, fwiw, here are the distributions of pixel values for the R:2,2 S:1,1 sensor (the very central one) for the three different cases:

and rebinned into 8x8 boxes (taking the mean over each box to preserve the x-scale):

This just confirms what one can see by eye, but it at least gives us a means of attaching a number, e.g. the IQR of the rebinned data, to the spatial variation in the images.

hi-
a few things:

the distortion of the field near the edge is on in these runs.  that will approx. come off if you use flats.  is that what you are talking about, Simon/Rachel?
a trivial short-term fix would just be to set airglowvariation to a number less than 1.  we can fix it comprehensively later.  robert, can you give an upper limit for variation for the HSC r-band?  i couldn't see the scale on your plot.  its looking like the airglow variation is about 10/700=1.5% in these sims, so we could scale down from that level.
there are clouds on here, but i don't think they matter much on chip scales.  they are more relevant on degree scales.

@johnrpeterson So you are not worried that the IQR value is higher than it should be? And you are happy with the results as they are? (as far as I understand you are suggesting leaving everything as is and just add a  smaller airglow variation compared to what we did in 1.1 previously)? Thanks!

i think the IQR values are fine, right?  for 700 counts, i get IQR of 35 expected from poisson noise. or am i wrong?
i think the main problem is just there is some correlated patterns (mostly from some larger airglow variation than expected, we now know).  that's why you only see the issue when Jim bins it 8x8.  so yeah, i would first just advocate to reduce the airglow variation number as a really easy fix.

The pattern in the background when airglow is off is also an issue.  How do we get rid of that?

that could be the optimization. or could there even be higher density of wings of bright stars there? or something else?
I think Tom should try a run with "airglowvariation 0.2", which should be realistic and then we can see where we are.
you could also have another run with "airglowvariation 0.2" and "backalpha 0.01" which would be slow, but then we could see at what effect the optimization is exactly.

I'm not keen to keep debugging the airglow model.  I think we should turn it completely off and do a backalpha 0.01 run to make sure that looks reasonable.  We can then figure out where we stand in terms of runtimes.

Just to be fully clear:
airglowvariation 0.2 and |backalpha ||0.1 (the original value)|
|or
|
airglowvariation 0.2 and |backalpha ||0.05 (as in the last test where
backalpha was changed)|
|and
|
airglowvariation 0.2 and |backalpha ||0.01 for comparison?
Or all three? Or should we run a test with switching off the
quickbackground completely instead of the last option?
Thanks!
|||||
||
On 3/6/18 2:49 PM, johnrpeterson wrote:

that could be the optimization. or could there even be higher density
of wings of bright stars there? or something else?
I think Tom should try a run with "airglowvariation 0.2", which should
be realistic and then we can see where we are.
you could also have another run with "airglowvariation 0.2" and
"backalpha 0.01" which would be slow, but then we could see at what
effect the optimization is exactly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370923267,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jA2xMAgKMRg5t5bu9eyzrQZpaUAkks5tbvZugaJpZM4ScjMo.

yeah, i like all three of what you said.   backalpha=0.05 and backalpha=0.1 are probably quite similar in terms of run time, but then backalpha=0.01 would be noticeably slower.   so we need that one so we can clearly see the effect go away.

Thanks! And then coming back to Simon's question, should we for now keep the airglowvariation = 0 for some of these cases?
So:
-----|-----|-----|-----
ORG|airglowvariation 1.0| backalpha 0.1| DONE
000000|airglowvariation 0.0| backalpha 0.1| DONE
000001|airglowvariation 1.0|backalpha 0.05 | DONE
000002|airglowvariation 0.0|backalpha 0.01
000003|airglowvariation 0.2| backalpha 0.1
000004|airglowvariation 0.2| backalpha 0.05
000004|airglowvariation 0.2| backalpha 0.01
What will be the criterion that says what's the best choice? So we would have 000002 as baseline and then we dial up airglow again. Maybe this is too much? (Though the good thing is that it will all run more or less at the same time otherwise we'll have to wait longer). @TomGlanzman Could you set up at least 000002 and 000004? I think we definitely need/want those.

Thanks! And then coming back to Simon's question, should we for now keep the airglowvariation = 0 for some of these cases?

Just to be clear: Does airglowvaration=0 mean no airglow (that is what Simon suggested).  Or is it the amount of variance?  How would we actually turn it off?
What is the difference between 0.2 and 0.0 in the model? You said that 0.2 would be more realistic do you mean relative to the default (as opposed to relative to 0)?
Thanks!

I really meant airglow variation off.  I'm assuming we need the constant airglow to get the correct SNR on the measured sources.
I really don't think we should mess with the airglow variation at all.  Just turning it down won't get rid of the non-physical structure at small scales.

Four new tests, per @katrinheitmann, have been configured and started.   I've pinned a Table of tests to the issue statement (top of this issue thread) and will be updating as needed.

0.2 means 1/5 of the default, and 0.0 means off.
i looked back through the logic of why we set it the level that we did and i think 0.2 would be more reasonable for the continuum portion of the airglow (which is most important).   so that's why i'm advocating 0.2.
but simon how can that be true?  its only at the 1% level of variation even if we kept it the way it was originally.  how can DM be this sensitive to this?

but simon how can that be true? its only at the 1% level of variation even if we kept it the way it was originally. how can DM be this sensitive to this?

How can what be true?  The model of the spatial skyglow variation is demonstrably wrong.  I don't see the point of turning a wrong model down.  We should just turn it off.

0.2 means 1/5 of the default, and 0.0 means off.

OK thanks.  But I'm confused.  Can't we see airglow in the pictures above even for the airglow = 0 case?  Or am I misinterpreting the plots?

yes, Chris, that probably is the optimization, but not sure until we do the next round of tests.
simon, i think "demonstrably wrong" is oversimplifying.  IR bands can look like that, and there is variability in the continuum as well, but it could be hard to see in real images, because other instrumental effects dominate.  i think it is overly-cautious to turn it off entirely, but certainly thats a valid argument.

The defaults are certainly demonstrably wrong.  Unless the phosim team can provide some support for the model of the skyglow variation, I think it is ill advised for us to leave it on.

I think that we need to run a model without optimisation to confirm that the backgrounds are actually correct, and only then see which features to turn on.  For DC2, I'd be happy to have uniform backgrounds.
DM does care.  We know about astronomical data and that's baked into our algorithms.

John, regarding this:

IR bands can look like that

This is r-band.  So at minimum there seems to be some incorrect wavelength dependence of the airglow variation.

yes, i understand.  its at a lower level in r-band, but not low enough.

Thanks John.
So to make sure we are on the same page about next steps: we should run with airglowvariation=0 and several backalpha options, as Simon said (and both Chris and John appeared to agree), so as to understand whether residual variations can be attributed to the optimization.   We should not at this time run with airglowvariation set to a nonzero value, because there are clearly some effects we don't understand even with airglow variation turned off.   Please thumbs up this if you agree with my summary, or comment otherwise.

yes, Chris, that probably is the optimization, but not sure until we do the next round of tests.

Hi John,
OK I think I understand my confusion now.   I assumed the variation was just on a component we were separately controlling and could turn off in the model. I see it is unified together now.  But I'm looking at the source code and I see for example
airglowintensity = 22.08;
and also the ability to turn the background off all together with backgroundmode=0.
So, just to make sure I understand: if I wanted a non-varying sky background I would set the variation to zero and the intensity to some scaled value of the intensity magnitude I wanted and this would also scale the continuum? Is that correct?

yes, Rachel, that is a fine plan.  having only a couple hours this morning to study it, i think i can make the case that 0.2 would be a reasonable better estimate, but its up to everyone's comfort level if we should just turn it off.

I think at this point we might as well run all the tests on the list and
see where it gets us ...
On 3/6/18 5:08 PM, johnrpeterson wrote:

yes, Rachel, that is a fine plan. having only a couple hours this
morning to study it, i think i can make the case that 0.2 would be a
reasonable better estimate, but its up to everyone's comfort level if
we should just turn it off.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370962224,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jO8OklSHEWtnrcsQT18MdSfFkvfOks5tbxb3gaJpZM4ScjMo.

Because of the interest in airglowvariation=0.0, I've added one more test run with that plus backalpha=0.05 to round out the testing space.

Before we start: John: do you think we should turn off clouds?  What kind of extinction effect might they cause?

i think we should leave them on for now, because i think those are larger scale.  if getting the backalpha near 0 doesn't work, then at that point we should study clouds, i think.

Ok, I think this mode of sequential discussion is not all that helpful. I propose we have a telecon with the PhoSim folks to decide what to do next (probably best after the next set of runs).
I don't even understand the airglowvariation 0.0 model because there is clear chip to chip variation (compare the bottom right to top center), why is that? Moreover, there is no reason why we can't get an appropriate spatial variation in the airglow model, by simply using a lower-k cutoff, so running with airglow completely off does not seem to be necessary, nor even desirable.

I think we should turn off clouds.  We want to confirm that there is no spatial structure, and if we see some we'll not be sure where it comes from.

Moreover, there is no reason why we can't get an appropriate spatial variation in the airglow model, by simply using a lower-k cutoff, so running with airglow completely off does not seem to be necessary, or even desirable.

The problem is that there really isn't

Salman, the high spatial freq is likely the optimization.  and the cutoff is not needed as it converges to 0.  i agree that sequential discussion is not ideal though.
Robert, one of the runs will be with the optimization turned off.  so the residual will then be due to things like clouds.  however, i expect it to be rather flat at that point.  but i agree that would be the next step if its not.  my advice is to simply tick down from the most likely to least likely one at a time.

After our woes trying to sort this out I very strongly recommend that we start with the trivial case where everything is boring.  We could turn off the stars and galaxies too if it'd be convenient.

Moreover, there is no reason why we can't get an appropriate spatial variation in the airglow model, by simply using a lower-k cutoff, so running with airglow completely off does not seem to be necessary, or even desirable.

My worry is that we don't have data to constrain what the spatially variable component of the airglow should be in r-band.  There's really no reason to assume it has the same form as in the IR since they are different processes.

If we add another trial case with the clouds off (and due to the goal to
sort this all out soon I'd rather start more than fewer tests), would we
also use the very low background setting? That will slow down things but
I wonder if we would not we might still not know what causes what. Or
would it be ok to leave the background parameter at 0.5 since we have a
comparison run that would then tell us what the clouds do?
On 3/6/18 6:47 PM, Robert Lupton wrote:

After our woes trying to sort this out I very strongly recommend that
we start with the trivial case where everything is boring. We could
turn off the stars and galaxies too if it'd be convenient.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-370982398,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jFh8IylRpEdYmzTT6pEHh5arTXf3ks5tby4hgaJpZM4ScjMo.

you could do an extra one with:
clearclouds
airglowvariation 0.0
backalpha 0.01
i would expect this similar to this one on chip-scales:
airglowvariation 0.0
backalpha 0.01

@SimonKrughoff and all, what do you think about using the model in sims_skybrightness (https://github.com/lsst/sims_skybrightness) to predict the spatial variation in the different bands? At least for estimate the order of magnitude it should be good, right? (Last time we checked that using that model the spatial variation within a chip was smaller than the expected Poisson noise given the median background level: see the presentation by @belaa https://confluence.slac.stanford.edu/download/attachments/228297704/skyModel_SSIM_2_NOV_2017.pdf?version=2&modificationDate=1509669378000&api=v2)

@SimonKrughoff , my point was not to get the actual variation in r-band to be "correct" but to have a reasonable amplitude and a spatial variation that can be handled by DM; at the minimum that's all we need.
@johnrpeterson , so if I understand correctly what you are saying is that 1) at fixed amplitude, even white noise background (flat power spectrum) will have a spatial clumping due to the optimization setting, and 2) at the current amplitude setting of the airglow, this effect is swamping the actual spatial variation of the airglow model?
If the point about airglow above is correct, we need to run tests with everything turned off as advocated by @RobertLuptonTheGood and then figure out the appropriate optimization setting (i.e., any spatial artifacts on top of pure noise would be too small to be problematic for any DM-related processing). Makes sense?

yes, that is the commands i sketched out 3 posts earlier.  i do think its important to define the absolute level where this would matter as well.

@TomGlanzman Can you add that test also to your suite? I think it will be good if we can have this all covered and look at it at once. Thanks!
clearclouds
airglowvariation 0.0
backalpha 0.01

Ok, good, so moving forward optimistically, we have the following scenario: @TomGlanzman runs some tests at various backalpha settings and we discover the setting at which everyone is happy, meaning that all "optimization" artifacts are below some agreed-on DM happiness threshold. This is not to say that, for example, airglow and cloud models would be perfect but that DM would not be afraid to deal with them. At this point the remaining question would be how much extra time it takes to run PhoSim under the new settings; if it's within a factor of two of the old value, I would not worry too much but if it's substantially longer we will need to come up with a solution.

@katrinheitmann the seventh test with no clouds has been configured/submitted.  (See table above)

(i.e., any spatial artifacts on top of pure noise would be too small to be problematic for any DM-related processing)

To be clear the problem isn't in DM, it's any astronomical interpretation of this data.

Tom's streams<PHONE_NUMBER>06 have finished.  At least in terms of IQR the sensor visits are all very similar, with IQR = 37 or 38 and medians ranging from 687 to 708 with most ~700 electrons/pixel.

@RobertLuptonTheGood , I understand the meaning of the word artifact ;-). At this point, the interaction with DM is more important than any more general issue. So, as long as you DM people are not unhappy with the images, we don't want to worry about any other potential issues. They are second order in this sense.

All of the currently defined tests have completed.  Please have a look:
/global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-test_WFD-r/output/
(See table at top of issue for key to  parameter.)

I edited my posting from last night to include IQR and median for each of the sensor images for each of the streams.

I'll make the binned image mosaics ASAP.  I'm in sessions at the JTM.

Actually already on it :-) Your instructions from yesterday worked well.
Give me another 15 minutes or so and I am ready to post the results.
On 3/7/18 12:10 PM, Simon Krughoff wrote:

I'll make the binned image mosaics ASAP. I'm in sessions at the JTM.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371230219,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jJ68dlaVXZAfsMN_ePe8M5J_mAyBks5tcCKPgaJpZM4ScjMo.

summary.pdf
So I am not 100% sure how people want to look at the outputs -- I can post them image by image but then I don't remember which model was which. So for now I just compiled them into a pdf file and put the values at the top of each page. Please let me know if you would prefer that each image is displayed here separately. The problem is that the tests are not conclusive at all. There seems to be too much variation when only one parameter is varied. @SimonKrughoff When I use your instructions, I assume that the settings for all the zscale etc are kept the same right? So we are comparing apples and apples here, right? I am worried that the mosaicing might not be doing the same for all outputs. Anyway, take a look -- (and again, let me know if the pdf is not useful for the discussion).

Do I see clouds in test 000007?  The command file does contain "clearclouds", and that is confirmed in the log.

I am blinking the pdf from p.7 and p8, and I do not see any difference actually : the images look identical.

I am blinking the pdf from p.7 and p8, and I do not see any difference actually : the images look identical.

Yes. there's clearly no structure coming from clouds.

ok, so sorry.  i just realized with quick background it isn't sufficient to just change backalpha alone.  to truly remove the optimization you'd have to also modify backgamma.  sorry, somewhat subtle.  so a new run with:
backgamma 10.0
backalpha 0.01
airglowvariation 0.0
clearclouds
should work and produce a rather flat image (i hope!).  and i guess we also learned that clouds are not dominant on these scales (i had suspected that it is only on much larger scales (~degree) that you'd notice anything).

We just took a careful look at the data loaded by Katrin. It is not possible to compare the images because one gets opposite trends as a function of which chip one is looking at (when comparing one raft run with one set of options to another), and second, it is obvious that the actual structure within corresponding chips is different. [To see what I mean, compare the runs with backalpha 0.1, airglowvariation 0.0, clearclouds no (Test 000000) with Test 000002, which has backalpha 0.01, and is otherwise the same.] This means that the code is not being run in a repeatable fashion.
Perhaps this has to do with different random number realizations? I had asked about  precisely this issue earlier. Even with the new runs that John mentions, we still need to fix this.

@katrinheitmann my code uses defaults for the zscale so could change from one to the next.  That can be changed, but seemed to be working well to see the structure in the background.

ok, I did make a mistake with the last image ... sorry for that (it was just before lunch, so I was in a hurry ...) Corrected image uploaded:
summary.pdf
Also, if anybody wants to look at the outputs from Simon's set-up the files are here: /global/cscratch1/sd/heitmann/output/
at NERSC. @SimonKrughoff Thanks for the reply!

The latest test run, with backgamma=10 is under way (now awaiting SLURM to dispatch).  This will be test 000008, per table at top.

actually, Salman, you may have accidentally proven that the optimization is this issue for the remaining residual.  we can wait for this run from Tom, but the parameters of various things chosen from distributions are repeatable.  however, the optimization means the sources get defined in bunches of  photons differently when you change those numbers.  the fact that those images look different when only optimization parameters are different suggests that the optimization is the dominant pattern in those images.   but lets wait for the results from 000008....

Hi John,
I am curious, we have a couple more background parameters:
|backalpha ||0.1|
|backbeta ||4.0|
|backgamma ||1000.0|
|backdelta ||1.0|
|activebuffer ||600
(from the original setting). Is there a chance that others would need
adjusting as well? Do you have some test results handy somewhere that
show what the different parameters do? Thanks a bunch!
|
On 3/7/18 1:30 PM, johnrpeterson wrote:

ok, so sorry. i just realized with quick background it isn't
sufficient to just change backalpha alone. to truly remove the
optimization you'd have to also modify backgamma. sorry, somewhat
subtle. so a new run with:
backgamma 10.0
backalpha 0.01
airglowvariation 0.0
clearclouds
should work and produce a rather flat image (i hope!). and i guess we
also learned that clouds are not dominant on these scales (i had
suspected that it is only on much larger scales (~degree) that you'd
notice anything).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371256026,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jCo4ElXayNQmWIyZZROOKPdj3k71ks5tcDU8gaJpZM4ScjMo.

@johnrpeterson , I agree with your remark about the internal structure potentially depending on the optimization parameter -- do you have an explanation for why entire chips have the opposite brighter/dimmer relationship seemingly randomly in the comparison I mentioned?

@johnrpeterson - can you please clarify something: are you saying that without quickbackground optimization, it is possible to get completely repeatable PhoSim runs by setting random seeds, but with optimization, it's not possible to do so?  i.e., the photon bunches are not tied to the input random seeds in a way that makes the process repeatable?  Sorry to be so pedantic but I want to make sure I understand the process and implications for how we design and interpret further tests.

katrin, the others (beta, delta, and activebuffer) just have to do with the size of the bunching and a buffer, so we don't need to change those.  in the limit that alpha and gamma goes to 0, then phosim goes to a pure photon simulation of the background.
salman, i don't know-- is the mosaic scaling them so they are at the same absolute level?

Rachel-
complicated question.  what i said earlier is that i'm making a simpler point.  when you change background optimization parameters it is going to take pieces of the background and bunch them differently.  so then any artifacts (if due to optimization) are going to be different.  not related to repeatability.
as for repeatability-

all phosim setup is completely repeatable with different runs.  so this means anything related to what the atmosphere looks like or patterns of turbulence or background levels or optics distortions, etc. are all repeatable (and platform-independent and multithreading-independent).

the photon propagation (so how the photon responds to all that when it uses random numbers) is repeatable as well if you use a single thread.

however, the photon propagation is not guaranteed to be repeatable when you run it in an multithreaded way.  so this means that the images should be statistically identical but not necessarily exactly the same electrons in each pixel.  we did work on making this perfect as well, but it was too hard to get into v3.7 in time, because its rather complicated.

the type of tests you would typically do should be primarily sensitive to #1.   i think that would cover most cases of where in practical terms you would really need repeatability-- so for instance if you rerun an observation the background levels will be the same, the clouds will be the same, etc.  #3 is probably more important if there is an ultra rare seg fault or something like that and you want to debug it.

@katrinheitmann I do wonder if we should change activebuffer to be bigger.  That may be contributing to the edge effects we saw in the original test runs.

Without wishing to add more noise to this discussion, I previously played
with backalpha and backbeta and found that increasing backbeta seemed to
provide a smoother background. It appeared to scale the radius of the
distribution of photons. backalpha did not make much of a difference in the
fluctuations. The issue with larger backbeta was that it caused edge
effects on the chips (which might mean you need a larger buffer when
defining the backgound sources). This work was from a few years ago so
might be out of date relative to the current code but if the current runs
with no optimization are taking a long time it might be worth exploring.
The values I used for backalpha was 5 and I turned off airglow and cloud
On Wed, Mar 7, 2018 at 4:34 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Two more test runs, 000008/000009, in progress. '009 will take
considerably longer than the others as it will simulate full backgrounds.
(See table at top.)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371335857,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDow0n56ywX_SKwNXGWN8gL-tlOLTeqks5tcHyUgaJpZM4ScjMo
.

Thanks Andy! This is definitely not noise but very helpful --
What did you use for backbeta in the end? And for the buffer value?
I think right now more tests are useful to find out what the best
setting is. I hope the tests will converge to a reasonable answer in the
end. It looks like lowering backgamma is adding compute time. What did
you find for that parameter?
Both John, Andy, is there a brief description that explains all the
parameters?
Thanks! Katrin
On 3/8/18 1:18 PM, Andrew Connolly wrote:

Without wishing to add more noise to this discussion, I previously played
with backalpha and backbeta and found that increasing backbeta seemed to
provide a smoother background. It appeared to scale the radius of the
distribution of photons. backalpha did not make much of a difference
in the
fluctuations. The issue with larger backbeta was that it caused edge
effects on the chips (which might mean you need a larger buffer when
defining the backgound sources). This work was from a few years ago so
might be out of date relative to the current code but if the current runs
with no optimization are taking a long time it might be worth exploring.
The values I used for backalpha was 5 and I turned off airglow and cloud
On Wed, Mar 7, 2018 at 4:34 PM, Tom Glanzman<EMAIL_ADDRESS>wrote:

Two more test runs, 000008/000009, in progress. '009 will take
considerably longer than the others as it will simulate full
backgrounds.
(See table at top.)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub

https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371335857,

or mute the thread

https://github.com/notifications/unsubscribe-auth/ABDow0n56ywX_SKwNXGWN8gL-tlOLTeqks5tcHyUgaJpZM4ScjMo

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371593755,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jN_-81rVCjlFMTv_LvbNokKcpRgnks5tcYQNgaJpZM4ScjMo.

@katrinheitmann John updated the (brief) description of the 'commands' in bitbucket, including the back* parameters, here:  https://bitbucket.org/phosim/phosim_release/wiki/Physics Commands

I used 5 in my tests but would try 5 and 6. I didnt know what parameter set
the buffer of sources used to define the background (which would reduce the
roll off at the chip edges) but I think you mentioned "activebuffer" and I
would increase that by 50% if John says that is the correct parameter
On Thu, Mar 8, 2018 at 11:30 AM, Tom Glanzman<EMAIL_ADDRESS>wrote:

@katrinheitmann https://github.com/katrinheitmann John updated the
(brief) description of the 'commands' in bitbucket, including the back*
parameters, here: https://bitbucket.org/phosim/
phosim_release/wiki/Physics%20Commands
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371597147,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDow7N9n40zN_J-JL7Y35M_-JkL8D97ks5tcYbXgaJpZM4ScjMo
.

Test 000008 failed to complete within 24 hours, so it has been resubmitted to a 48h queue -- along with the 000009 test.  Based on DC1 experience, there is a good chance both tests will complete within that time span...but not guaranteed.

@TomGlanzman I think we should add Andy's suggestions as well. Maybe increase the backgamma value again to make the test faster? Thanks!

backalpha 5.0 (original 0.1, increased following Andy's comment above)
backbeta 5.0 (original 4.0, increased following Andy's comment above)
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
@connolly @johnrpeterson Just to double check, did you really mean above backalpha 5.0 or backbeta? Do the settings proposed above look reasonable? Thanks again!

Sorry I meant to just change backbeta and the buffer (if possible)
run 1:
backbeta = 5
backalpha = 0.1
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
run 2:
backbeta = 6
backalpha = 0.1
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
On Thu, Mar 8, 2018 at 1:39 PM, Katrin Heitmann<EMAIL_ADDRESS>wrote:

backalpha 5.0 (original 0.1, increased following Andy's comment above)
backbeta 5.0 (original 4.0, increased following Andy's comment above)
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
@connolly https://github.com/connolly @johnrpeterson
https://github.com/johnrpeterson Just to double check, did you really
mean above backalpha 5.0 or backbeta? Do the settings proposed above look
reasonable? Thanks again!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371633459,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDow4caZPdz-GhGdRspzSU9doYk23r7ks5tcaUfgaJpZM4ScjMo
.

Perfect, thanks, that's what I guessed you meant.
On 3/8/18 3:42 PM, Andrew Connolly wrote:

Sorry I meant to just change backbeta and the buffer (if possible)
run 1:
backbeta = 5
backalpha = 0.1
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
run 2:
backbeta = 6
backalpha = 0.1
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
On Thu, Mar 8, 2018 at 1:39 PM, Katrin Heitmann<EMAIL_ADDRESS>wrote:

backalpha 5.0 (original 0.1, increased following Andy's comment above)
backbeta 5.0 (original 4.0, increased following Andy's comment above)
backgamma 1000.0
backdelta 1.0
activebuffer 900 (original 600, 50% increased)
airglowvariation 0.0
clearclouds yes
@connolly https://github.com/connolly @johnrpeterson
https://github.com/johnrpeterson Just to double check, did you really
mean above backalpha 5.0 or backbeta? Do the settings proposed above
look
reasonable? Thanks again!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub

https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371633459,

or mute the thread

https://github.com/notifications/unsubscribe-auth/ABDow4caZPdz-GhGdRspzSU9doYk23r7ks5tcaUfgaJpZM4ScjMo

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371634267,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jHaPuyctj019S6X53nVsz3UdCqm3ks5tcaXOgaJpZM4ScjMo.

After discussion, the two test runs recommended by Andy Connolly have become tests 000012 and 000013.  Tests 000008 through 000013 are running now.

Tests 10,11,12,13 are complete.   Tests 8 and 9 are still running ... and likely will be for another 1+ days.

summary_new.pdf
Thanks @TomGlanzman! And here are the additional images from the test (I just added them at the end of the file). @connolly With a theorist's eye, I would say your suggestion for backbeta=6.0 is very helpful -- what do you think? Is it worthwhile to switch on the airglowvariation and the clouds again at this point?

Yup, looking much better. Agree we can do some runs now with the spatial variation turned on --

Yes looks better. I would see if there is a way to get the roll off at the
edge of the chips to be less before turning the clouds and air glow back
on. I'm not sure what the best number should be for backBuffer to reduce
the roll off but increasing it to 300 to see if anything changes would be a
good test. Maybe @johnrpeterson  would have a suggestion
On Thu, Mar 8, 2018 at 9:26 PM, Salman Habib<EMAIL_ADDRESS>wrote:

Yup, looking much better. Agree we can do some runs now with the spatial
variation turned on --
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371717244,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDowy6umCmKNFJmXSBFNjYRJhoXYnSMks5tchJsgaJpZM4ScjMo
.

For the record, here are the profiles in the x direction for stacked, filtered CCD images for streams 10-13.  The scale is the same as for the profiles I posted yesterday for streams 0-7.

summary_new_2.pdf
An important test finished over night -- Test000009, full background. The result has been added to the others. The run with low backgamma is still not done -- so that is doesn't seem an option then? I have put the new outputs from Simon's code to the rest here: /global/cscratch1/sd/heitmann/output
@TomGlanzman How much slower was 000009 compared to e.g. 000002? Thanks!

So 000009 shows uniformity, and it looks like the roll off at the edges almost completely disappeared (Seth will tell). There are some spots that look as if they are outside of the almost perfect sensor squares. For my own education, what are they?

@katrinheitmann The wall-clock time for test 000009 (full background) was approximately x3 longer than test 000002 (reduced background).

ok, some thoughts:

some of the suggestions i had (like setting alpha=0.01 and gamma=10.0) was simply to verify that the optimization is the problem, which we now have verified.  that isn't a realistic option computationally.

andy is right that you can definitely change backbeta to get flatter backgrounds.  this has the advantage that this has very little CPU cost, if any.  there is a slight risk that things like vignetting & large-scale background gradients will get somewhat distorted if you set this too high.  backbeta=4 has been thoroughly tested though, so changing it to 5 or 6 should be fine.

in terms of a better optimization, i'd suggest

A) modest increases in "backbeta" (5 or 6 seems to be working nicely)
B) slight decreases in "backalpha" (the CPU time will scale inversely with this, so don't go as low as 0.01, but you could always go to 0.05 or simply keep the default of 0.1)
C) keep backgamma the default for quickbackground = 1000 (i think unless anyone finds that this makes a difference between say 1000 and 100)
D) keep backdelta the default for quickbackground
E)  as for buffers there are actually two:   a buffer it has to think about light on the sky (buffer) and one near the sensor (activebuffer).  either one could cause some problem.  both are in pixels.  both could have some effect.   the actual buffer for the "buffer" is actually 3beta50+300+buffer with the default of buffer (100) this gives 1000.  so i'd suggest activebuffer just change to 1000.
so i'd say either do:
backalpha 0.05
backbeta 5
activebuffer 1000
and everything else the same as quickbackground
or
backbeta 6
activebuffer 1000
and everything else the same as quickbackground

I've updated my posting above to include the profile for stream 09.  (I found that the original version was showing the profiles for streams 00-03 instead of 10-13 as I had intended, so I have fixed that too.). The profile for stream 09 is flatter in the approach to the edge, but falls off much more steeply and more deeply near the edge.

@TomGlanzman Thanks! The timing information is definitely useful. In the meantime, two more tests (16,17) finished. The clouds are back on and the backbuffer was increased (and backgamma decreased). Those results are slightly confusing again -- maybe easier to interpret when the clouds and no quickbackground is done.
summary_new_3.pdf

in 000016 the two bottom left sensors seem out of family no?

Some more results -- while test00008 is not yet quite finished, I included what's there already which doesn't look promising. I'll update it when it's fully done. Probably best to start with the second pdf I am posting here and follow the comments.
summary_new_4.pdf
summary sorted.pdf
The first pdf file contains the tests in the order we ran them and they are given in the table above. The second pdf file sorts them a little bit differently. The first three images are all without quickbackground, first with no clouds and no airglowvariation, then coulds switched on, and then clouds and airglowvariation also switch on (after we started the test with airglowvariation, John sent a message to not switch that on... so maybe just ignore that one for now). The next four images have no clouds and vary both backalpha and backbeta (the default backalpha = 0.1, backbeta=4.0). The next two have the activebuffer increased (the hope was that the edges would improve) and we switch the clouds back on (assuming that's what we want in the end in DC2). The next three show the variation with backalpha and clouds (those don't look very promising). The next three show the same and additionally airglowvariation turned on (again, after John's message about airglowvariation probably can be ignored). The second to last has a large airglowvariation (early test) and can probably be ignored. The last one changes backgamma and clearly produces artifacts.
So -- what I would like to know from the experts is: is what we see in the first two images what we want for DC2? Just "looking" at the images is not very informative in my mind (I loaded some of the fits files into ds9 and can produce as junky or as smooth pictures as I want) so at this point with all these tests we need some metric about what is good. I very much hope that the first two or three are in fact acceptable. If they are then we have established our baseline, maybe we can try some more to get closer with varying the parameters we looked at so far. Thanks!

@johannct Yes, you are absolutely right ... as I think Salman asked before, there is a lot of variations in the sensors in a lot of places -- variations within one test (as you point out in this case) and variations comparing tests where only one parameter is changed (e.g. one would expect all nine sensors be darker due to a parameter effect but then a few a darker and a few a brighter which is difficult to understand). Not sure at all where this is coming from.

and now with a complete result for test 8
summary sorted.pdf
summary_new_4.pdf

This thread has been fascinating and educational. I am sure you already taught about it,
but just in case: it would be great to compare your final sky background brightness
distribution in each band to the values we assume in LSST forecasts elsewhere. Copying
from the overview paper: the expected median zenith sky brightness (AB mag/arcsec2)
in ugrizy is:  22.99	& 22.26	& 21.20	& 20.48	& 19.60	& 18.61.
If you want to go beyond the medians and compare the distributions, I recommend to
use Yoachim's sky brightness model values for each field in baseline opsim output.
If your images will be substantially off from the expected medians above, we should
think carefully about potential consequences for processing and later inferences.

The table below characterizes the roll-off toward the edges of the CCD e-images in the 18 test runs.  I filtered and stacked the e-images as described for the plots above.  The units are electrons/pixel.  The 'Reference' column is the average for CCD columns 1600-2400.  The columns 'X=0', 'X=40', and 'X=400' show how far below the reference value the average profile is at those locations.  So for example, streams 9, 13, and 14 have flat profiles until very near the edge, where they fall off steeply.  Each of the profiles has some roll-off.  Streams 6 and 7 have the least.

Stream
X=0
X=40
X=400
Reference

000000
9
6
0
704

000001
9
7
1
699

000002
11
8
0
703

000003
10
8
0
702

000004
9
7
0
703

000005
10
8
1
702

000006
6
4
0
705

000007
6
4
0
744

000008
18
6
0
745

000009
89
3
0
743

000010
15
11
1
741

000011
24
19
2
741

000012
14
10
1
742

000013
23
18
1
741

000014
84
2
0
704

000015
84
2
0
705

000016
10
8
1
701

000017
9
7
0
704

@sethdigel - thanks for this table.  It seems like a really useful way to quantify the edge effects.
In the spirit of quantifying the amount of structure in the images, which is the other aspect of interest (and indeed it's the one that led to this thread!) -- can we use the power spectrum?  In previous discussions, when we talked about comparing data vs. sims, the power spectrum seemed a bit annoying to implement because the structure in actual objects (which dominates on small scales) will differ in the data vs. sims, complicating the comparison.  So some kind of masking would be needed.  But in this case, we have 18 sims that all have the exact same sets of stars and galaxies, so their contribution to the PS should be the same, and we can use the difference in the power spectra between the 18 cases to quantify changes in the typical scales of the background fluctuations.
Does this make any sense?  Paging @fjaviersanchez @RobertLuptonTheGood @connolly @SimonKrughoff for some additional opinions.  My impression is that this would be a quick thing to implement if we don't have to mask out the objects, so if it's useful to do this then we might want to go ahead?
If we later want to tie this to data, then the PS might be difficult to use; but at least for comparisons between simulations that have the same sets of stars and galaxies, the PS is a more objective measure of structure than inspecting images with different scalings.

One last test added: test 13 with a bigger backgamma or test 16 without clouds. Again in order of model number and sorted by parameter variation. I had hoped that 18 would be much closer to 13 but the variation sensor by sensor is again puzzling.
summary_new_5.pdf
summary_sorted_2.pdf

In case anyone is interested, I ran processEimage.py on all of the available streams using the current master of obs_lsstSim and w_2018_07 of lsst_distrib.   The output repos are
/global/cscratch1/sd/jchiang8/DC2/phosim_Run1.1p/phosim_bg_studies/output_??

where the ?? indicates the stream number.  I've tried comparing the various catalog outputs, but nothing illuminating jumped out at me.
 For the Cls and w I tried to mask the sources (I just considered the pixels within the 5th and 95th percentiles). I used the test number 9 as reference and computed the ratio between each test and number 9, and subtracted 1. Some caveats: Instead of using the mosaic, I am pasting together the 9 images so some spurious power may arise. Also for the computation of P(k) I didn't mask anything. The notebook that I used to calculate this is here

Please, let me know if this is useful or if you think that this approach is wrong and I should change things to make it useful.

@fjaviersanchez those power spectra are great to have.  It would help to label them all in arcmin (possibly on a 2nd x axis) for easier comparison and to remind us what range of scales we're worrying about here, presumably 1-10 arcmin since sub-sensor background variations were the most obvious concern in the images.  Masking upper and lower extremes is a clever way to avoid objects, which should primarily contribute on smaller scales, but I wonder if objects noticeably affect >5% of pixels - you could test that by restricting to 10th-to-90th percentile instead.

@sethdigel suggested that this might give us more insight:

@egawiser, thanks for the tip. I will try to mask that regime as well. I checked the masks by eye given by the 95th percentile and they seemed to be more or less ok (example below -- blue masked, yellow unmasked):

In the case of the P(k) I am not masking any pixels and I am wondering if in the different values for the plateauing in the high-k regime that we are seeing show some problems with the noise due to the background optimization.

@fjaviersanchez Note that the calexps I created from these data have mask planes from source detection step that you could also use.

+1 for above.  You can get the mask from the calexp and then apply it to the eimage.

ok, thanks for all the hard work.  we are very close even though this is a confusing thread.  let me explain.
here are the simple conclusions:

keep airglowvariation off.  [i know what's wrong about it, by the way.]
keep clouds on.   they look good and subtle in comparing test 14 & 15, and the structure function is validated.
we need to do some version of "quickbackground".   all the run times are probably within the noise except 8 (which is basically close to no optimization) and 9, 14, 15 which are variations of "normal optimization".

so here's the subtle one i didn't quite expect (which is very good news):
4)  backgamma was apparently set too aggressively for quickbackground and is the source of the problem.  if you compare 16 & 17 you can see that 17 looks a lot better than 16.  furthermore, the normal optimizations look great (but are too slow) in 14 & 15.  those have optimizations as well, so its not an inherent problem with the optimization, its just a problem with being too aggressive with photon grouping in the sensor.
so something like 17 is probably ideal.   the power spectrum looks very flat.  i'd prefer just because of previous testing to keep beta near 4, but for completeness we could finish this off with the following
beta=4 gamma=1000
beta=4 gamma=300
beta=4 gamma=100
beta=6 gamma=1000
beta=6 gamma=300
beta=6 gamma=100
lower gamma & higher beta makes the approx better, and we simply could choose based on the power spectrum and run time tradeoff.  these would all be with "airglowvariation 0.0" and "backalpha 0.1" and "backdelta 1.0" and "activebuffer 1000"
the only other issue is the edges.  could someone look into the edges with the different values of the active buffer (compare run 13 & run 16?)?

bottom line is that backgamma is the problem for those who don't want to read all of this thread.

As discussed in today's telecon, I would be much happier only after a set of objective metrics have been stated as requirements and shown to be passed by the tests. This includes spatial variation due to clouds.
I would argue that we 1) keep all sources of spatial variation off, 2) agree on a set of tests (median sky background in ugrizy, etc. as suggested by Zeljko; comparison to actual observations from ESSENCE, HSC, DECAM; comparison of PhoSim-generated flats against data at BNL, etc.) under these conditions, and then check if 1) PhoSim run with no approximations passes the tests, and if so, whether runs with some approximate settings (for example, like the ones John mentioned above) also pass the tests. We cannot live in a world of randomly mucking around with parameters and saying something "looks good" or "does not look good".

there are formal phosim validation tests for background level (w/ either other data and various optimizations), clouds structure function, flats at BNL, airglow in IR.  so we have that.
the only thing we didn't have covered was this was the first big run with "quickbackground".   hence, this backgamma was set wrong.

I just started a new issue for discussing validation tests. Although it is titled PhoSim validation tests, obviously the issue is more general and applies to any other image production tool as well.

Thanks to Tom's hard work, we have another set of test results (19-23) -- 24 is almost finished.
summary_new_6.pdf In these cases backbeta and backgamma are varied.

I've updated my table (posted above in this issue thread) of the x-profiles to include the new streams.  Regarding 000013 and 000016, the latter rolls off about half as much at the edges.

Last update -- added test 24.
summary_new_7.pdf
@TomGlanzman It will be interesting to see the new timings. @sethdigel Is it easy to fill in the last 6 models in your table? @fjaviersanchez Could you add the power spectra (or better ratios) for the new runs? Maybe you can make a few plots with fewer models each to make it easier to see how the compare.  @johnrpeterson Any thoughts on the new results? Thanks all!

The chip to chip variation is still very large -- see test 22 for example.

Thinking back to Salman's comments on having a set of metrics to decide on
which configurations are acceptable what are the bounds on acceptable run
times for a single image using phosim for DC2 (given the compute time we
have available)
On Tue, Mar 13, 2018 at 8:27 AM, Salman Habib<EMAIL_ADDRESS>wrote:

The chip to chip variation is still very large -- see test 22 for example.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-372705749,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDow7Pbkg07iZ57OEJQodw50GIoJTAFks5td-VpgaJpZM4ScjMo
.

My thoughts (happy to hear other people's thoughts):
I don't have the time per image in my head but if you look at the table,
Run 1.1 was what we thought we can do just with the NERSC time (80M) and
Salman had added a 40% buffer there (so if one is optimistic ~50M). We
decided so far to not include airglow variation and clouds, that does
save a little bit of time. If you compare Run 1.1 with Test 9 (which is
still my favorite) the difference on the small test is 2.4 (Tom will
quickly warn us that the test is not necessarily telling us everything
about the timing so this might be too optimistic).
So that would get us to 120M which given that we do have quite a few
resources not impossible (100M at NERSC, and we can ask for more if we
finish that up early, 70M at Argonne (45M Argonne hours = 70 M NERSC
hours), resources in France and the UK).
And we can always ask for more at Argonne as well.
Due to the phosim team's hard work on threading and figuring out how to
run efficiently on the KNL we are far from the ~1B hour estimate that we
had after DC1.
In my mind, it's very important that we get good and useful results --
if we spend 50M hours on something we are not happy with in the end it's
a complete waste of resources.
So if it takes a few times more but the results are satisfying we should
go for that in my opinion.
On 3/13/18 9:35 AM, Andrew Connolly wrote:

Thinking back to Salman's comments on having a set of metrics to decide on
which configurations are acceptable what are the bounds on acceptable run
times for a single image using phosim for DC2 (given the compute time we
have available)
On Tue, Mar 13, 2018 at 8:27 AM, Salman Habib<EMAIL_ADDRESS>wrote:

The chip to chip variation is still very large -- see test 22 for
example.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub

https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-372705749,

or mute the thread

https://github.com/notifications/unsubscribe-auth/ABDow7Pbkg07iZ57OEJQodw50GIoJTAFks5td-VpgaJpZM4ScjMo

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-372708534,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jE0Us_WY2K-fY7S2YEatnq-Phl54ks5td-cxgaJpZM4ScjMo.

I've run processEimage.py on the most recent runs, i.e., through run 24.  The output repos are
/global/cscratch1/sd/jchiang8/DC2/phosim_Run1.1p/phosim_bg_studies/output_??

Re computer time and not wasting resources is another reason I would like us to be very conservative when running DC2. The result must be useful and free from frustrating artifacts. Better to be simple rather than add some complexity that ruins interpretation. Note that 100M hours of compute is ~$1M of taxpayer dollars so we do need to be responsible.

Updated P(k) with the latest runs and only showing those that are relatively close to run 9 (which I am considering the "truth"):

The procedure is the same as above: no masking (unlike for the Cls and w that I did mask), and rebinned images so that 1 pixel in the analyzed images was 16 x 16 in the original.

What is the binning in this P(k), @fjaviersanchez ?

I use linear binning with 400 bins each of size 1/pixel_size, where pixel_size=3.2 arcsec. I think I probably should try to make more bins to see the region of interest properly.

@SimonKrughoff kindly showed me how to make higher resolution images of the central raft that @katrinheitmann has been producing.  Below is a side-by-side comparison showing how dramatically different these images can be.  The originals bin 50x50 pixels, while the hi-res version is 2x2 pixels.  Although eyeballing is not a particularly objective technique, the edge roll-off and sensor-to-sensor variation are much diminished in the new images.

High resolution central raft images are available for tests 000009, 000016 and 000018, along with the default (50x50) versions:    /global/cscratch1/sd/glanzman/output__, where  = 000009, 000016 or 000018, and  = 50 or 02.

So, I looked at these images and it's really odd that the chip to chip variation disappeared at the highest resolution. This means one can't trust the coarse-grained visualization for eyeball tests. On the other hand, didn't Seth do his edge effect analysis at full resolution?

@salmanhabib I can see the same background variations at the highest resolution - they are small amplitude effects versus pixel-to-pixel noise and sources so become much clearer when the binsize gets bigger.  I don't actually see interchip variations in either, but the strongest intrachip fluctuations (by eye) at low resolution are also the strongest ones at high resolution.

Seth rebinned the data using 8x8 pixel bins.  If you rebin at that level, the rolloffs are clear, even when looking at a single (rebinned) row in a single sensor.

@salman I believe Seth indicated he used 8x8 binning.  Below is another ds9 screen shot showing the same region of the sky (R22_S02) for test 000018 but with four different binnings (starting LOWER-LEFT and moving clockwise): 50x50, 20x20, 8x8, 2x2.  The top and left edges are sensor edges.  These images were all produced using Simon's makeFpSummary.py script.

And just for fun, here is the raw phoSim image of the same part of the sky:

I hope I'm not throwing a spanner into our plans, but I'm seeing some troubling behavior in the pixel distributions for the "full background" runs, i.e., 9, 14, and 15.  Here are histograms of the ADU values per pixel at 4 different rebinnings (taking the mean over the binned pixels to keep the x-axis the same):

For comparison, I've also plotted the distributions for runs 4 and 8, which look similar to the other runs in the 1x1 case.  Does anybody know what the shoulders or secondary peaks below the main peak arise from in the full background cases?

I think the shoulders/secondary peaks are from the treering-like structure commented on in the #desc-dc2-eyeballs channel.  The non-full background runs seem to lack them.

But they should have the tree rings as well, no? But you are correct
that the tree ring like structures are not obvious in the other images
either.
On 3/13/18 5:22 PM, James Chiang wrote:

I think the shoulders/secondary peaks are from the treering-like
structure commented on in the #desc-dc2-eyeballs channel. The non-full
background runs seem to lack them.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-372852205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jDTMVYwnUvs0hUmwgGMcKnMSG38Jks5teFTLgaJpZM4ScjMo.

@jchiang87 - another option: can you plot the locations of the pixels in the shoulder?  THe full background runs are the only ones that have such a sharp dip in the background level at the very edges, which could produce such a shoulder, right?  If that's the right explanation then they should be clearly isolated at the edges, whereas if it's tree rings then they will be in a tree ring-y pattern.

But they should have the tree rings as well, no?

Right.  It's hard to turn that off. You have to be explicit about it as we were discussing in slack.
Rachel's idea is good.  Even, easier check: if you make the same plot for the central part of the sensor are the edges still there?

Has it been confirmed that the quick background optimization preserves the tree rings?  It seems to me that bunching and unbunching photons has the potential to erase small-scale structures from the sky background, which could explain why they are showing up only in the runs with full background.  But I presume this has been checked.

I was just looking at this.  Yes, the low pixels in 09 are all around the edges, where the steep roll-off is.  Here is an image of the S11 sensor FITS image, rebinned down by 8x in both dimensions, showing in white the pixels that are < 650 electrons.  (The divot in the lower right is just an artifact of how the windows are displayed in my system.)

Oh

Has it been confirmed that the quick background optimization preserves the tree rings? It seems to me that bunching and unbunching photons has the potential to erase small-scale structures from the sky background, which could explain why they are showing up only in the runs with full background. But I presume this has been checked.

Sorry... I misread above.  I thought the full background didn't have them.  Yes we know that the quickbackground wipes them out.

Okay, progress!  So:

the shoulders are due to the edge roll-off, which Alex D-W seems to be saying may be a real effect due to fringing fields modifying the effective pixel sizes (at least there is such a thing in DECam if I understand correctly).  It would be good to confirm if that is what this is!  Can somebody please confirm whether such an effect is in PhoSim?  Incidentally that could explain why it's so much stronger with full background mode: because it's a small-scale effect that gets washed out by the optimization in quickbackground mode.

and the tree rings issue with full background showing them and quick background not showing them is expected for the same reasons

There should be a roll off right at the edge these are due to the electrostatic potentials held at the edges. It is usually pretty close to the edge though...  I might have some old plots where I was looking at this.

There is definitely edge rolloff in both vendor's CCDs.
C
Christopher Stubbs
Samuel C. Moncher Professor of Physics and of Astronomy
17 Oxford Street
Harvard University
Cambridge MA 02138
<EMAIL_ADDRESS>Think this email is terse? Here's why:
http://www.emailcharter.org
On Mar 13, 2018, at 8:05 PM, Rachel Mandelbaum<EMAIL_ADDRESS>wrote:

Okay, progress! So:
the shoulders are due to the edge roll-off, which Alex D-W seems to be saying may be a real effect due to fringing fields modifying the effective pixel sizes (at least there is such a thing in DECam if I understand correctly). It would be good to confirm if that is what this is! Can somebody please confirm whether such an effect is in PhoSim? Incidentally that could explain why it's so much stronger with full background mode: because it's a small-scale effect that gets washed out by the optimization in quickbackground mode.
and the tree rings issue with full background showing them and quick background not showing them is expected for the same reasons
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

The left should be the astrometric shift (charge sees the potential and gets pulled to the edge).  The right is (I believe) lab data taken by Paul O'Conner showing the flux roll off with both flats and spots.

This is pretty old and I bet Chris has better data for this but Paul
OConnor had a publication showing this effect see Figure 1
https://arxiv.org/pdf/1402.5454.pdf
On Tue, Mar 13, 2018 at 5:15 PM, Chris Walter<EMAIL_ADDRESS>wrote:

[image: screen shot 2018-03-13 at 8 13 30 pm]
https://user-images.githubusercontent.com/5562740/37376527-0db1a97c-26fb-11e8-8149-61b19a1e2682.png
The left should be the astrometric shift (charge sees the potential and
gets pulled to the edge). The right is (I believe) lab data taken by Paul
O'Conner showing the flux roll off with both flats and spots.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-372862421,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDow-5anyDcR5mEfhYVAasCL1HKsMkSks5teGEIgaJpZM4ScjMo
.

Just for fun, here are Tom's high-res images for Test 18, 16, and 9. (For 18 also the other resolution). You can clearly the the "tree rings" in Test 9, but not really in 16 and 18.
phosim_res_test_2.pdf

Sorry, was away from my computer.....
Another thing to note about tree rings is that they should redistribute charge, so there should be a symmetric secondary set of features on the high signal side of the median pixel value, which isn't apparent at the same level, so it makes sense that those shoulders are from the edge effects.  Regarding the electrostatic edge rolloff, I believe it is in phosim, but the effects we've been seeing in these sims seem to extend much further into the CCD than the lab results (100s of pixels in Seth's profiles vs ~20 pixels, as shown from Chris's plot).   Also, changing the activebuffer size past a certain point should not affect the electro-static edge rolloff.

I don't see how this effect is 100s of pixels in Seth's profiles with full background, though?  Seth, can you please confirm the size of the feature in test 9?  (I think we should only be evaluating distance scales of features like this in the full background sims at this point, since we know quick background mode should smooth out the feature.)

Extent of the surface brightness perturbation is less than ~10 pixels in extent, in lab tests.
C
Christopher Stubbs
Samuel C. Moncher Professor of Physics and of Astronomy
17 Oxford Street
Harvard University
Cambridge MA 02138
<EMAIL_ADDRESS>Think this email is terse? Here's why:
http://www.emailcharter.org
On Mar 13, 2018, at 9:10 PM, Rachel Mandelbaum<EMAIL_ADDRESS>wrote:

I don't see how this effect is 100s of pixels in Seth's profiles with full background, though? Seth, can you please confirm the size of the feature in test 9? (I think we should only be evaluating distance scales of features like this in the full background sims at this point, since we know quick background mode should smooth out the feature.)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-371728139
Shows a gentle, but significant rolloff in run 9.  There is a steeper part in that profile that seems consistent in extent with the observed rolloff in the lab.

I think that Jim is referring to the plot that I posted in this thread 5 days ago, here.  The profile for stream 09 is the flattest up until close to the edge, and then rolls off steeply.  In the profile, it is clearly rolling off gently starting ~500 pixels from the edge, before it drops sharply at about 30 pixels from the edge.  The profiles are binned by 8 pixels in x, so the step size is 8 pixels in the plot.

I see.  I read that gentle slope as being due to something else, considering the overall variation is ~0.5% over a large number of pixels, so I was only reading the steep part as being part of the edge roll-off.

@jchiang87 - is your concern that the edge roll-off model in PhoSim is unphysical compared to real devices, or that there's some other issue interfering with it?  (i.e., that the sims have edge roll-off but they also have some other effect near the edges)
I did some poking around and learned that edge roll-off is incorporated in PhoSim (we even refer to it a few times in this repo when talking about what should be on/off for Run 1.1 in e.g. #66 ) but was unable to find much info about the model, so I can't tell whether it's something that could easily be adjusted to look more like the lab data or what.  Perhaps @johnrpeterson could comment on this.
If it looks unphysical in form, in a way that is going to be challenging to model, then perhaps the best thing to do for DC2 is turn it off?

The BNL folks must know what the CCD behavior is regarding the edge effects (@astrostubbs as well, see 10 pixel remark above). All we need is for them to send us the data. Either PhoSim can do this right, or we mask it.
Additionally, we can run PhoSim flats with the same settings as in the main run to take out this effect before sending the data on to DM (@johnrpeterson had mentioned this earlier).

We can easily mask the roll-off in processing.  I think the questions are:
a) Will the flats correct for this to a level where we don't need to mask
b) If not, can we tune phosim so the border we need to mask is taking a realistic bite out of the total number of pixels we have to work with.

I did some poking around and learned that edge roll-off is incorporated in PhoSim (we even refer to it a few times in this repo when talking about what should be on/off for Run 1.1 in e.g. #66 ) but was unable to find much info about the model, so I can't tell whether it's something that could easily be adjusted to look more like the lab data or what. Perhaps @johnrpeterson could comment on this.

Yes, it is in the model (and caused the astrometric shift in the picture I posted above). I suspect but am not sure it is controlled by
edgeSurfaceCharge -5e9
in the silicon.txt file in the instrument directory but don't know the details of how it works and what the units are there.  I believe it is part of the static Poisson calculation that is done with the fixed potentials when the model is initialized.

I agree with @SimonKrughoff's comment about masking. I think it would be conservative to choose an edge mask that is large enough to be robust against whether we are completely modeling the edge effects. This is a tricky area and astrometric features from edge distortions may mean that we don't want to use these pixels for precision analyses.
Simon and I also discussed the fact that the simulation of sky background photons might not be completely accurate at the edges (something about discrete Gaussians not extending beyond the sensor edges). A larger mask width could also help mitigate some of these issues.

What's the maximum edge masking we can get away with in terms of effect on science? I suspect PhoSim is not getting the edge effects right at the level of a few 10's of pixels from the edge. Some of this can be mitigated by running flats. However, I want a number for the worst-case masking, so we can figure out what we need to do next.

@salmanhabib if we dither strongly, this comes down to just a reduction in effective field of  view.  If we mask 50 pixels, this is a ~5% reduction in overall field of view.  So, will result in a 5% reduction in coverage.

Thanks, @SimonKrughoff I'll check that off the front burner list then.

To respond to @rmandelb 's question, I'm hoping (along with everyone else I'm sure) that we can mitigate both edge effects we see in the run 9 profile that Seth generated, i.e., the steep part that looks like the laboratory-observed electrostatic rolloff and the more gentle sloping rolloff that can probably be handled by adjusting phosim parameters related to the sky background.  Regarding the electrostatic rolloff specifically, I would like to see a phosim simulation that includes really flat illumination (such as we can obtain in the laboratory) along with an array of spots, so that we can characterize both the rolloff in flux towards the edges and the astrometric effects, and be sure that they are both maskable in a reasonable way, i.e., not extending too far into  the sensor.  If they are not, then I suggest we turn off the electrostatic edge rolloff part.

ok, three more tests -- this time, test 9 settings (no quickbackground) without defects and (i) activebuffer 10, (ii) activebuffer 600, and (iii) activebuffer 1000. I am surprised that they are not closer to the original test 9.
summary_new_8.pdf

so, it looks like the backgamma is the critical parameter.
based on both the timings and the fidelity, we should use:
airglowvariation 0.0
backalpha 0.1
backbeta 6.0
backdelta 1.0
activebuffer 1000.0
and then we should set backgamma based on what is at least a factor of ~3 below realistic background variations.  visually, it looks like
backgamma 300.0
would be the best for now, but we could test it quantitatively.

I've updated the table of edge roll-offs (posted above, four days ago) to include the new streams<PHONE_NUMBER>27.  Streams 26 and 27 have essentially no roll off at all, with profiles decreasing by only one electron/pixel at the edge.

Thanks! Do you actually also have the profile plots? I think they would be very valuable in addition to the table.

And test 28 finished -- same as test 9 but no sources. Now the tree rings jump out. I don't understand why the center one is so much brighter than the rest.
summary_new_9.pdf

Thank you, Katrin.  I've updated the table of profiles to include stream 28.  Because the table is hard to find now, I've copied it below along with the original description.  Streams 26 and 27 have essentially no roll off.
I also include plots of the profiles for the streams.  I've divided them into two groups (with and without clouds), and have limited the y axis ranges to try to make them readable.  For the streams with large roll-offs at the edges, the full range is not shown.

The table below characterizes the roll-off toward the edges of the CCD e-images in the 19 test runs.  I filtered and stacked the e-images as described for the plots above.  The units are electrons/pixel.  The 'Reference' column is the average for CCD columns 1600-2400.  The columns 'X=0', 'X=40', and 'X=400' show how far below the reference value the average profile is at those locations.  So for example, streams 9, 13, and 14 have flat profiles until very near the edge, where they fall off steeply.

Stream
X=0
X=40
X=400
Reference

000000
9
6
0
704

000001
9
7
1
699

000002
11
8
0
703

000003
10
8
0
702

000004
9
7
0
703

000005
10
8
1
702

000006
6
4
0
705

000007
6
4
0
744

000008
18
6
0
745

000009
89
3
0
743

000010
15
11
1
741

000011
24
19
2
741

000012
14
10
1
742

000013
23
18
1
741

000014
84
2
0
704

000015
84
2
0
705

000016
10
8
1
701

000017
9
7
0
704

000018
7
5
0
744

000019
11
8
1
705

000020
10
8
0
704

000021
9
6
0
705

000022
11
9
2
704

000023
11
9
2
704

000024
10
7
1
705

000025
6
5
1
744

000026
1
0
0
745

000027
1
0
0
745

000028
92
6
1
741

And test 28 finished -- same as test 9 but no sources. Now the tree rings jump out. I don't understand why the center one is so much brighter than the rest.

The increment in brightness for the central CCD is about 2 electrons/pixel; the smoothing and scaling increases the contrast a lot.

I've run processEimage.py on runs 25, 26, 27.  I also added the makeFpSummary.py task to my little pipeline so those runs have the focal plane summary image for the eyeball tests.  As usual, everything is in /global/cscratch1/sd/jchiang8/DC2/phosim_Run1.1p/phosim_bg_studies/output_??.   I'll run the fp summary script for runs 0-24.

Four new tests complete,<PHONE_NUMBER>32, all with NO sources in the catalog: background only.  Only the phoSim background parameters have been tweaked.  I am now routinely running Simon's focal plane summary with 2x2 binning, here: /global/cscratch1/sd/glanzman/output__02 .

I've updated my roll-off plots and table above to include streams<PHONE_NUMBER>32.

Thanks @sethdigel ! Can you make one more plot where you just show the pairs 9/28, 23/29+30, 17/32, and 13/31?

Here are plots with just those pairs.  I ended up making separate plots for clouds/no clouds.  So the profiles for streams 29 and 30 are in different plots.  The y-axis range spans 41 electrons/pixel in both plots, so the shapes of the curves can be compared between plots.

@sethdigel Nice, thanks very much!

I'm confused why the edge distortion goes away when there are clouds.  Am I missing something?

I've done a simple comparison of the background variation in some of the tests vs. a real DECam exposure.  Specifically, I took the median counts in 1000x1000 pixel boxes, computed the standard deviation and the mean, and expressed the final result as a fractional variation:

test
std. dev. / mean of median counts in 1000x1000 pixel boxes

000001
0.011

000009
0.0016

000014
0.0029

000017
0.0020

000023
0.0029

DECam exposure 328612
0.0030

Most of the tests were selected from Katrin's "promising candidates" from her slide 27 for today's DC2 telecon. The intent is to check by a simple metric if the background variation is below that in real data, which appears to be the case except for the problematic test 1, which is much higher.  The selected DECam exposure is r band, with 100 sec exposure time (also note a DECam pixel is about 0.26");  more and/or shorter DECam exposures could also be selected for this kind of check.

@hlin730 Many thanks -- just the kind of thing we need!

I found a bug in my code (I was missing a 1/Npixels factor in the k-axis). These are the results with the new version:

I put the code here: https://github.com/fjaviersanchez/quick_pk_bkg/quick_pk.py. There are some instructions about how to use it in the README.md file.

This looks more reasonable, thanks @fjaviersanchez .

@hlin730 Nice.  Thanks!
Can you give us sense of the counts in the DECam case?  I'm curious about the statistical error so we know at what level we can do comparisons.
When you say: "I took the median counts in 1000x1000 pixel boxes, computed the standard deviation and the mean" does that mean you split the LSST sensor into 4 pieces and calculated the median count rate of the pixels in each of those 4 sections and then calculated the std dev and mean of those 4 numbers? Or do the 1000x1000 pixel boxes slide over the sensor so that each box is not completely statistically independent?
Also, since the PhoSim exposure is 30 seconds, are they rates scaled to the DECam exposure?

[It could be a coincidence but test 9 and the DECam exposure are different in ratio by the sqrt(100 sec/30 sec)]

Oh ignore the last comment.. I did it reversed. The fractional error would be smaller for the larger exposure if it was just statistical. Time for sleep..

@hlin730 I've selected a set of short (3s to 15s) DES standard star exposures if you want to explore shorter exposure times. Since all standard star exposures are public, I think that I'm free to put these on NERSC if others would be interested in taking a look (add a thumbs up to this comment if this would be helpful).

The last two tests, 000033 and 000034, motivated by yesterday's DC2 meeting, are complete.  Please have a look.

I've updated the roll-off table and plots that I posted yesterday to now include streams 33 and 34.  They are quite flat until about 30 pixels from the edge.  By the edge, they roll off by 11-12%.  I don't know enough to say whether that is a physical effect being simulated or an artifact of the combination of parameters.  In terms of photoresponse requirements for CCD acceptance for the Camera, it would take a 20% deficit for a pixel to be flagged as bad.

  Background only (NO catalog sources) with four of our favorite test configurations, including "standard optimization" background settings but with larger activebuffer, with and without clouds, with and without tree-rings/brighter-fatter.

I've updated the profile plots and table above to include streams 35-38.

@cwwalter Yes, I did split each LSST e image into 4 non-overlapping 1000x1000 pixel boxes and then calculated things as you indicated.  No scaling of either the DC2 images or the DECam images was done.   The DECam counts (in the 100 sec exposure I used) was about 1500 electrons.
@kadrlica Yes, I can try a similar check using the shorter DECam standard star exposures.

@katrinheitmann asked me to calculate difference maps between Phosim test streams that had no stars or galaxies and the background model maps calculated by the DM stack for corresponding test streams that did include stars and galaxies.  This is by way of evaluating the background models.
The images below are for the S11 sensor.  They have been smoothed with a 10-pixel Gaussian kernel, and are scaled linearly from -10 to +10 electrons/pixel.  The FITS versions are on cori at ~digel/DC2/images/diff*.fits.  Sorry that the png image dimensions are not all the same.
(a): The stream number for the run with no sources
(b): Stream number for the run that had sources and was processed by DM stack to generate a background model map.
(a) - (b):  The difference map

Stream (a)
Stream (b)
(a) - (b)

35
33

36
34

37
26

38
27

@sethdigel Thanks Seth! After chatting with Salman,  we are a little confused though that you still see tree rings in the first two. Here is the thought: DM can't really take out the tree rings so DM bkgd should still have them. PhoSim no sources will have the tree rings as well. So if you subtract DM bkgd (with tree rings) from PhoSim no source (with tree rings) shouldn't the treerings sort of cancel out? I downloaded your fits files and looked at them, and indeed I don't see the tree rings. We also measured the pixel distribution in the difference image and it seems consistent with just noise. The images are in this pdf file (including calexp files for 33 and 34 from Simon):
33-34_invest.pdf
It would be great if the expert could comment on how this now all look like (the profiles up above, the outputs from the calexp and the difference between the phosim-no-source and DM background estimate). For the non-expert (=me) this looks like pretty convincing!

@katrinheitmann If you smooth the Phosim images and set the scaling to span a narrow range around the image average, you will see the tree rings (if the simulation included tree rings).  I did this, but did not post the images.  If you do not smooth them enough or set the scaling right, you will not see them.  Similarly, if you smooth the difference images and show a narrow range around zero, you will see the tree rings, as the images I posted show.  The reason that they show up in the difference images (if the Phosim simulation had them in the first place) is that the DM background map includes only lower-frequency structure.  As Simon mentioned, the assumption seems to be that tree rings will be removed by flat fielding (with flats), and it is not the intent that the DM background fits would pick them up.

I don't think the DM background model is capable of modeling the tree rings.  It's a 6th order polynomial fitted to representative values in 128x128 bins.  The tree ring structure is on too small a scale to be captured by that sort of model.

ok, thanks! I guess I was under the wrong impression that DM would leave
the treerings with the background untreated.
So we have (naively) from DM = sources + background + treerings and now
the treerings are kept in the file that has the sources (calexp and I
have seen the treerings in there). And thanks Seth for letting me know
about the resolution part.
On 3/19/18 8:05 PM, James Chiang wrote:

I don't think the DM background model is capable of modeling the tree
rings. It's a 6th order polynomial fitted to representative values in
128x128 bins. The tree ring structure is on too small a scale to be
captured by that sort of model.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-374437429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jJjtD-o2YNGH17wGWXyUmrTpixprks5tgFXugaJpZM4ScjMo.

ok, thanks! I guess I was under the wrong impression that DM would leave
the treerings with the background untreated.

I think it is not totally decided yet and an expert can comment, but depending on the scale the tree rings may be handled by a WCS transformation that shifts the charge. So it is separated from the background model. See the discussion here where I was trying to understand this better:
https://lsstc.slack.com/archives/C77DDKZHR/p1521236405000143

The tree rings are variation in the effective size of the pixels.  The "proper" way to get rid of them it to move charge between pixels to correct for this, and I'm thinking about doing so (despite the resulting correlations).  At present we (and DES?) correct for them by pretending that they are QE variations and taking them out by dividing by a flat field.
This leads to e.g. astrometric residuals, but if the tree ring scale is large compared to objects then you can handle them this way (you'd have to include them in the PSF models too, of course -- actually, that'd fix the astrometry too).
I suppose that you could subtract them out as a background, but that's no better (and maybe worse?) than flat fielding them out, and much harder to do.
In short:  you're seeing them because we don't have flat fields from phoSim, and we are not yet handling them in the more sophisticated ways that we're planning.

What about the rest? I am not that concerned about the treerings (sorry
if my comment was confusing and we can always switch them off if we want
to) -- I was wondering if otherwise these results are now acceptable?
On 3/20/18 8:17 AM, Robert Lupton wrote:

The tree rings are variation in the effective size of the pixels. The
"proper" way to get rid of them it to move charge between pixels to
correct for this, and I'm thinking about doing so (despite the
resulting correlations). At present we (and DES?) correct for them by
pretending that they are QE variations and taking them out by dividing
by a flat field.
This leads to e.g. astrometric residuals, but if the tree ring scale
is large compared to objects then you can handle them this way (you'd
have to include them in the PSF models too, of course -- actually,
that'd fix the astrometry too).
I suppose that you could subtract them out as a background, but that's
no better (and maybe worse?) than flat fielding them out, and much
harder to do.
In short: you're seeing them because we don't have flat fields from
phoSim, and we are not yet handling them in the more sophisticated
ways that we're planning.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LSSTDESC/DC2_Repo/issues/140#issuecomment-374593414,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMQ9jN_Vsnu44sMhXJmyuykyX61pcSflks5tgQFggaJpZM4ScjMo.

I looked at the autocorrelation of the background-only simulation in stream 37 and it seems sensible.  Here is a plot of the sum of the product of the image for S11 with itself shifted by N pixels to the right or up.  The curves overlap and any correlation is limited to offsets of only one or two pixels.

Out of curiosity I looked at the same for stream 26, which has the same background + sources.  The red curve is for shifts in y (rows) and the green dashed curve is for shifts in x (columns).  I am not understanding why the latter has such a long tail.

@sethdigel could it be bleed trails from saturated stars?

Thanks, @kadrlica, that is it entirely.  Also, I had mixed up 'row' and 'column' because the PhoSim images are oriented in the Camera coordinate system, which has the CCD parallel transfer direction horizontal.  The saturated stars really dominate the autocorrelation, and the longest bleed trail in the S11 image has an end-to-end length of about 320 pixels.

Closing this issue as a new set of phoSim background configuration parameters has been agreed upon and Run 1.2p has commenced.