[BUG] LoadBalancer stuck Pending in the Import Existing RKE2 case

Describe the bug
In the Import Existing RKE2 case,we encounter LB can not deploy due to Error syncing load balancer: failed to ensure load balancer: wait ip timeout

To Reproduce

Create VM on Harvester with pre-gen cloud-provider & csi-driver cloud-config

cloud-providercurl -sfL https://raw.githubusercontent.com/harvester/cloud-provider-harvester/master/deploy/generate_addon.sh | bash -s RKE2_CLUSTER_NAME default

csi-drivercurl -sfL https://raw.githubusercontent.com/harvester/harvester-csi-driver/master/deploy/generate_addon_csi.sh | bash -s RKE2_CLUSTER_NAME default RKE2

SSH to VM and setup RKE2 manually

Ref. https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher#1-install-kubernetes-and-set-up-the-rke2-server
/etc/rancher/rke2/config.yaml should betoken: my-shared-secret
tls-san:
  - my-kubernetes-domain.com
  - another-kubernetes-domain.com
disable-cloud-controller: true

Rancher -> Cluster Management -> Import Existing -> Generic

In Agent Environment Vars, add cattle-cluster-agent : preferredDuringSchedulingIgnoredDuringExecution
Following command to let join VM.

Install harvester-cloud-provider from store

Create nginx deployment and LB direct to it.
${\color{red} \text{Status should stuck at Pending.}}$

Expected behavior
Load balancer should be successfully deploy and forward to nginx deployment.
Support bundle
supportbundle_lb_2024-03-12T10-42-20Z.zip
Environment

Harvester

Version: v1.3.0-rc4
Profile: baremetal, 1 nodes (48C/128G/2T)
ui-source: Auto

Rancher

Version: v2.8.2
Profile: rancher-vcluster (1C/4G)
harvester-cloud-provider: 103.0.0+up0.1.14

harvester-cloud-provider 103.0.1+up0.2.3 is in rancher/charts dev-v2.8 branch but not in release-v2.8 branch yet.

dev-v2.8: https://github.com/rancher/charts/tree/dev-v2.8/assets/harvester-cloud-provider
release-v2.8: https://github.com/rancher/charts/tree/release-v2.8/assets/harvester-cloud-provider

I think that's why we only have 103.0.0+up0.1.14 to choose from on the marketplace. cc @w13915984028

Since harvester-cloud-provider 103.0.1+up0.2.3 and 103.0.2+up0.2.4 were already there, we can verify again to see if the issue persists. Thanks. cc @albinsun

Now the latest 103.0.2+up0.2.4 is also available on release-v2.8.
https://github.com/rancher/charts/blob/release-v2.8/assets/harvester-cloud-provider/harvester-cloud-provider-103.0.2%2Bup0.2.4.tgz

Having a quick test using harvester-v1.3.1-rc2 + rancher-2.8.4 + harvester-cloud-provider-v0.2.4.

LB works

But we observe following uncommon behavior:

When adding LB, no configurable IPAM field (DHCP or Pool)

LB works
Implicitly use IPAM Pool by default

Is this expected?

@albinsun Is it about Add-on tab is missing and you can't set IPAM field? By default IPAM is set to pool?

@albinsun Is it about Add-on tab is missing and you can't set IPAM field? By default IPAM is set to pool?

Yes that's what I mean.

Can create LB and working but IPAM is not configurable
From the assigned IP it's from Pool

For imported existing guest clusters, the dashboard does not show the IPAM tab. We need insights from frontend developers to know what the missing bits are to show such a tab. cc @a110605 @torchiaf
With that said, I feel that this is not related to the original issue. As long as you can get a VIP from either the pool or DHCP server, the LB CR will not be stuck in the pending state.

Agree with @starbops, we can use another ticket to track the UI issue and close this one.

Close as the original issue is resolved and will track the IPAM missing issue in harvester/harvester/issues/6122.


⚠ Bofrost.de has degraded performance

In 78fe71a, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1016 ms

Resolved: Bofrost.de performance has improved in d60dc1e.


Run the correct command for the sidekiq docker container

The Sidekiq container produced was still running the default command bin/rails server.
About the newline at the end of the file, do you have a preference? My editor is configured to add one automatically, but I'm happy to remove it if you want.

Good catch!
I switch back and forth between vim (which adds a newline) and vscode (which does not).  All I care about is the tests passing.  :-)


Discussion on restructuring Event Container structure

Since it's come up a few times that things in the current Event container structure are a bit haphazard, let's start a discussion on what should move.  For reference, here is what it is now:

event_type Event type
index event indexing information

event_id event identifier
obs_id observation identifier

r0 Raw Data

obs_id observation ID
event_id event id number
tels_with_data list of telescopes with data
tel[X] map of tel_id to R0CameraContainer

trigger_time Telescope trigger time, start of waveform readout, None for MCs
trigger_type camera's event trigger type if applicable
num_trig_pix Number of trigger groups (sectors) listed
trig_pix_id pixels involved in the camera trigger
waveform numpy array containing ADC samples(n_channels, n_pixels, n_samples)

r1 R1 Calibrated Data

obs_id observation ID
event_id event id number
tels_with_data list of telescopes with data
tel[X] map of tel_id to R1CameraContainer

trigger_time Telescope trigger time, start of waveform readout
trigger_type camera trigger type
waveform numpy array containing a set of images, one per ADC sampleShape: (n_channels, n_pixels, n_samples)
selected_gain_channel Numpy array containing the gain channel chosen for each pixel. Shape: (n_pixels)

dl0 DL0 Data Volume Reduced Data

obs_id observation ID
event_id event id number
tels_with_data list of telescopes with data
tel[X] map of tel_id to DL0CameraContainer

trigger_time Telescope trigger time, start of waveform readout
trigger_type camera trigger type
waveform numpy array containing data volume reduced p.e. samples(n_pixels, n_samples). Note this may be a masked array, if pixels or time slices are zero-suppressed

dl1 DL1 Calibrated image

tel[X] map of tel_id to DL1CameraContainer

image Numpy array of camera image, after waveform extraction.Shape: (n_pixel)
pulse_time Numpy array containing position of the pulse as determined by the extractor.Shape: (n_pixel, n_samples)

dl2 Reconstructed Shower Information

shower[X] Map of algorithm name to shower info

alt reconstructed altitude [deg]
alt_uncert reconstructed altitude uncertainty [deg]
az reconstructed azimuth [deg]
az_uncert reconstructed azimuth uncertainty [deg]
core_x reconstructed x coordinate of the core position [m]
core_y reconstructed y coordinate of the core position [m]
core_uncert uncertainty of the reconstructed core position [m]
h_max reconstructed height of the shower maximum
h_max_uncert uncertainty of h_max
is_valid direction validity flag. True if the shower directionwas properly reconstructed by the algorithm
tel_ids list of the telescope ids used in the reconstruction of the shower
average_intensity average intensity of the intensities used for reconstruction
goodness_of_fit measure of algorithm success (if fit)

energy[X] Map of algorithm name to energy info

energy reconstructed energy [TeV]
energy_uncert reconstructed energy uncertainty [TeV]
is_valid energy reconstruction validity flag. True if the energy was properly reconstructed by the algorithm
tel_ids array containing the telescope ids used in the reconstruction of the shower
goodness_of_fit goodness of the algorithm fit

classification[X] Map of algorithm name to classification info

prediction prediction of the classifier, defined between [0,1], where values close to 0 are more gamma-like, and values close to 1 more hadron-like
is_valid classificator validity flag. True if the predition was successful within the algorithm validity range
tel_ids array containing the telescope ids used in the reconstruction of the shower
goodness_of_fit goodness of the algorithm fit

mc Monte-Carlo data

energy Monte-Carlo Energy [TeV]
alt Monte-carlo altitude [deg]
az Monte-Carlo azimuth [deg]
core_x MC core position [m]
core_y MC core position [m]
h_first_int Height of first interaction
x_max MC Xmax value [g / cm2]
shower_primary_id MC shower primary ID 0 (gamma), 1(e-),2(mu-), 100A+Z for nucleons and nuclei,negative for antimatter.*
tel[X] map of tel_id to MCCameraEventContainer

photo_electron_image[X] reference image in pure photoelectrons, with no noise
🔴 photo_electron_image[X] (ERROR): this is a Map that couldn't be accessed: 'NoneType' object is not callable
reference_pulse_shape reference pulse shape for each channel
time_slice width of time slice [ns]
dc_to_pe DC/PE calibration arrays from MC file
pedestal pedestal calibration arrays from MC file
azimuth_raw Raw azimuth angle [radians from N->E] for the telescope
altitude_raw Raw altitude angle [radians] for the telescope
azimuth_cor the tracking Azimuth corrected for pointing errors for the telescope
altitude_cor the tracking Altitude corrected for pointing errors for the telescope

mcheader Monte-Carlo run header data

run_array_direction the tracking/pointing direction in [radians]. Depending on 'tracking_mode' this either contains: [0]=Azimuth, [1]=Altitude in mode 0, OR [0]=R.A., [1]=Declination in mode 1.
corsika_version CORSIKA version * 1000
simtel_version sim_telarray version * 1000
energy_range_min Lower limit of energy range of primary particle [TeV]
energy_range_max Upper limit of energy range of primary particle [TeV]
prod_site_B_total total geomagnetic field [uT]
prod_site_B_declination magnetic declination [rad]
prod_site_B_inclination magnetic inclination [rad]
prod_site_alt height of observation level [m]
prod_site_array site array
prod_site_coord site (long., lat.) coordinates
prod_site_subarray site subarray
spectral_index Power-law spectral index of spectrum
shower_prog_start Time when shower simulation started,
CORSIKA: only date
shower_prog_id CORSIKA=1, ALTAI=2, KASCADE=3, MOCCA=4
detector_prog_start Time when detector simulation started
detector_prog_id simtelarray=1
num_showers Number of showers simulated
shower_reuse Numbers of uses of each shower
max_alt Maximimum shower altitude [rad]
min_alt Minimum shower altitude [rad]
max_az Maximum shower azimuth [rad]
min_az Minimum shower azimuth [rad]
diffuse Diffuse Mode On/Off
max_viewcone_radius Maximum viewcone radius [deg]
min_viewcone_radius Minimum viewcone radius [deg]
max_scatter_range Maximum scatter range [m]
min_scatter_range Minimum scatter range [m]
core_pos_mode Core Position Mode (fixed/circular/...)
injection_height Height of particle injection [m]
atmosphere Atmospheric model number
corsika_iact_options Detector MC information
corsika_low_E_model Detector MC information
corsika_high_E_model Detector MC information
corsika_bunchsize Number of photons per bunch
corsika_wlen_min Minimum wavelength of cherenkov light [nm]
corsika_wlen_max Maximum wavelength of cherenkov light [nm]
corsika_low_E_detail Detector MC information
corsika_high_E_detail Detector MC information

trig central trigger information

gps_time central average time stamp
tels_with_trigger list of telescopes with data

count number of events processed
inst instrumental information

subarray SubarrayDescription from the instrument module

pointing[X] Telescope pointing positions

azimuth Azimuth, measured N->E [rad]
altitude Altitude [rad]

Does anyone remember why event.mc.tel[x].photo_electron_image is a Map, not  Field containing an ndarray? (any why the Map is not a map to any specific type, which seems to break things?)

I think the docu of tels_with_data could be updated to "set of tel_ids for telescopes with data"

Can you clarify again what is meant to be the difference between tels_with_data and tel.keys()?

Related issues: #1041 #722

We are still missing a place to store dl1b (hillas etc.)

@MaxNoe I agree .  The old scheme was trying to follow the top-level data model naming scheme, which wasn't really meant to be a storage hierarchy.    The only question for me is whether or not to swap the data level and telescope, since separating things by data level is still useful (since you don't need to pass DL0 info on to higher steps, etc).   So another option (perhaps a bit more complex) is:
event.dl0.array.telescope
as opposed to:
event.array.telescope.dl0.
I'm thinking mostly of the use case of parallelization, where you want to be able to send around the least amount of info to a remote process.  So if I'm doing the reconstruction step, I only need to "scatter" the dl1 info to multiple processors, and I don't want to also keep the dl0 info.    I suppose it depends on where the scatter operation happens, at the telescope event level, or array event level (certainly for reconstruction, it's at the array event level)

Can you clarify again what is meant to be the difference between tels_with_data and tel.keys()?

I think the rational was this:

tels_with_data and tels_with_trigger should only be in the trigger container (they are things given by the subarray trigger, not necessarily the telescopes).  For example a telescope can trigger, but be busy and not read out,  and the tels_with_data pattern can be used to construct the keys of the subsequent levels during stereo event building (and also as a cross check that we didn't lose a telescope).  It's an index of the expected data, so something that should be monitored, but maybe not used directly in the analysis (except during stereo event building).
right now there is no nice method to write the tel.keys() pattern to an output column (but that could and should be fixed, if we want to write out index info like that (or we just rely on the PyTables automatic indices, which also is fine).
we should use tel.keys() everywhere except in the trigger container, since it is the place where we might drop a telescope during analysis, etc.

I am also hugely in favor of Maxs proposal. In another issue @MaxNoe also mentioned the service approach used e.g. in fact-tools to get information, that is not naturally synced with the event timing, such as pointing. I think telescope events should also carry refs to their respective services. I did not follow this discussion closely but I assume these services would be accessed lazily, right?

fact-tools to get information, that is not naturally synced with the event timing,

There's another discussion about that in the various calibration issues, but no real resolution. What you are talking about is basically the difference between Monitoring data, and Event data that has been interpolated from Monitoring data.  Monitoring data (a time-series of measurement that are not synced to the event stream, but rather taken once a second or so) is so far not treated in a nice way. But the idea (perhaps similar to the FACTTools services) is that you would have a Component that loads up monitoring data, constructs an interpolator, and writes the interpolated result to the event.pointing container (for pointing), or even the event.array.tel[x].calib.pedestal (or wherever we put that).
Therefore what is in the event are instantaneous values of those quantities, interpolated to the current event time.  Other COmponents that use them do not need to know how they were interpolated (or even if they were computed on-the-fly like with pedestals, perhaps), just that the value in the Container is the right one for this event. Therefore they don't need to distinguish between data coming from a "service" or from an event stream.

For @MaxNoe's proposal, the question is wether it is better to have a single "event data" as shown above, or something like we have now (but cleaned up), where there is a clear separation of input and output data.
What we have now is like this:

Which means you could parallelize (or just serialize to disk) one step of the data flow, by data level.  If we invert the tree, this is harder (even if nicer conceptually in memory)

where there is a clear separation of input and output data.

I'm sorry but I do not see the clear separation in the image above:

DL1 has 3 inputs. DL2 has 2 inputs. From this I learn: "multiple Components can write into a DL".
DL0 has 3 outs, DL1 has 2 outs: "multiple components can read from a DL"

So what, I ask, can prevent me from writing a Component or accidentally misusing a Component, that reads from DL0 and writes back into DL0?

@dneise Yes, both your statement are correct.  I'm just showing the current design, where everything reads and writes to one central data structure called "Event".  There is nothing stopping a rogue component from modifying another's data (other than regression tests, etc).  Are you suggesting another design?   In the suggestion from @MaxNoe, it's even more linked, since the data levels cannot be separated into separate objects (and there can only be one central "array event" object shared with each stage of the analysis).
In principle we could implement a "owner" system to prevent writing to places where one shouldn't, but not sure it's worth the effort.

I was trying to point out that this statement here:
For @MaxNoe's proposal, the question is wether it is better to have a single "event data" as shown above, or something like we have now (but cleaned up), where there is a clear separation of input and output data.
seemed not entirely true to me. From my personal point of view, both Maxs proposal and the current solution are equal with respect to the question how linked the data levels are. I had the impression you were pointing out a downside of Maxs proposal and I wanted to point out that this is not a unique downside of Maxs proposal but of both solutions. I did not want to say this is an issue, that needs to be fixed.

Perhaps it's useful to look at the simplified data flow just from DL0 → DL1, right now the situation looks (something) like this, where the order of calling each component goes from left to right:

(of course right now we don't do the monitoring interpolation, the realtime calibration coefficient production part. In #1163 , the image processing is 2 components and the writing is not yet split into a component. But this is the design we are working toward so far).

The question is: does this all still make sense, and is the container structure designed correctly? (clearly not so far).
It gets more complex when you start to think of further stages:

The next steps might not all operate with an event loop, for example computing the per-telescope energy estimate or discrimination parameters in a standard Hillas-style analysis might happen in a tool that just reads the DL1 file and attaches columns to it (since no image info is needed)
the "Stage 2" part must read these input files (or continue in-memory if done simultaneously), and do the reconstruction, etc.

@MaxNoe I agree . The old scheme was trying to follow the top-level data model naming scheme, which wasn't really meant to be a storage hierarchy. The only question for me is whether or not to swap the data level and telescope, since separating things by data level is still useful (since you don't need to pass DL0 info on to higher steps, etc). So another option (perhaps a bit more complex) is:
event.dl0.array.telescope (which is not far from what is there now, but perhaps can be simplifed)
as opposed to:
event.array.telescope.dl0.
I'm thinking mostly of the use case of parallelization, where you want to be able to send around the least amount of info to a remote process. So if I'm doing the reconstruction step, I only need to "scatter" the dl1 info to multiple processors, and I don't want to also keep the dl0 info. I suppose it depends on where the scatter operation happens, at the telescope event level, or array event level (certainly for reconstruction, it's at the array event level)

I like the "event.array..." approach (Max's) for the event container structure, but I am not sure I understand (probably I don't) why it poses a problem when sending around data of specific levels - the structure does not force one to fill it completely, right? Is it just that you expect it to be much less efficient if you have to strip it of all the unneeded data, e.g. DL0?

I also like @MaxNoe proposal, one could also think to have r1, dl0 and dl1 at the telescope level and the dl2, dl3 at the array level:
event.array.telescope.r1/dl0/dl1
event.array.dl2/dl3
Just, the "array" name is perhaps a bit misleading (given the numpy array), perhaps better  "cta_array"  or even only "cta"  : event.cta.telescope.dl0 , event.cta.dl2 ...)

@FrancaCassol yes exactly. DL2 / DL3 is array event wise

Also, I don't mean event.array
I mean array_event as the top level container.

yes @MaxNoe, I was looking at Karl's naming scheme which reflets the standard data model categories. The intermediate level is useful if part of the telescopes are pointing to different targets. I don't know how the data streams will be handled then, but in the case they will not be totally separated, one could introduce the  "observation/target/subarray" level  (call it as you like):
event : with everything concerning the whole array (e.g. atmo, (?) )
event.obs[n] : with everything concerning a specific observation/target/subarray  (e.g.  event.obs[n].dl2 )
event.obs[n].tel[m] with everything concerning a specific telescope (e.g. event.obs[n].tel[m].r1)

event.array.dl2/dl3

There are still telescope-wise data at the DL2 level (but not DL3) - recall the discussion at the f2F ASWG meeting.

event.obs[n] : with everything concerning a specific observation/target/subarray (e.g. event.obs[n].dl2 )
event.obs[n].tel[m] with everything concerning a specific telescope (e.g. event.obs[n].tel[m].r1)

I don't think we should mix observation-level things in here - all processing is done for a single observation at most, with a single sub-array.

I'd like to keep this as close as possible to the top-level CTA data model, rather than invert it. The point of this conversation was more about how to fix places where we are not consistent, rather than fully re-designing a new data model.    I see some nice points of going for "telescope → dl0/dl1..., subarray → dl0/dl1", but at the same time it makes writing and reading the data much more complex (since it will be stored closer to how it is organized now)

@kosack I don't think the CTA data model is meant or should be treated as something that should be exactly represented in code. It's about the general concepts and entities, not about how to layout storage in memory or in file.
Different use cases might prefer the one or the other hierarchy in memory or in a file. That does not change how they relate to each other.
I discussed with @FrancaCassol again, and I am much in favor now of using for the in memory representation for the preprocessing until DL2 a version that looks like this:
array_event
  mc
  ...
  telescope_events
      trigger_info
      mc
      dl0
      dl1
      dl2
  dl1
  dl2

I think this as several key advantages:

telescope events can be indenpendently processed and easily merged after all telescope events for the same array event have been processed
It solves several design problems we had with where to store meta information for telescope-wise data (should trigger_type should go into r0? r1? in every telescope data level container? A new map?)
We avoid inner loops over telescopes in things like calibrators. Just two immediately visible and easy to understand loops over array and then telescope events.

but at the same time it makes writing and reading the data much more complex (since it will be stored closer to how it is organized now)

You are referring to single-telescope files? How is this easier with the structure you proposed? I see the same challenges for both, not a general difference.
I think we should make splitting and merging full telescope events easy. Currently this is hard, has you have to update lot's of telescope Maps. And one cannot easily see, if a map is really mapping telescope ids and not something else.

Yes, I think it might be clearer this way, and since the DL1 (and DL2) writer components I'm working on don't really care about the in-memory structure, reading and writing shouldn't be too hard to accomplish.
However, can you explain a bit your diagram/structure above? I'm not sure I get it. For example, trigger_info is something right now that is written by the SWAT, so it's array-level information. Or is that the telescope-trigger information? and then where does the array-trigger go?
Perhaps if you add one-level of detail lower, it will be clear to me.

Or is that the telescope-trigger information? and then where does the array-trigger go?

Ah sorry, that was not really meant to be an exhaustive or even clear diagram. I just wanted to make the point array_event → list (or mapping) of telescope_events.
Both the array event and the telescope_event will have trigger info. Array with central array information and telescope with telescope trigger info (e.g. trigger type, time, fired trigger patches...)

OK, yes that makes more sense. So basically you have:

event

subarray data
list of telescope data  [ tel1, ... telN]

The same could be use for  (non-interpolated) monitoring:

monitoring

array monitoring data
list of telescope-monitoring data

Monitoring is more service-like, isn't it?
So it will probably be an api like:
pedestal = monitoring.camera.get_pedestal(
    tel_id=tel_id,
    timestamp=tel_event.trigger.timestamp,
    strategy='closest',   # or earlier to only allow data known previously
)

Yes, that is what I intended for monitoring, in fact.  However, you still need to write out some monitoring data (pedestals computed every few seconds, data quality parametersetc), so having a hierarchy would be nice,

One more question: how ten do we deal with the fact that the list of telescopes changes between  data levels?   At R1/DL0 you have all telescopes that triggered.  At DL1 you might only have parameters for a subset that passed cleaning.  At DL2 you only have reconstructions for a further subset that had good parameters, etc.   Right now, that is easy to deal with, but in this scheme, we would need to add a way to keep track of this, so you don't loop over things that are missing.

Yes, this is the thing that is harder in the current scheme. However, with using proper missing value markers or flags, this should be easy, right?
Or just setting the respective dl1/dl2 containers to None if they weren't found in the file.
We probably should implement something to be able to disambiguate defaults from unset fields in containers. The current behaviour of setting the default for things  not found in the input file seems dangerous.

perhaps defining a common place to set an algorithm-specific quality flag is one option, or setting the Container to None if it is not included (but that might cause problems for writing).
In fact, this might help the problem that we might apply multiple algorithms (2 cleanings, for example), in which case the list per algorithm method got messy (you'd need 2 lists).
I just want to think about how algorithms will look after this change.  For example, before we just could pass a list of HillasParameterContainers to a Reconstructor, which didn't need to know about the event structure at all.  In this case, i guess you could just use list comprehensions to basically go back to the way we stored things before:
params = [t.dl1.parameters.hillas for t in event.tel if t.dl1.parameters.hillas]
shower = reconstruct(params)
event.subarray.dl2.shower = shower

or you force all algorithms to know about the event structure (less ideal, especially if there are more than one algorithm that produces the output):
reconstruct(event.tel)

here a loop over tel events is made inside the Reconstructor, but then it need to know to get the parameters from tel.dl1.parameters.hillas.  If there are 2 sets of hillas parameters from 2 cleanings (say tel.dl1.parameters_wide_cleaning.hillas, for example, it then becomes complicated.

Also, I guess when you say "list of telescopes", it's really a dict by tel_id, otherwise you have trouble connection the telescope to the instrument description

Anyhow, I'd still like to see an example of how this makes the loops for cleaning, parameterization, and reconstruction cleaner than before, since I still feel like it requires a lot of bookkeeping, given that the algorithms should not be too tied to the event structure.

Yes, I think we have now seen that there are usecases that are easier and simpler in either format.
So without an actual implementation, discussions will probably not be able to solve this.

But especially from the single telescope's perspective, it would be nicer if they would just fill their TelescopeEvent container with all the relevant info and this could be merged later into an ArrayEvent.
This is also probably close to what real data taking will look like, right? First there are single telescope events that are then merged into array wide data structures.

Yes, I do like that the TelescopeEvent can be separated (at least at the DL0 level)

One more question: how do we deal with the fact that the list of telescopes changes between data levels? At R1/DL0 you have all telescopes that triggered and read out. At DL1 you might only have parameters for a subset that passed cleaning. At DL2 you only have reconstructions for a further subset that had good parameters, etc. Right now, that is easy to deal with, but in this scheme, we would need to add a way to keep track of this, so you don't loop over things that are missing/dropped in the previous step.

I think that all events should be kept until actual selection at DL2 level, as you suggest with a reconstruction quality flag.

I think that all events should be kept until actual selection at DL2 level, as you suggest with a reconstruction quality flag.

It's not that easy.
"Keeping all events" is something else than "using all events for stereo reconstruction".
You can make all kinds of (telescope) event selections for the different reconstruction algorithms, which will probably be necessary to achieve a good performance.
I agree we should keep all telescope events, but we also need to have a way to do the reconstruction only on a subset of those events (more than N pixels, less than X leakage, etc)

I think that all events should be kept until actual selection at DL2 level, as you suggest with a reconstruction quality flag.

It's not that easy.
"Keeping all events" is something else than "using all events for stereo reconstruction".
You can make all kinds of (telescope) event selections for the different reconstruction algorithms, which will probably be necessary to achieve a good performance.
I agree we should keep all telescope events, but we also need to have a way to do the reconstruction only on a subset of those events (more than N pixels, less than X leakage, etc).
So already at the DL2 level, you will have discarded some telescope events for some stereo reconstruction. (Or even Dl1 telescope event reconstruction in case of no or just a couple pixels after cleaning)

Yes, and I meant keeping them not using all of them. Those not passing cleaning, for example, are obviously not used. Yet they should not disappear between DL0 and DL1/2 but can be flagged as such.

After working on the LST EventSource again and all the calibration steps connected to that (e.g. time_shift), I came to the conclusion that the hierarchy

ArrayEvent

TelescopeEvents

DataLevels

will be much easier to work with.
Right now, LST defines it's own LSTArrayEvent, since it needs out-of tree telescope-event-wise data.
I think it will many things much cleaner and I don't yet see a real downside.
Especially the usecase of merging the different telescope streams will be much easier.

@maxnoe In the inverted hierarchy above, can you expand the example to show where all subarray info  as well as monitoring goes?   I think it will work well, but would be nice to describe it in more detail.
Some other nice points:

We already have the problem in the current model that monitoring is not organized by data level, even though it should be.
This hierarchy would also fix the fact that the calibration part of the event is already organized in an inverted way compared to other info (event.calibration.r1.tel[x]  instead of event.r1.tel[x].calibration).


Fix Uncaught TypeError: #12528

Fixes #12528
This uncaught exception occurs during scrolling if the cursor hovers over a block decoration containing a SVG element.
It fails because scrollevent.target doesn't have a property dataset since it's a path inside the SVG instead of a DIV.
I discovered this bug during redesigning the output area of Hydrogen in https://github.com/nteract/hydrogen/pull/516.
/cc @mangecoeur

/cc @as-cii

Thanks @lgeiger!

@as-cii Thanks for merging.
Since the diff is minimal, can we backport the fix to Atom 1.12 and 1.13?
I'm happy to make PRs to the relevant branches.

Agreed it would be nice to have it on stable/beta. I'll cherry pick the commit into the relevant branches.

Thanks @as-cii 👍

That's a much simpler fix than I was expecting! +1 for backporting, would be nice to drop my current workaround in preview-inline which is to overlay a div over SVG.


Add project-keeper source for go

Add project-keeper source for go
Blocked by https://github.com/exasol/project-keeper/issues/307

Requires https://github.com/exasol/project-keeper/pull/318


Running project in production: swagger project start vs. node app.js

Is there a recommendation on whether one should use the swagger project start vs node app.js to run your app in production?
Sorry if this is documented somewhere, but couldn't find mention of any tradeoffs.

I suggest using swagger project start for development and node app.js for proper deployments.

Perfect, thanks!

@fehguy I'd be happy to add this to the docs.  After looking through the /docs directory, I thought maybe at the end of Quick Start would be appropriate.  Does that seem good?

yes, great

cool, done. and thanks for accepting! #415

i think we can using

pm2 start app.js

proper deployments.

Is there any reason to not use node app.js for both development and production?


Object.values not a function on IOS Safari v10

Polyfill Object.values is not served in browser IOS Safari 10 (and it should be)
User agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1

Strange because it seems to be well detected: https://cdn.polyfill.io/v2/polyfill.js?ua={Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1}
And well configured: https://github.com/Financial-Times/polyfill-service/blob/master/polyfills/Object/values/config.json#L11

Hi @testerez, when making a request to https://cdn.polyfill.io/v2/polyfill.js without declaring specific features to polyfill, we will serve a default set of polyfills. The default set does not contain Object.values, if you want to include Object.values alongside the default set you can add the query parameter features=default,Object.values to the polyfill.io url.
Here is an example of serving only the Object.values polyfill and forcing the polyfill service to target iOS safari 10.
https://cdn.polyfill.io/v2/polyfill.js?features=Object.values&ua={Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1}

Oh makes sense thanks!
I'm just curious to know how you decide what's part or the default polyfills or not.

@testerez This issue should hopefully explain how it is decided what goes into the default set -- https://github.com/Financial-Times/polyfill-service/issues/1203

Thanks for the link. I feel like Object.values kind of matches the criterias to be part of the default polyfills:

Demand (should be high): Not sure about this one. There is at least me :)
Polyfill size (should be small): it is: https://github.com/Financial-Times/polyfill-service/blob/master/polyfills/Object/values/polyfill.js
Number of browsers with native support (should be at least 2): It's supported by all major browsers except IE: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Object/values

.values is unsupported in many browsers - you can use .map to get an array of all the values:
var vals = Object.keys(countries).map(function(key) {
    return countries[key];
});

See MDN doc: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/values or Official doc: https://tc39.github.io/ecma262/#sec-object.values

@AlexLaforge support for .value is no so bad:

And yes it's easy to replace with Object.keys(o).map(...) but it also means that it's pretty easy to polyfill...


Python instances of deoplete-jedi open, open slowly, and never close on Windows 7

Problem summary
On a moderate python project, approximately 20 instances of python are opened per possible completion, and remain open. The processes open approximately 1 per second, and only when they all open does the completion window display. Moving to another completion causes the same thing to happen, leaving even more python instances each time.
This gives an approximately 20 second wait for any completions to be displayed, and can eventually take up many gigabytes of memory (my test got up to 10 GB)
Environment Information

OS: Windows 7
Neovim version: v0.3.1

Provide a minimal init.vim with less than 50 lines (required)
let g:python_host_prog  = 'C:\Python27_64\python.exe'
let g:python3_host_prog = 'C:\Program Files\Python36\python.exe'
set runtimepath+=C:\Users\tyler\AppData\Local\nvim\bundle\deoplete.nvim
set runtimepath+=C:\Users\tyler\AppData\Local\nvim\bundle\deoplete-jedi
let g:deoplete#enable_at_startup = 1
call deoplete#custom#option('jedi', 'debug_enabled', 1)
call deoplete#enable_logging('DEBUG', 'C:\temp\deopleteJedi.log')

Steps to reproduce the issue after starting Neovim (required)

Open a file in a python project. I used deoplete as an example
:e C:\path\to\deoplete.nvim\rplugin\python3\deoplete\__init__.py
Type something that will cause a completion to be built.  For my example, I went to line 52 (where global_deoplete was defined), opened a new line, and typed "global_deoplete"
Open the task manager, and watch as python processes start, approximately 1 every second.
Delete the line you just typed, and re-type it to spawn even more python processes

Screen shot (if possible)
This was taken after typing "global_deoplete" only twice (from the reproduction steps above)

Upload the logfile(s)
deopleteJedi.log

Might be / sounds like https://github.com/davidhalter/jedi/pull/1238 - jedi 0.13.2 should fix it then.


Changes jib-gradle-plugin to use new Jib Core API.

After merging #1126 .
For #1118
This adds NBuildStepsRunner that should replace BuildStepsRunner once jib-maven-plugin is changed to use Jib Core as well.

Merged master.

Gonna wait for one more approval since this touches a large portion of the critical code path. @GoogleContainerTools/java-tools


No possibility for borderStyle other than solid

When trying to have border style 'dashed', 'dotted' or any other, xhtml2pdf draws solid one.

border styles not working, except solid


Improve memory load efficiency for shape_availability calculation

Changes proposed in this Pull Request
Improving the efficiency of shape_availibility() with respect to memory load through changes in dtypes and the method of mask summation.

Description
Two main changes are made to reduce the memory load while running shape_availability() within gis.py:

dtype transformations via np.astype() to int32 are removed on several occasions to keep matrices returned from functions as rasterio.features.geometry_mask() and scipy.ndimage.morphology.binary_dialation()  in dtype bool.
The dtype transformation to float64, applied to the final mask that is returned by the function, is removed.
Instead of saving individual masks for every exclusion raster or geometry to a list and summing them on the return, a new method is introduced, which adds the masks on every loop iteration. This is done via the | OR operator, to keep the single mask in memory as dtype bool.

Motivation and Context
With higher resolution rasters or greater land area covered, the underlying matrices, when calculating the eligible area via  shape_availability(), quickly grow in size. E.g., a raster with 50 meter resolution bound by the shape of Germany produces an array of shape (17086, 12679). With the current method of storing an individual mask for every raster added to the ExclusionContainer() in dtype int32 this can lead to infeasible memory usage for conventional systems. The changes enhance the efficiency of storing information, as well the method of combining information form multiple rasters.

How Has This Been Tested?
So far tested locally with the latest version of atlite. Tests included rasters with buffers, inverted rasters as well as geometries.
pytest did not bring up unexpected errors.

Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] I tested my contribution locally and it seems to work fine.
[x] I locally ran pytest inside the repository and no unexpected problems came up.
[ ] I have adjusted the docstrings in the code appropriately.
[ ] I have documented the effects of my code changes in the documentation doc/.
[ ] I have added newly introduced dependencies to environment.yaml file.
[ ] I have added a note to release notes doc/release_notes.rst.
[x] I have used pre-commit run --all to lint/format/check my contribution

Running without and with this patch land availability investigation for Italy as an example:

One geometry layer (WDPA)
One raster layer (slope derived using GEBCO data)

Output of %mprun for both scenarios:

Lines 447, 467: Same change in memory usage when generating the masked array
Lines 464, 468: No memory increase from converting into a different datatype or temporarily storing the masked version in a list (in patched version)

Overall significantly lower memory footprint. Nice! I'll run a test with more rasters and geometries.

Comparison with multiple rasters and geometries:

Notice the overall significantly lower Mem usage inside the function which continous to increase as more data is temporarily stored in the list exclusions (no-patch version left side)
Some peak Increment values when large datasets are processed in both, patched and unpatched, versions
Overall maximum Mem usage in patched version does not exceed ~4100 MiB´ despite adding multiple datasets. In the unpatched version the Mem usageexceeded14000 MiB` and would have grown even further if a larger raster/cutout or more datasets would have been selected

Looking very good @calvintr !

Running @FabianHofmann recent commit 31f9b0d (left) against the previous one c67bebd (centre) and one where we convert to float before returning the value (right):

Converting to float obviously increases the memory consumption, but only once at the end of the function. For the sake of backwards compatability I believe that is something we should do....

What are the breaks you saw or have in mind? The follow-up functions run can work with it as well as plt.imshow (as used in the examples). I cannot think about any use case that is affected here. And from a conceptional point of view, boolean make more sense, just indicating whether it is eligible or not.

What are the breaks you saw or have in mind? The follow-up functions run can work with it as well as plt.imshow (as used in the examples). I cannot think about any use case that is affected here. And from a conceptional point of view, boolean make more sense, just indicating whether it is eligible or not.

I agree that returning the mask in boolean makes sense from a conceptional point of view.
Also, the transformation of the final mask on the return to float was what formerly caused the Memory Allocation Error for me. This should be much less likely now with the memory saving changes implemented. However, with the dtype changes and mask summation, the float transformation would now account for the biggest chunk of memory load apart from the projected_mask().
In my (limited) workflow I did not encounter any breaking errors with the boolean arrays in plotting or writing it to a new .tif file with GDAL. There is however one issue I ran into that might be relevant also for the examples in the documentation:
When summing the boolean array for large masks and multiplying it (to calculate the eligible area for example) this creates very large integers that lead to an overflow warning [RuntimeWarning: overflow encountered in long scalars] / wrong output for me, following the code example of atlite. It can be resolved with a dtype transformation on the summation. See in the attached screenshot, example is run with my commit returning the mask as dtype bool:

I get your arguments.
@calvintr Your issue is related to what my issue with changing to bool as dtype is, see here: https://stackoverflow.com/a/41705902 for the reason you presumably get the issue (tl;dr: numpy on Windows uses by default int32 in some cases).
My issue is how this feature might usually be used in regular code:
The masked array is usually used for further calculations and modificiations. Since we are changing the dtype we are also changing on how it behaves.
Take e.g. assigning NaN to masked where there are 0s:
masked[masked == 0] = np.nan

for a dtype float, that code works without issues.
However if the dtype of masked is bool, this will result in an array full of True, as np.nan is automatically casted to a boolean value and the truth value of np.nan is True.
Example:

We can:

Add a FutureWarning for users and change the return type in the 2nd-next version
Add a Warning about the changed dtype in the function, release notes (always) and change it right away
Don't change the return type and accept the fact that the casting is memory intensive

We should:
Defintely have a test which catches these types of changes. Is there something that Python offers which allows for simple checking if function return types / signatures change e.g. using pytest?
I only caught this issue by chance and I would have preferred it to be notified about it automatically by the CI :/

Thinking about a middle ground where the return dtype is an int (preferably int8). Then, we do not encounter the warning posted by @calvintr  and numerical operation should well-behave. The nan assignment would still lead to a weird behavior. However, I cannot think about an use-case for having nan's in a mask (what should these express? missing data? already given by nodata which is a defined int value.)
I'd go for option 2 of @euronion since it is not only solving the memory issue also speeding up the process in pypsa-eur and I don't think this is anything severe.

I don't see any benefits of the middle ground solution, more like only downsides:

It explicitly is meant to fix @calvintr problem
It breaks the nan issue, as nan cannot be represented by int arrays:

np.int8(np.nan)
ValueError: cannot convert float NaN to integer

(btw.: nodata is not directly Python but a predefined value via rasterio in our case. int doesn't have a nodata value)

It softens up the idea of returning True and False for the map

Based on that I think going with option 2 might be the better one to go with.

Alright, then let's go for pure option 2. Just for the background of the "middle ground" option: For the availability matrix computation, the boolean masks have to be transformed to int arrays anyway, right before passing it to rasterio reproject. Meaning, the conversion would not lead to a any memory overhead and the effective change of this PR would be changing the output dtype of shape_availability from float to int instead of float to bool. On the same time, it would solve @calvintr 's warning. But that said, I prefer having a boolean output anyway.
So, I'd run final tests on the pypsa-eur workflow. @calvintr could you add a warning to the shape_availibility function and update the release notes? Then, we'd be ready to merge.


Undo typing in zsh as in a normal OS X text

If I'm typing into this GitHub comment box and I type CMD+Z, it will undo my typing, bit by bit.  Is it possible to do this in zsh/bash?  Would help if you typed the wrong thing when you tab to to the wrong folder and have to simply hold the delete key forever to back up.
Just a thought, this is probably not the best place to post, but I thought it would get in front of others who care about it.

Undo in zsh is Ctrl+_ (Ctrl+Shift+Hiphen). The best place to post is probably unix.stackexchange.com


Add details on qiskit-terra warnings to install guides

Summary
This commit adds details to the install guide and the install from
source guide on new warnings being emitted by terra in the 0.9 release.
Added in Qiskit/qiskit-terra#2842 when qiskit-aer or
qiskit-ibmq-provider can not be found by terra on initial import it will
emit a warning to inform the user of that state. This is done because
today the user will have no idea if they forgot to install these
components (or more likely python packaging screwed up the installation)
and aer or ibmq can't be found. They will then try to use either
qiskit.Aer or qiskit.IBMQ only to be met by a confusing ImportError.
This is a very common occurrence for users and causes a great deal of
confusion. However, since emitting a warning on import is potentially
undesirable behavior for any users that only use qiskit-terra by
itself. If they never have any intent of importing qiskit.Aer or
qiskit.IBMQ then emitting a warning is fairly annoying. This commit
documents these warnings and more importantly how to suppress them for
these users, so that they do not have to be concerned about seeing the
warnings in perpetuity moving forward if they do not want to.
This updates both the install guide and the contributing guide since
it's conceivable a user would refer to either guide when installing
terra as a standalone component.
At the same time a stale reference to a warning that as of #304 merging
no longer will be emitted when installing the qiskit metapackage is
removed from the install documentation.
Details and comments

This is also set to be included in the terra release notes here: https://github.com/Qiskit/qiskit-terra/pull/2919/commits/b768804c968736b4965a38277023be59bfa9793b as part of Qiskit/qiskit-terra#2919


Migration from 5.6.9 to latest 6.*

PHP Version
7.4 / 8.0
Shopware Version
5.6.9 / <IP_ADDRESS>
Expected behaviour
Migrate the old site with 0 errors.
Actual behaviour

[warning] SWAG_MIGRATION__SHOPWARE_ASSOCIATION_REQUIRED_MISSING_PRODUCT_MANUFACTURER
Associated product_manufacturer not found
The product_manufacturer with the source id "67" can not be found but is required for promotion_discount.

[info] SWAG_MIGRATION__SHOPWARE_UNSUPPORTED_OBJECT_TYPE
Unsupported object type
seo_url of object type "listing" with source id "36541" could not be converted.

[error] SWAG_MIGRATION__WRITE_EXCEPTION_OCCURRED
A write exception has occurred
{
"entity": "property_group_option",
"dataId": "ffc684f07f064647ae26104381c8f6f4",
"error": {
"code": "c1051bb4-d103-4f74-8988-acbcafc7fdc3",
"status": "400",
"detail": "This value should not be blank.",
"template": "This value should not be blank.",
"meta": {
"parameters": {
"{{ value }}": "null"
}
},
"source": {
"pointer": "/1090/translations/2fbb5fe2e29a4d70aa5854ce7ce3e20b/name"
}
}
}

[error] SWAG_MIGRATION__WRITE_EXCEPTION_OCCURRED
A write exception has occurred
{
"entity": "customer",
"dataId": "e7f79e3eccd34b85b3ff2a5489c15ab5",
"error": {
"code": "c1051bb4-d103-4f74-8988-acbcafc7fdc3",
"status": "400",
"detail": "This value should not be blank.",
"template": "This value should not be blank.",
"meta": {
"parameters": {
"{{ value }}": "null"
}
},
"source": {
"pointer": "/234/addresses/1/street"
}
}
}

[error] SWAG_MIGRATION_RUN_EXCEPTION
An exception occurred
Entity: promotion, sourceId: edd013c3853f4ee1837ec5df7a9d62f4
There are 1 error(s) while writing data.1. [/1/code] Code may not have any leading or ending whitespaces
How to reproduce
Migrate with our local database, and the following plugins installed:
bin/console plugin:install TrustedShops
bin/console plugin:install SwagCmsExtensions
bin/console plugin:install SwagMigrationAssistant
bin/console plugin:install SwagPlatformDemoData
bin/console plugin:install SwagFlowBuilder
bin/console plugin:install NimbitsThirdPartyIconsNext
bin/console plugin:install SwagLanguagePack
bin/console plugin:install PickwareDhl
bin/console plugin:install NetzpNotifications6
bin/console plugin:install ebizoptimiweinccTrackingServicesSW6
bin/console plugin:install SwagB2bPlatform
bin/console plugin:install SwagSocialShopping
bin/console plugin:install IntediaDoofinderSW6
bin/console plugin:install SixProductShortDescription
bin/console plugin:install SwagPublisher
bin/console plugin:install DreiscSeoPro
bin/console plugin:install SwagCustomizedProducts
bin/console plugin:install MillProductPropertiesTab
bin/console plugin:install SemesHotjarIntegration6
bin/console plugin:install NimbitsExtraBadgeNext
bin/console plugin:install SwagDynamicAccess
bin/console plugin:install SwagEnterpriseSearchPlatform
bin/console plugin:install MillCustomFieldTabs

Could you give me a couple pin pointers where to look or fix those errors?

I managed to fix errors 4 and 5, the other ones, especially the missing translations value one still remains. If I just knew what source table it refers to.

This is not an issue with platform itself. Please create a issue in the issue tracker for that plugin


Pref fix

This pr contains the following edits:

Unit tests for preference deployment
edits in solver to take into account when the values (preference, capacity) are equal

This pr closes #134 #131

Looks good! Did you try running pytext tech_pref_test.py ?

yup and they all pass after the fix from 6 to 7 for the cross test

Great!


about Resnet18_8s

Hello, I am very impressed by your great work! However, I am a little confused when I look at your Resnet18_8s network. I assume Resnet18_8s follows your approach in your paper "Deep Residual Learning for Instrument Segmentation in Robotic Surgery", which employ dilated convolutions to keep resolution. But in resnet_dilated.py, I could not find any dilated convolutions in Class Resnet18_8s. Could you please give more detailed explanation on the structure of Resnet18_8s? Many thanks.

Hi, @MahlerMozart .
For this repository we use a custom fork of pytorch/vision where we add
dilated convolutions.
Look here:
https://github.com/warmspringwinds/vision/blob/fully_conv_resnet/torchvision/models/resnet.py
I will update the front page Readme to better show that.

@warmspringwinds Perfect explanation. Thanks!


Definitive data checking: yearmean* file not found

Environment: Version 0.4.7 (Python 2.7)  in Mac OS X 10.12
The definitive data checking routine can not find the yearmean.imo file from given path. The binaries are found.

The problem had its origin in a wrong interpretation of the comment section. If comments startet with 1 and 2 they were tried to be interpreted as data lines. This crashed and the file could not be read.
Solved that in 1aa8cf560232d7d6c3050332c0f0cc1a29bd5789

Issue is fixed from 0.9.3 onwards. Please note: yearmean files are freeform text files and it is very difficult to interprete all imaginable variations within a not well structured file.


asking

Hello, thank you for sharing the data, I am not familiar with ros, may I ask how you use the aracati2017 data set into your program？Thank you very much!

You need to install ros in Ubuntu first. Please refer to the following link for the installation process of ros.
Then, after downloading the rosbag from the following link, put it in the bags folder of this link, and follow the github readme of that link.


Add support for Application Default Credentials.

@thobrla PTAL. This needs the oauth2client changes to get released before we can actually release this.


Pr readme grammar

What is the purpose of this pull request?
Some really minor change of grammar, free t-shirt right?
How is this achieved?
What issue number does this fixed?

Thanks for the fix. Merging now.


Convert Create/Edit Mid Fi (MATS/Supp Methods )

Definition of Ready

[x] Does this meet the INVEST criteria?
[x] Does this have a clearly defined user?
[x] Does this have an estimate?
[x] Does this have acceptance criteria?
[x] Does this have sufficient context (wireframes, business rules, etc.)?
[x] Is this traceable to a feature, epic, or PI?

User Story
As a user
I want to create or edit MP MATS/Supp Methods using the mid fidelity design
So that I have a better experience
Acceptance Criteria
Given a user has accessed MP MATS/Supp Methods
When it is created or viewed
Then it is displayed according to the wireframes
Context
https://xd.adobe.com/view/df8c27d8-1cff-494d-946d-7c8cf1a2bdc3-8adf/
https://xd.adobe.com/view/eb6c3176-9c62-46e0-af54-8be9a2771b7a-be3f/
Testing
Test Case:
Tester:
Definition of Done

[ ] Have all Acceptance Criteria been agreed to and validated by the scrum team?
[ ] Has the code been unit tested and peer reviewed?
[ ] Have the functional tests been executed?
[ ] Have all defects been reviewed and dispositioned, resolved, or deferred?
[ ] Has the user story been accepted by the Product Owner?
[ ] Is the user story ready to be deployed to the test/staging environments?

Modal not built yet. Will address responsiveness when modal is built.


Rust doc does not fullfill the whole page on firefox with minimum font size setted

browser: Firefox 97.0.2 (64-bit) stable
Can be reproduced on macOS and Linux.
Not sure it's a firefox bug or rustdoc.

Thanks for reporting! I believe this is a duplicate of #1669. I recommend closing in favor of that issue.


Remove intl polyfill

Summary
This polyfill is only needed by Safari 9 which is not supported anymore as stated in the issue.
Ticket Link
Fixes

https://github.com/mattermost/mattermost-server/issues/12016
https://mattermost.atlassian.net/browse/MM-13540

Related Pull Requests
https://github.com/mattermost/mattermost-server/pull/12912
regards,
Nicolas
#Hacktoberfest

Creating a new SpinWick test server using Mattermost Cloud.

Mattermost test server created! :tada:
Access here: https://mattermost-webapp-pr-4053.test.mattermost.cloud

Account Type
Username
Password

Admin
sysadmin
Sys@dmin123

User
user-1
User-1@123

New commit detected. SpinWick will upgrade if the updated docker image is available.

Mattermost test server updated with git commit 2aa9c7bf1b651c2c18af9099c0aa95eda174f6b4.
Access here: https://mattermost-webapp-pr-4053.test.mattermost.cloud

Test server destroyed


Add putty-cac

Add PuTTY CAC (with CAPI and PKCS support).

Replace checkver and au to:
    "checkver": {
        "github": "https://github.com/NoMoreFood/putty-cac",
        "regex": "tree/([\\w.]+)"
    },
    "autoupdate": {
        "architecture": {
            "64bit": {
                "url": "https://raw.githubusercontent.com/NoMoreFood/putty-cac/$version/binaries/puttycac-64bit-$version.zip"
            },
            "32bit": {
                "url": "https://raw.githubusercontent.com/NoMoreFood/putty-cac/$version/binaries/puttycac-$version.zip"
            }
        },
        "hash": {
            "url": "$baseurl/puttycac-hash.txt",
            "regex": "$sha256.*?$basename"
        }
    }


Create application component hierarchy and styling

Description
This pull request removes the .babelrc file from the project as it is no longer needed.
What changed
The .babelrc file was deleted. This file was previously used to specify how Babel, a JavaScript compiler, should behave. However, it is no longer necessary in our project.
How to test
Since this is a deletion of a file, there are no direct features to test. However, you should ensure that the project still builds and runs correctly without this change. If there are any issues, they may be related to this change.
Why make this change
The .babelrc file was initially added to configure Babel for our project. However, with recent updates and changes in our project setup, this file is no longer necessary. Removing it helps to keep our project clean and free of unnecessary configuration files.

Current dependencies on/for this PR:

main

PR #20   👈

PR #21

This comment was auto-generated by Graphite.

Merge activity

Oct 25, 12:41 PM: @aniledev started a stack merge that includes this pull request via Graphite.


Update resources

Hi!
I run  "SwtorCaster.JediPedia", but I do not know where find the key of "https://swtor.jedipedia.net/ajax/getAbilities.php?auth={authKey}"
Could you help me?

That was an auth key that JediPedia gave me to access his API when i first started out this project. So its not part of the project because its not a public sharable key. I'm unsure if it still works or if his API has changed. You would probably need to reach out to him to get a unique Key - since the Key was only used to build and package the project prior to release.


Xcode 7.2

New selectors are great however it means we can't build if we're on 7.2 still...

Bien

Hey @LukeDurrant , thanks for the PR! Supporting older versions of Xcode (and with it Swift) is not in my intentions for this repo.
With that said, if you find any improvements that will help with Swift 3.0, submit another PR.


TF 2.3 broken hierarchical functional model loading (e.g. HAN) [ValueError: Unknown layer: Functional]

System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.3
Python version: 3.6.9
CUDA/cuDNN version: v10.2
GPU model and memory: GeForce GTX 1070 - 8117MiB
Describe the current behavior:
I cannot load a model trained with TF 2.3 in TF 2.2 -> breaking change
In TF 2.3 the release notes mention the following:
Functional models now get constructed if any tensor in a layer call's arguments/keyword arguments comes from a keras input. Previously the functional api would only work if all of the elements in the first argument to the layer came from a keras input.

Describe the expected behavior:
Model loading stays stable between releases...
Standalone code to reproduce the issue
I cannot share the code due to proprietary rights... yet since it is reported in the release, it should be straightforward to find a solution.
E.g. should I change the way I load my model?

@Jordy-VL,

I cannot share the code due to proprietary rights... yet since it is reported in the release, it should be straightforward to find a solution.

Could you please provide a dummy model or a minimal working example to reproduce the issue reported here? Thanks!

@amahendrakar The similar errors has been recenlty reported in TF mailing list as well as at SO: https://stackoverflow.com/questions/63068639/valueerror-unknown-layer-functional

Exacty as @lithuak mentions. However, the solution proposed to recompile and only load the weights does not solve the issue...

@Jordy-VL Thanks for reporting this issue. Can you please provide a standalone code to reproduce the issue? Thanks!

@Jordy-VL Thanks for reporting this issue. Can you please provide a standalone code to reproduce the issue? Thanks!

I'll start on a dummy model first thing tomorrow. Thanks!

@Jordy-VL I am not sure how much we can resolve without looking at the model creation code as the current error is "Unknown layer". But, let's try it. Thanks!

Sorry for the delay @jvishnuvardhan
For best resolution, I created a colab gist with the model creation code; instructions are contained inside.
https://colab.research.google.com/drive/1fL9q5-ThAbYXJ_EgALhAHQ3x2OXb99jd?usp=sharing
Thanks for nudging me to complete this model gist ;) I was a bit too swamped the last days.

@Jordy-VL Why do you backend.symbolic_learning_phase()? When we define a keras function, we need to provide input and output of the model but you are providing learning_phase also.
when I removed learning_phase from the keras function, it didn't throw any error. Thanks!
model_mc = tf.keras.backend.function([new_model.input], [out_layer])

I need it to use MonteCarlo Dropout, where you activate learning rate to simulate dropout at inference time. If you know of an alternative, do tell :)

Hi , i am getting similar issue while training a model.

project link:- https://github.com/kairess/eye_blink_detector
first run preprocess.ipynb and the train.ipynb, after this model is trained and stored .
Now when i try to load model as below, gives above error

Issue resolved , actually problem was that while training i was using tensorflow 2.3.0, while running i was using tensorflow 2.2.0.
Hence this issue occured.
Later i tried compiling and running with tensorflow 2.2.0 only and it got through.

Still happening during loading (tf 2.2) h5 model trained in colab (tf 2.4)
Shouldn't this be fixed in a way that even older versions of tf2+ (2.0, 2.1, 2.2) are compatible with tf 2.3, tf 2.4 +?

I do not think make the training TF version same with the  prediction TF version is a good idea. I met this issue because i have deployed the prediction model to our customer site with TF version 2.2, but out training system recently upgrade TF version to 2.4. Therefore, our new trained model cannot use on our clients' site. Upgrade TF version of all our clients is infeasible, nor keep our GPU training system to TF 2.2 is a good idea i suppose.
Could we add an args to model.save() to export model in old format?

Was able to replicate the issue in TF 2.6.0-dev20210530,please find the gist here..Thanks !

We are sorry. But the solution to this error is very simple, ex. the reason is that you have trained the model on version '2.3.0' of Tensorflow & '2.4.3' of Keras (On Colab or local). and now you are accessing the saved model(.h5) via another version of Keras & TensorFlow. It will give you the error. The solution is that re-trained model with upgraded versions or downgrades your TF&Keras to the same version as on which model is trained.
You can also look at the solution here


United States Navy seal

          I'd just like to add to the list:

[x] test it with the default zoomSnap: 1 option
[ ] test it with a fractional zoomSnap, like 0.25
[ ] test it with zoomSnap: false or 0

For development of fitBounds I believe the best is to use zoomSnap: 0, as snapping to an integer zoom level will usually add a big padding on it's own.
Originally posted by @hyperknot in https://github.com/Leaflet/Leaflet/issues/4528#issuecomment-216535056

I just love to say hello to all you wonderful and beautiful girl out there I hope you have a very wonderful 4th of July and a beautiful weekend I hope to hear from some of you or somebody soon thank you God bless and stay vigilant

I'll text all you beautiful people live looking for your hearing from all of you

And this one Navy SEAL just want to let you know that I love all of you out there and it was definitely a pleasure to serve and take every one of you thank you


preliminary configuration files for ipit

adding files for ipit  robot

@davidetome tested the PR.
Further refinements/amendments can be applied later on, if needed.


Check also unversioned SLE-12 for existing jobs

fixes https://progress.opensuse.org/issues/137006

Closeing in favor of #142


There are no two parameters in DQN():checkpoint_freq and checkpoint_path

Traceback (most recent call last):
File "D:/Personal/My_Subject/Simulation/MySubject_simulation/gym_vrep-master/gym_vrep/envs/train_vrep.py", line 101, in
main(args)
File "D:/Personal/My_Subject/Simulation/MySubject_simulation/gym_vrep-master/gym_vrep/envs/train_vrep.py", line 86, in main
tensorboard_log='./log',
TypeError: init() got an unexpected keyword argument 'checkpoint_freq'
There are no two parameters in DQN():checkpoint_freq and checkpoint_path,but these two parameters are used in the function callback().How to solve this problem？

@lzg0825 Hi, sorry for late reply. I think this proplem is caused by the different api between the different versions of stable-baselines. As you see there are no two  parameters in this version stable-baselines, but this version stable-baseliens has these two parameters. So, you can try the latter one. And I also upload the the version of stable-baselines to the Google Driver I used before. You can install it locally.

Traceback (most recent call last):
File "D:/Personal/My_Subject/Simulation/MySubject_simulation/gym_vrep-master/gym_vrep/envs/train_vrep.py", line 101, in
main(args)
File "D:/Personal/My_Subject/Simulation/MySubject_simulation/gym_vrep-master/gym_vrep/envs/train_vrep.py", line 86, in main
tensorboard_log='./log',
TypeError: init() got an unexpected keyword argument 'checkpoint_freq'
There are no two parameters in DQN():checkpoint_freq and checkpoint_path,but these two parameters are used in the function callback().How to solve this problem？


Multiple namespaces installation problems

If you do the full installation on a different namespace, the previous installation breaks.
We need to have 2 things to fix things:

In the installation to check if the dataset operator is running in another namespace and prevent re-installation
Explain on the wiki how to extend existing installation to multiple namespaces

FYI @davidyuyuan

In this case, it might be better to make the operator to watch all namespaces by default. We give users an option to watch a list of namespaces if they want to, and document how they can update the list later.


CoC Command Not Found

When using CoC for completion on enter it shows CoC Command not found

After pressing enter or arrow keys to complete it:

Fixed the picture

I am not coc expert so I guess you should have asked this question on coc github

I mean this plugin doesn't do anything coc specific.

Yes i would've done that but before submitting i was checking with my other files and it only happens with godot files. Other files works fine with CoC

Well, you can remove vim-godot and try again coc with gd file. If issue is still there (it should) then ask coc team. If not we will try to debug it.

If i remove vim-godot then the godot code completion won't show up it'll only show up based on text that already exist in the file. And if i complete it with for example _physics_process it got no problem at all. But when i use vim-godot again all the godot completion show up but when i pressed enter it deleted the completion and showed Command: not found instead

If i remove vim-godot then the godot code completion won't show up

That is strange. Probably coc depedns on filetype definition.
Could you, with removed vim-godot, do following:

open your gdscript file (godot engine should be already running)
your gdscript file should have empty filetype, i.e. set ft? should echo filetype=
run vim command :set ft=gdscript and check coc-completion

For me it looks like this:

Okay thank you that showed the completion but the problem still persist. I checked the CoC log since this only exist with godot files and found this

Looking at this my first thought was this problem has something to do with the language server instead of CoC or your plugin. But i'm very unsure

Okay thank you that showed the completion but the problem still persist. I checked the CoC log since this only exist with godot files and found this

Sure. I just wanted to let you know that vim-godot does nothing with coc or language server.

Looking at this my first thought was this problem has something to do with the language server instead of CoC or your plugin. But i'm very unsure

Looks like godot language server has some issues with your coc/vim setup.
Just for reference, I have following setup:
https://github.com/habamax/.vim/blob/22a7363b95494d5dbe317c5cd68a2efb6719e453/coc-settings.json#L8
https://github.com/habamax/.vim/blob/master/autoload/habacoc.vim

Yeah i'm not sure currently i've tried deleting my coc/vim setup still doesn't work but thank you so much for the help

Did you manage to fix the problem?

I found this workaround
https://github.com/godotengine/godot/issues/34523#issuecomment-674573792
I believe this happens with coc#_select_confirm() command. Coc have an example config which set  to this if pumvisible. If you don't use it it shouldn't be a problem. I personally feel this is not a good workaround.

My workaround was to disable coc from completing the first entry and just using tab to auto-complete. This is a bit annoying but it does make my sanity better.

My workaround was to disable coc from completing the first entry and just using tab to auto-complete. This is a bit annoying but it does make my sanity better.

I don't have that issue at all as @excalios mentioned it emerges when you setup coc to call _select_confirm which (probably) is not  implemented in godot lsp.
With my setup (https://github.com/habamax/.vim/blob/d3dba0102341bf392bc82ff7575f54f81efdfe33/autoload/lsp.vim#L40) Coc autocompletes with no errors given.


fix back button

the second time you hit the back button, the app.ui.buttons.paste reference is broken, due to this clone/swap hack, and el.parentNode is null

cheers


Clarification on README

As mentionned in #71, using the program option for type-checking can lead to Invalid source file: xxx.ts. Ensure that the files supplied to lint have a .ts, .tsx, .js or .jsx extension.
Using the base option of gulp.src corrects it, thought mentionning it in README can save some time for users.

Thanks 👍


Added configuration files to onboard new users

To onboard new users, I've added configuration files for Sophia Jobe, Verena Chung, Jared Hendrickson, and Wei-An Chen.

@BrunoGrandePhD please note that IT has not yet added them to the Jumpcloud groups, so if this is necessary before merging please hold off until I comment again.

@BrunoGrandePhD users have been added to the JumpCloud groups, so this should be ready to merge pending your approval

I've given myself read-write access to Verena and Sophia's project for the workshop. I would prefer to keep Jared and Wei-An's projects separate, and I have read/write access on them as well, so if you're not worried about batch limits on strides then we can leave it?


Revamp of Python directory structure and import system

Package import improvements to pyFUNtoFEM
Contents of the pyfuntofem directory are organized into driver, interface, model, and optimization subfolders.
New import statements take the form:
from funtofem import TransferScheme
from pyfuntofem.model import FUNtoFEMmodel, Variable, Scenario, Body, Function
from pyfuntofem.interface import createTacsInterfaceFromBDF, Fun3dInterface
from pyfuntofem.driver import FUNtoFEMnlbgs

Yup the import statements you put in the header of the pull request are correct with the updated pyfuntofem. Looks good to me. Can you add Dr. Kennedy as a reviewer as well?


Error: no buildable Go source files in...

Unsure if this is an issue with gvt, looking to get any feedback.
I was able to download the packages to vendor/ folder.
gvt fetch golang.org/x/sys
But now I am getting the error that it can not build the packages, even though the package is there.
no buildable Go source files in /Users/jesus/.gvm/pkgsets/go1.6/global/src/client/theclient/thepackage/vendor/golang.org/x/sys/windows/svc/eventlog

All other packages imported with gvt work fine.

Quick guess: are you building on a non-Windows machine, and do the files in that package all have a Windows-only build tag?
Sent from a small keyboard

On 19 mag 2016, at 22:45, Jesus Galvan<EMAIL_ADDRESS>wrote:
Unsure if this is an issue with gvt, looking to get any feedback.
I was able to download the packages to vendor/ folder.
gvt fetch golang.org/x/sys
But now I am getting the error that it can not build the packages, even though the package is there.
no buildable Go source files in /Users/jesus/.gvm/pkgsets/go1.6/global/src/client/theclient/thepackage/vendor/golang.org/x/sys/windows/svc/eventlog
All other packages imported with gvt work fine.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

Ahh, thanks.
And thank you for this package! Super helpful.


usb: fix enumeration issues by propagating buffer overflow error.

Summary
Fixes USB enumeration issues with fast host controllers.
The USB spec permits SETUP for control transfers to be sent at any time. The device is not permitted to STALL or NAK, so the USBD controller ignores BK0RDY status when it receives a SETUP.
Some faster host controllers may send a SETUP very soon after the previous OUT, for example the Raspberry Pi5 has been confirmed on USB analyzer to SETUP in as little as 20us after the previous OUT. This is a very narrow window to handle the OUT and maintain proper state.
In my project (atsamd21 at 32mhz cpu clock with USB at mid priority), by the time the OUT in the endpoint buffer is read it has already been overwritten by the SETUP packet. The USB stack is expecting a zero length packet to end the previous descriptor read, so it has passed in a zero length slice. The bank read finds that the endpoint has more than zero data and throws a BufferOverflow.
This change allows the error to propagate without dropping the SETUP in the endpoint buffer. The USB stack will then exit the previous descriptor read and proceed with the SETUP of the next control transfer.
In my project this fixes (at least) enumeration issues with Pi5, Pi CM4, and so far a couple of x86 class PCs running Windows or Linux. So far all hosts known to have enumeration issues that were tested with this change have been resolved.
Checklist

[ X] CHANGELOG.md for the BSP or HAL updated
[ ] All new or modified code is well documented, especially public items

Rebased on present HEAD~1 to avoid inheriting CI errors.

@mattthebaker, I'm going to run a few tests with this PR, but it seems like it solves some USB issues on my HW as well.

There's some infrastructure for testing USB, related to the usb-device crate.  Offhand, I don't know if it would be feasible to make a test case that reproduces this issue as the host side of that uses libusb, however it would be worth running the suite against this PR to catch any regressions (and to dust off the tests :) ).
A good entry point to the current situation might be this issue - sorry I've not advanced it, but am now gainfully unemployed and maybe even getting past a period of burnout so could try to get back in to it.  https://github.com/rust-embedded-community/usb-device/issues/100

This will need to be rebased to fix the ci errors.

@mattthebaker, I'm going to run a few tests with this PR, but it seems like it solves some USB issues on my HW as well.

Excellent!

There's some infrastructure for testing USB, related to the usb-device crate. Offhand, I don't know if it would be feasible to make a test case that reproduces this issue as the host side of that uses libusb, however it would be worth running the suite against this PR to catch any regressions (and to dust off the tests :) ).
A good entry point to the current situation might be this issue - sorry I've not advanced it, but am now gainfully unemployed and maybe even getting past a period of burnout so could try to get back in to it. rust-embedded-community/usb-device#100

Hmm, I'm not sure if it can be easily created in user space tests since the OS will typically handle enumeration and believe SETUP is primarily used there? Any test that resets a device might be able to look for re-enumeration within some short period of time, but the host would need to be confirmed to be susceptible to the issue.
Any kind of extended regression coverage would be nice.
Fix is already in production for me and will quickly see use on at least a couple hundred devices and a lot of unique hosts (pretty much all Linux though). During debug I tested against usb_device 0.3.1 and 0.3.2 -- new debug hooks there ultimately helped identify the root cause. My device only uses CDC/ACM.

@mattthebaker - thanks very much for your work on this, and apologies it's taken me so long to do anything with the PR!  I've run the usb-device test suite against your branch on SAMD21 and SAMD51 (Adafruit Metro M0 and M4 boards), can't see any change there.
I'm not quite sure this fixes the underlying issue, but I'm still getting back up to speed on USB and don't have a firm idea of a different fix.  A couple items I'm currently pondering, which you might have considered already:
In the circumstance where Inner::read()'s caller expects a ZLP, should an incoming SETUP result in the transfer (perhaps not the right terminology - what I mean is the data that might span multiple reads or require a ZLP to terminate) being discarded or accepted?  My hunch is that the transfer should be discarded, like this PR does.
How about the case where the caller expects more data in a transfer, so supplies a larger buffer than an unexpected SETUP?
I wonder if we should split Inner::read() in to two methods like Inner::read_setup() and Inner::read_data(), because it seems like we're currently trying to do two semantically different things with the same function.
Tangentially related: we should update the usb-device crate used by the HAL.  I'll put that on my TO-DO list.

The SAMD21 datasheet (DS40001882H) in section <IP_ADDRESS> describes management around SETUP transactions being handled largely in hardware.  The USB peripheral will overwrite the endpoint buffer with an incoming SETUP transaction so long as EPINTFLAG.RXSTP is 0; effectively, SETUP transactions can't overwrite each other, but they can overwrite non-SETUP transactions.
In a hypothetical Inner::read_data() method, I think the best we can do is check received_setup_interrupt() after the read has been attempted (and that needs to happen if the endpoint returned a buffer overflow error, to cover the ZLP case).  But, even that seems to leave a timing hole, because the datasheet describes the USB peripheral writing data before it sets the RXSTP bit...

I'm thinking the next step is to create a test case that aims to reproduce the issue that this PR is meant to fix.  How does the failure manifest - is it just that the device doesn't enumerate successfully at all, does it take noticeably longer (I guess due to a retry(, something else?

No problem, trying to help others that have likely seen the issue.

In the circumstance where Inner::read()'s caller expects a ZLP, should an incoming SETUP result in the transfer (perhaps not the right terminology - what I mean is the data that might span multiple reads or require a ZLP to terminate) being discarded or accepted? My hunch is that the transfer should be discarded, like this PR does.

So the specific case I'm aware of where this happens, the status stage of a control transfer gets its ZLP clobbered by the next control transfer's SETUP. In this case, that ZLP is the final action of the first control transfer, and on reception by the device it will be ACKed if the OUT buffer is ready. The device could STALL if its not ready, but then the host would try again rather than proceeding.
From the host's perspective the control transfer is now entirely complete with no errors. If it is a fast host it may move immediately into another control transfer SETUP.
If there were something the device side stack could discard, in this case I believe you would prefer to accept it. However, the only case I'm aware that this happens is when the control transfer is reading -- IN transfers -- so afaik there isn't any device bound transfer that could be uncertain. To resolve the issue the next control transfer will need to start correctly.

How about the case where the caller expects more data in a transfer, so supplies a larger buffer than an unexpected SETUP?

I'm not aware of a case where the host would behave this way, but I'm not an expert. AFAIK the rapid SETUP only comes due to how the final status stage works. In other cases if the state machines get out of sync or have bugs they would STALL / NAK until the host decides to just reset the device bus and try again. Only the SETUP per spec bypasses those mechanisms and can lead to this issue.
Splitting the read function into two might make sense.

In a hypothetical Inner::read_data() method, I think the best we can do is check received_setup_interrupt() after the read has been attempted (and that needs to happen if the endpoint returned a buffer overflow error, to cover the ZLP case). But, even that seems to leave a timing hole, because the datasheet describes the USB peripheral writing data before it sets the RXSTP bit...

Do you have a spot in the datasheet where you saw that comment about RXSTP? It seems that reading  received_setup_interrupt() may not reliable if it has its own race with the buffer state. If the expected ZLP reads incorrectly, aborting the read but keeping all other state valid for the next state advance seems appropriate still. Its closely equivalent to what you're describing other than the received_setup_interrupt() will be flagged when the interrupt fires or next state advance that has the read instead.

I'm thinking the next step is to create a test case that aims to reproduce the issue that this PR is meant to fix. How does the failure manifest - is it just that the device doesn't enumerate successfully at all, does it take noticeably longer (I guess due to a retry), something else?

It depends on how rapidly the host sends the SETUP and how quick the device is with clearing:

If the host is only marginally faster than the device at handling, the enumeration would often succeed, but some packets (like say the manufacturer name string) would get dropped. Or sometimes the first enumeration step would glitch and not recover, the usb controller would time out, restart and try again. So the enumeration time could be very slow (5-10s or more).
If the host is very fast compared to the device, it would fail enumeration on the first control transfer every time and never enumerate.

Harness triggers a restart of the device, expects it to enumerate within some reasonable amount of time

This should hit it directly with the caveat that the host its run on would need to have fast setup behavior and/or artificial delays slowing down the device to force the behavior. For reference it seems like many host controllers will only send 1 control transfer per start of frame, so 1ms, but others will blast them out with basically zero delay.

2. Harness writes some data to the firmware via a control transfer, then reads it back via another control transfer.  The harness should compare the data, and use amounts of data aimed at hitting problem areas (where ZLPs might be expected, and not).  If the device is too slow in reading out control data, it might erroneously get some of the next SETUP data.

If that is possible with the host libraries that would be great for extra targeted coverage.
A possible 3. is considering the timing dependent nature, would be to have a device build with intentional long delays (somewhere around 1-3ms?) inserted between handling events, and run significant parts of the existing regressions against it.

Cool, sounds like progress.  I don't have much time for programming today, but will try to implement a couple tests and a possible fix (which will involve changes to usb-device) soon.

I'm not aware of a case where the host would behave this way

It may be that a reasonable host implementation would never do that - I'd forgotten that the SETUP includes a length field, so the device knows how much data to expect.

Do you have a spot in the datasheet where you saw that comment about RXSTP?

Section <IP_ADDRESS> though it's a bit narrative.
I'm envisioning that read() will be changed to check the RXSTP bit near the end, and if it is set return an error (maybe InvalidState) without changing the peripheral state - this says the data (if any) has been overwritten by a new SETUP.  A new read_setup() will work much like read() now does, but return an error if there is data but RXSTP is not set, and clear RXSTP before returning successfully.  The logic in usb-device for control pipes will need to change slightly so that the new read_setup() is used when a setup is expected, and the places it reads regular data or ZLPs may need to handle that error.

It seems that reading received_setup_interrupt() may not reliable if it has its own race with the buffer state

Maybe it's not a real problem if the OUT ZLP is the only case where this occurs, since the device behaviour doesn't change depending on whether it successfully reads the ZLP or if the SETUP arrives before/during the ZLP handling.  I guess this hinges on whether a host can write OUT data that the device needs, then follow up with a SETUP too quickly (as above) - my hunch is that this doesn't happen.

It depends on how rapidly the host sends the SETUP and how quick the device is with clearing:

Do you have examples of devices (or maybe device+driver combos) in those two categories?  I've got a small assortment of SBCs and laptops, have seen the occasional weird enumeration issue but nothing as consistent as your 2.

If that is possible with the host libraries that would be great for extra targeted coverage.

I think it should be - the usb-device test suite is pretty minimal, but based on libusb which can do control transfers.

have a device build with intentional long delays (somewhere around 1-3ms?) inserted between handling events

Yep, that seems like a great idea.  Should be easy to do between calls to poll() from the normal device firmware, which I imagine would be sufficient.  Might be a bit awkward to put that functionality inside the usb-device code though, if that's necessary.

I'm envisioning that read() will be changed to check the RXSTP bit near the end, and if it is set return an error (maybe InvalidState) without changing the peripheral state - this says the data (if any) has been overwritten by a new SETUP. A new read_setup() will work much like read() now does, but return an error if there is data but RXSTP is not set, and clear RXSTP before returning successfully. The logic in usb-device for control pipes will need to change slightly so that the new read_setup() is used when a setup is expected, and the places it reads regular data or ZLPs may need to handle that error.

Sounds pretty clean to me.

Do you have examples of devices (or maybe device+driver combos) in those two categories? I've got a small assortment of SBCs and laptops, have seen the occasional weird enumeration issue but nothing as consistent as your 2.

The two I've debugged personally are my desktop PC -- an Intel i7-12700K / Z690 chipset and Win10 Pro -- and the raspberry pi 5. This is in conjunction with a samd21e clocked at 32mhz and RTIC where USB had to be lower priority than some real time tasks. Older firmware build that had a polling loop instead was slightly more reliable with the pi5 (more likely to glitch or slowly enumerate), but never worked on the desktop PC. My users had issues on some other various other embedded linux boards but I would need to hunt down old notes.
A Pi5 would be the simplest way to get a hardware test setup going, but need to note now that some field updates may have to be rolled back -- they issued their own fix (1 SETUP per SOF) as unrelated devices had similar issues, I'm not sure if they even ran a rust stack. https://forums.raspberrypi.com/viewtopic.php?t=365410
I'll be able to test any proposed alternate fixes in a pi5 setup.

I've started working on this today, but only got a bit of refactoring the test suite done.  WIP on usb-device is at https://github.com/ianrrees/usb-device/tree/enumeration .

I'll be able to test any proposed alternate fixes in a pi5 setup.

Great, thanks!  I don't have a Pi5 yet.

I've implemented the enumeration time test and found that the control loopback test exists already[1].  With my laptop and a Metro M0 Express (SAMD21, no bootloader) I see about 95% of enumerations happening in something like 8-900ms, but some are about 6 seconds.  Haven't yet made the delay test; I started looking at where to put the delay and didn't find a spot where it seemed to make the enumeration problem more likely.
Proposed fixes span usb-device and atsamd-rs, I'd suggest using https://github.com/ianrrees/test-usb-device/tree/enumeration as an entry point to try the changes out.  It's a little bit of a Git hack as the usb-device submodule commit is one from my usb-device repo.  Probably the easiest thing to do is clone the test-usb-device repo with --recursive, cd in to the usb-device directory and do git remote add ianrrees git@github.com:ianrrees/usb-device.git, git fetch ianrrees, cd .., git checkout enumeration, git submodule update.
With those changes, I haven't seen any enumeration failures in ~40 runs of the test suite.
[1] here, though we might want to revise the list of lengths.

Forgot to mention, there's a little bit of Cargo secret sauce that you probably know about already, but is quite helpful here.  The "patch" feature that can be used to replace the source of a crate through the whole dependency tree, for example here replaces usb-device.

@mattthebaker, just checking in on this as it's been a few days.  I've had a brief chat with the original author of usb-device and it seems the mixing up of SETUP and OUT data is a known issue so it would be good to PR a fix to usb-device, and once that's done we'd update the HAL to suit.
With my laptop, I can see the changes I referenced above make enumerations more reliable, but it would be good to confirm whether is addresses the complete enumeration failures you've reported with RPi5 as well.
Great find!  I suspect that sorting this out will fix a longstanding annoyance with https://github.com/ianrrees/usb-spi which is used at my old job.

would be good to confirm whether is addresses the complete enumeration failures you've reported with RPi5 as well.

Sorry, just realised I misread and the complete failures were with a desktop!
@9names ran some tests of my proposed fix with RPi5 today and was able to reproduce the failures using current code, and see that the fix it appears to work.  So, I'm thinking the next step is to get those changes in to usb-device, and once that's done we can make corresponding changes to the HAL.  Will close this PR out in the mean time, since the new approach would overwrite these changes anyway.
Thanks everyone!


Reconsider type inference for globals

Aim
It's quite unintuitive (sorry if Bug is a strong word) ATM when you have a constant
global MAX_PLAYERS = 2;
And then multiply it with a Field, it is suddenly a Field everywhere in your program. Top-level symbols should probably be have explicit types, like Rust top-level consts
Expected Behavior
I expected a for i in 0..MAX_PLAYERS loop to work even though MAX_PLAYERS is used in field math
If MAX_PLAYERS wont be polymorphic, it should probably require an explicit type
Bug

To Reproduce

Installation Method
None
Nargo Version
No response
Additional Context
No response
Would you like to submit a PR for this Issue?
No
Support Needs
No response

global MAX_PLAYERS = 2;

fn main(some_field_value: Field) {
    let _x = MAX_PLAYERS * some_field_value;

    for i in 0 .. MAX_PLAYERS {
        assert(i as u32 < 10);
    }
}

Works for me, perhaps your code has another issue?
As for the original question, I personally don't find the behavior too confusing but you can always optionally specify a type annotation if desired. We don't generalize the types of globals but we could in theory since they are compiled by substituting each use of them with their definition.

Closing as I can't reproduce now. Will reopen if I can.


Select Entity

[ ] Show selected entity (sprite)
[ ] Show options menu of the entity

select works
(also deselects all others) but requires player to be created


Return 200 and an empty list if no warning notes are found for a person

Describe this PR
Previously, we were raising an exception when no warning notes were found for a person.
This changes the functionality so that if there are no warning notes for a person, this will return the response as an empty list with a status code of 200.
What changes have we introduced
When no warning notes have been found for a specific person, we return an empty response instead of raising an exception.
Checklist

[ ] Added end-to-end (i.e. integration) tests where necessary, e.g. to test the complete functionality of a newly added feature
[X] Added tests to cover all new production code
[ ] Added comments to the README or updated relevant documentation (add a link to documentation), where necessary.
[X] Checked all code for possible refactoring
[X] Code pipeline builds correctly

Follow up actions after merging PR
Test in Staging

Looks good to me! I think we just need to remove the Swagger comment about returning 404 when no warning notes is found though.

Well spotted! 🙌🏾


Add some generic methods for dealing with selectors

In a Redux world, most get methods will just be selectors. Selectors will be higher order functions. The outer function will take a UID (unique ID that is given to the underlying tree node) and return a memoized selector which takes a state object and returns the requested value.
Because the inner selector is memoized, we want to reuse the same inner selector across multiple calls. To achieve that, we could call the outer selector for each getter inside our constructor and stash the inner selector as a property on the instance: _getHeightSelector.
However, this requires doing extra work up front and could get rather tedious to type out.
Instead, I've taken the approach of creating them lazily and then storing them in a map for future use.
Additionally, I've added some helper methods to take care of the tedious work of:

Lazily get/cache the inner selector
Passing in the uid upon creation
Passing in this._store.getState() on call
Comparing the selector value to the Maki value (we want to do both in parallel to assert parity until we can rip out the Maki tree approach)

The result give s a really nice developer experience. Adding a new getter is just a matter of defining the selector and then wrapping the original return value in:
return this._compareToUidSelector(makiValue, selector);

This is very nice, but it is awfully abstract, and thus, probably very hard to track what's going on.

@jberg I'd love to hear your opinion: Have I gone off the deep end, or is this abstraction worth it?