RangeType / SWE Common Confusion

OGC API - Coverages implementors are extremely confused as to how to populate the range type.
CIS 1.2.3 says:

1.2.3    SWE Common
The coverage RangeType component (see Clause 6) utilizes the SWE Common [4] DataRecord. Consequently, the semantics of sensor data acquired through SWE standards can be carried over into coverages without information loss. See also Annex D.4.

The definition of a DataRecord would be what does the value represent, NOT its data type.
However most implementations of OGC API - Coverages currently fill definition with URIs intended for the SWE Common dataType which is found deep inside a BinaryEncoding element ( 8.6.1 BinaryEncoding Element of http://portal.opengeospatial.org/files/?artifact_id=41157 ). For example "ogcType:signedShort" or "ogcType:float32"
Also it seems that full URLs should be used as URIs.
SWE Common standard is 193 pages and extremely difficult to follow.
There does not seem to be any example JSON encoding of a BinaryEncoding, and how exactly one would define that the data is encoded as a 32-bit float.
Is this something which should be described in the RangeType, or should that be left to the binary encoding only?
Also, I wonder if in the case of multi-bands this is really supposed to be separate fields, or if it should be one field with components? The SWE Common specifications are really complex so it would be good to have someone who knows the standard better brief the SWG, which could then provide practical example JSON encodings of range type for the 95% common use cases in CIS/Coverages.

@ghobona could you maybe help us to approach SWE Common for help on implementing proper examples as suggested by @jerstlouis? Thanks

In particular it was pointed out that the example http://schemas.opengis.net/cis/1.1/json/examples/10_2D_regular.json definition might not be right:
	"rangeType": {
		"@context": "http://localhost/json-ld/rangetype-context.json",
		"type": "DataRecordType",
		"id": "examples:CIS_RT_10_2D",
		"field":[{
			"type": "QuantityType",
			"id": "examples:CIS_RT_F_10_2D",
			"definition": "ogcType:unsignedInt",
			"uom": {
				"type": "UnitReference",
				"id": "examples:CIS_RT_F_UOM_10_2D",
				"code": "10^0"
			}
		}]
	}

ogcType:unsignedInt (or the proper URL URI http://www.opengis.net/def/dataType/OGC/0/unsignedInt) is intended for the DataType of BinaryEncoding, not for the definition of what this value represents, which should probably be something like .../elevation (e.g. Radiance in some valid examples).

In SWE Common, http://www.opengis.net/def/dataType/OGC/0/unsignedInt is defined in Table 8.1 on page 112, "URI to use in “dataType” attribute".
If I understand it correctly, this is the relevant relationship I gather from the UML diagrams dispersed over the 193 pages of SWE Common specifiations:
https://gist.github.com/jerstlouis/5740f956898c268e8253b5008bb9723d
From this, the dataType attribute of a Component, inside a ComponentOrBlock's byComponent, inside a BinaryEncoding's member, as a way to define an AbstractEncoding, inside a DataArray or a DataStream,
A DataArray would then be a valid field of a DataRecord, which is what CIS RangeType is defined as.
But in those CIS 1.1 examples, dataType is being used directly for the definition attribute of a Quantity, which is inherited from AbstractDataComponent via AbstractSimpleComponent.
My point is that http://www.opengis.net/def/dataType/OGC/0/unsignedInt is not a URI intended for that definition, it's strictly intended for binary encoding components data types.

The “definition” attribute identifies the property (often an observed property in our context) that the data component represents by using a scoped name. It should map to a controlled term defined in an (web accessible) dictionary, registry or ontology. Such
terms provide the formal textual definition agreed upon by one or more communities, eventually illustrated by pictures and diagrams as well as additional semantic information such as relationships to units and other concepts, ontological mappings, etc.
Examples
The definition may indicate that the value represents an atmospheric temperature using a URN such as “urn:ogc:def:property:OGC::SamplingTime” referencing the complete definition in a register.
The definition may also be a URL linking to a concept defined in an ontology such as “http//www.opengis.net/def/OGC/0/SamplingTime”
The name could be “Sampling Time”, which allows quick identification by human data consumers.
The description could be “Time at which the observation was made as measured by the on-board clock” which adds contextual details.

None of that suggests binary data types... But "The definition may also be a URL linking to a concept defined in an ontology" may be giving some permissions to do pretty much anything here?
In my opinion, this is all way too vague to provide the necessary interoperability as far as OGC API - Coverages is concerned.
An implementor cannot be expected to try to make sense of this whole CIS dependency on SWE Common without some very clear guidance and examples.

yes, ideally no integer etc is used, but a more meaningful SWE URL. Unfortunately, neither SWE never completed that URL universe so it still seems missing today.

CC @KathiSchleidt @joanma747

You definitely want to reference whatever it is the number being provided in the rangeSet represents, something like:
http://vocab.nerc.ac.uk/collection/A04/current/SPOT/
Side comment (not sure if fits here or elsewhere), I noticed an error in the JSON Schema for rangeType, "type" attribute of "field" is an enum, only contains "QuantityType", there are quite a few more under SWE

If I interpret it correctly this URL only says "SPOT", nothing else. However, the range type should express the technicality allowing to interpret it. The fact that it is SPOT does not tell me whether this is optical, SAR, etc, so I would rather expect that SPOT URL  under the coverage metadata.
Re JSON issues, can we collect them in some place systematically for discussion? I could create a page on https://external.ogc.org/twiki_public/CoveragesDWG/WebHome, or do you know any other good place?

Hi Peter,
as you know, I'm pretty clueless as to what you satellite boys actually measure, just interested in the data structure. But, if you take the time to click on the link, you'll find out  more. Also clueless if SPOT fits your current purpose, but if you navigate up one level from SPOT via broader, you come to:
http://vocab.nerc.ac.uk/collection/A04/current/SatelliteImagery/
Maybe something more suitable there.
Also - further information could be provided with each concept, and there are probably better vocabularies out there. I checked the nerc vocab as I know it from oceanographic properties, was quite surprised that also covers SatelliteImagery
:)

@KathiSchleidt yes, that link was the second thing I tried; unfortunately, the server does not respond for me.
But anyway, these are just atomic facts, but not an ontology giving further insights to some tool evaluating it. Likely there is relevant work around (and most likely overlapping as you hint), but it is not my core field, too, so I am not sufficiently informed.
IMHO this is just too large to clarify in one ticket, it definitely deserves a research project - anybody interested to team up under a suitable funding scheme?

@pebau ah - how did you find the 2nd link if the server is not responding? ;)
And - mixing the ObservableProperty (WHAT you're providing) with the measurement methodology behind it (the HOW of what you're providing) is just bad style from an O&M/SWE perspective - reason I've been pushing the approach of utilizing Observations to describe rangeType. Maybe too large to clarify in one ticket, but not sure we need a research project for this.
To my understanding, the approach of utilizing Observations as Coverage metainformation is not new, some folks are putting Observations into the Coverage Metadata. rangeType would be more correct. Need to take a 2nd look, but would be interesting to try and insert the Observation here

ah - how did you find the 2nd link if the server is not responding? ;)

interestingly the first link worked, clicking on the other one ran into timeout. Must be quantum fluctuations :)

Ah - aurora borealis futzing with your connections up north? ;)
http://vocab.nerc.ac.uk/collection/A04/current/SatelliteImagery/ provides the following narrower options:
Narrower | http://vocab.nerc.ac.uk/collection/A04/current/MODIS/
Narrower | http://vocab.nerc.ac.uk/collection/A04/current/LANDSAT/
Narrower | http://vocab.nerc.ac.uk/collection/A04/current/StableLights/
Narrower | http://vocab.nerc.ac.uk/collection/A04/current/SPOT/

I've had to have a long look at SWE Common, since the STA Tasking extension uses it. Especially the DataRecord type.
As far as I can see, the DataRecord type is the equivalent of a json class: A set of named fields, each field being an AbstractDataComponent. It is meant for grouping other components. The DataRecord type itself does not hold "data" as such, the subComponents do. As such, a DataRecord
However, the DataRecord UML Definition is broken, because it states:

The name of each field must be unique within a given “DataRecord” instance so that it can be used as a key to uniquely identify and/or index each one of the record components.

However, the AbstractDataComponent class does not define a field "name"... So each field in a DataRecord must have a unique name field that does not exist?

hm, my understanding always was that this allows to define the record components with their particular names, such as red, green, blue. (And no, I did not take "each field" literally = recursive, but just thought of the first level underneath Data Record.) Admittedly I did not reflect much on this, thought the SWEeties have made it all work. Difficult, though, was the lack of examples.

@alexrobin @mikebotts Please have a look at the starting comment above and provide your views.
Cc: @ogcscotts

Hi all,
A DataRecord is indeed a group of fields, each one with a unique name. The field element with the name attribute is defined in the DataRecordType in the (XML schema)[http://schemas.opengis.net/sweCommon/2.0/record_components.xsd] and UML (Page 45 of the implementation spec), not as part as the AbstratDataComponent class.
The intent of the definition attribute on the DataRecord is purely semantic. It is meant to provide semantic information about the record as a whole, but should NOT be used to carry data type information. For example, in the case of a coverage range, it could be used to tag a group of bands together whenever it makes sense (e.g. something like http://someURI/LANDSAT/InfraredBands), although in most cases I think the semantic information should be provided on each band separately. I agree that it is probably not the best place to tag the data as being observed from a specific satellite (e.g. http://vocab.nerc.ac.uk/collection/A04/current/LANDSAT). I think higher level coverage metadata is probably a better place for this type of information than down inside the range description. I would also like to remind you that the DataRecord definition is not mandatory, while the definition of each range field is.
SWE Common separates structure and semantics from encoding, so the exact data type used for each range field should be provided separately. One way is to provide a SWE Binary encoding descriptor but it could also be done differently if it makes more sense in the Coverage API (for example, it can be provided by the encoding format itself, as most format such as NetCDF, TIFF, GRIB do already).
Regarding the JSON encoding, there is no official OGC standard that defines it but we have released an OGC Best Practice document with mappings from either SWE Common XML or UML to JSON. The BP document also provides examples that can hopefully help. To reuse the example provided above, the correct encoding would be:
"rangeType": {
		"@context": "http://localhost/json-ld/rangetype-context.json",
		"type": "DataRecord",
		"id": "examples_CIS_RT_10_2D",
		"field":[{
			"name": "band1",
			"type": "Count",
                        "id": "examples_CIS_RT_F_10_2D",
			"definition": "http://someURI/LANDSAT/Band7"
		},{
			"name": "band2",
			"type": "Quantity",
                        "id": "examples_CIS_RT_F_10_2D",
                        "label": "SWIR Band",
                        "description": "SPOT-5 HRG SWIR Band, 1.58-1.75µm",
			"definition": "http://someURI/SPOT5/SWIR",
                        "uom": {
                             "code": "W.sr-1.m-2.Hz-1"
                        }
		}]
	}

The main changes are:

The type values must not include the Type suffix (DataRecord -> DataRecordType, QuantityType -> Quantity)
Each field must have a name
'id' properties must be valid against the XML ID datatype so cannot contain the : characters
The field definition should not be a data type, but reference semantic information about the band
If no unit is provided, the coverage value is often better modeled as a Count (i.e. digital pixel value) instead of a Quantity. I added a second field to show an example with an actual radiometric unit.

Note that it is possible in SWE Common to provide more information about the range values (like spectral domain, etc.) via the quality element in a more formal manner. I can provide examples if you're interested.

thanks, Alex! Any and all examples are welcome. Correct me if I am wrong, but I feel that CIS has done it almost right according to your material:

use of DataRecord (independent from the format) is pretty much as described AFAICS, except for what follows:
id in JSON contains ":" which disqualifies it for XML: "examples:CIS_RT_10_2D". However, our philsophy (which you may or may not share) was to allow max freedom per format and not impose the union of all restrictions to each format. Similar to PNG whose restrictions are not propagated into NetCDF.
Quantity in practice is used for data types simply in lack of a commonly accepted ontology  (such as an OGC resolver branch). Once there is common agreement, that could be promoted into the communities.
which entails BTW that the uom is "1", in UCUM expressed as "10^0".

So a main question is: do we have a common terminology for measurement types? Also, while I understand that Quantity is on a higher semantic level, can we transport the data type somewhere in the DataRecord if needed?

Stupid question - where do you take the "name" attribute from? To my understanding, SWE does not inherit from GML (not featureTypes), can't find name either in the UML model or in the XML encoding
:?
Kathi

PS: that uom remark of mine was nonsense in the sense that it is not directly related to the data type (mis)use under discussion.

Hi Peter,
Yes, I guess I should have started by saying that using the DataRecord to describe a coverage range was the right choice. It makes complete sense and keeps things well correlated with what we do in Sensor Web standards. The changes I recommended are just details. I do agree with your philosophy regarding not imposing ID syntax to the JSON encoding. We tend to do that in our implementation to ease translation between XML and JSON but it's not a hard requirement.
The best way to add the data type directly in the DataRecord description would be to add it as an extra property of each field I think. In XML we would do it via the SWE Common extensions hook, but in JSON, I don't see any problem making it an additional property of the object, like so:
"field":[{
    "name": "band1",
    "type": "Count",
    "dataType": "http://www.opengis.net/def/dataType/OGC/0/float32",
    "id": "examples_CIS_RT_F_10_2D",
    "definition": "http://someURI/LANDSAT/Band7"
}

In that case, you could use any of the data type URIs defined in OGC registry here, but you could also use any other datatype taxonomy.
Regarding semantics of measured quantities, we tend to use QUDT for fundamental physical quantities, and we also maintain our own (often more specific) properties in the SensorML ontology. We also maintain a spectral band ontology that could be of interest to this group to better qualify coverage bands.

@KathiSchleidt The name is an attribute of the DataRecord/field element in the XML schema. It doesn't appear directly in the UML model because it is implied as part of the <> property stereotype which was used in the UML to XML encoding rules at the time.

@alexrobin Thank you so much for all the great insights on this issue!
I tend to agree that the data type is more a property of the encoding, so I would support leaving it up to the binary encoding to define this. Given that the CIS JSON representation of the data type is intended to describe the coverages potentially available in multiple encodings, which may well use different data types.
The missing semantic aspect, and misuse of the definition as a binary data type in the CIS 1.1 examples really was the core of this issue.
@pebau Regarding the units of measure, a unit of 1 or 10^0 tells me absolutely nothing as a user in what physical units things are measured. That is another part of the examples I had trouble with. W.sr-1.m-2.Hz-1 does.
@alexrobin Something else we were wondering (https://github.com/opengeospatial/ogcapi-coverages/issues/94) is how one could possibly describe in the DataRecord the minimum / maximum value that one might encounter, either in the dataset specifically, or in general. Would SWE have something for that?
And something related to the intersection of both these issues: how could one specifies how numerical values map to a physical unit, when some kind of scale factor, or linear mapping with an offset as well, has been established?

@alexrobin but - also doesn't seem to be in the 2.0 version of the XML Schema, least not the one I'm accessing!
are we both refering to the DataRecordType from:
http://schemas.opengis.net/sweCommon/2.0/record_components.xsd

@jerstlouis Yes, SWE has a mechanism to define constraints on each field. In JSON, a min/max constraint would look like:
{
    "name": "band1",
    "type": "Count",
    "dataType": "http://www.opengis.net/def/dataType/OGC/0/float32",
    "id": "examples_CIS_RT_F_10_2D",
    "definition": "http://someURI/LANDSAT/Band7",
    "constraint": {
        "type": "AllowedValues",
        "interval": [ [0,255] ]
    }
}

There can be several intervals, hence the double array.
There is no built-in way in SWE to define a scale factor and offset. Scale factors can be provided in the UCUM uom code, either as a multiplier (e.g. 25.m) or as a divider (e.g. [deg]/60), but this is probably not enough for the coverage use case.
One possibility would be to think of scale and offset as an encoding issue as well, since AFAIK, they are mostly used for data storage optimization/compression purposes. Most geospatial file formats can handle that (at least HDF, NetCDF, GRIB, GeoTIFF do), but I guess we could have an issue if one of them doesn't...

@KathiSchleidt We are definitely talking about the same SWE Common schema. The name is declared as an attribute of the field element of DataRecordType.
<complexType name="DataRecordType">
  ...
  <element maxOccurs="unbounded" minOccurs="1" name="field">
    <complexType>
      <complexContent>
        <extension base="swe:AbstractDataComponentPropertyType">
          <attribute name="name" type="NCName" use="required"/>
        </extension>
      </complexContent>
    </complexType>
  </element>
  ...
</complexType>

@alexrobin
Sorry, my bad, hadn't seen the name attribute tacked on, was looking for an element like gml:name. Should have just looked at my own old examples!
May I ask if you see any issues with the encoding below? What I came up with for air quality reporting years ago, would be curious if I did it correctly!
swe:DataRecord
<swe:field name="StartTime">
<swe:Time definition="http://www.opengis.net/def/property/OGC/0/SamplingTime">
<swe:uom xlink:href="http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"/>
</swe:Time>
</swe:field>
<swe:field name="EndTime">
<swe:Time definition="http://www.opengis.net/def/property/OGC/0/SamplingTime">
<swe:uom xlink:href="http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"/>
</swe:Time>
</swe:field>
<swe:field name="Verification">
<swe:Category definition="http://dd.eionet.europa.eu/vocabularies/aq/observationverification"/>
</swe:field>
<swe:field name="Validity">
<swe:Category definition="http://dd.eionet.europa.eu/vocabularies/aq/observationvalidity"/>
</swe:field>
<swe:field name="Value">
<swe:Quantity definition="http://dd.eionet.europa.eu/vocabulary/aq/primaryObservation/hour">
<swe:uom xlink:href="http://dd.eionet.europa.eu/vocabularyconcept/uom/concentration/ug.m-3"/>
</swe:Quantity>
</swe:field>
</swe:DataRecord>

@alexrobin Great that we can do things like 25.m or  or as a divider [deg]/60.
As UCUM is yet another dependency that implementors might not be familiar with, I suggest CIS/OGC API - Coverages should again provide guidance and clear examples for the 80% common use cases scenarios.

Most geospatial file formats can handle that (at least HDF, NetCDF, GRIB, GeoTIFF do), but I guess we could have an issue if one of them doesn't...

That is exactly the trouble. This comes up for example trying to use a generic 16-bit PNG to encode the values. The GeoTIFF Gridded Coverage Extension for example allows to define a linear transform on either a global or per-tile basis, and that transform is stored in ancillary tables.
This might be outside the scope of SWE Common/RangeType, but at the moment we have nowhere else for this. I wonder whether HTTP response header would be an appropriate place for this to complement the encoding that don't support this (which could be localized to the subset response)?

@alexrobin @Schpidi
A practical question regarding relating to implementing coverages styling in our client:
How could the range type / SWE Common DataRecord describe that the sentinel-2 radiance values are 12-bit values, which should be scaled down by 1.0 / 4096 for example? (assuming my understanding is correct)
Would that be something like "4096.W.sr-1.m-2.Hz-1" ?
If the GeoTIFF encoding uses 16-bit integers, is there any GeoTIFF tag inside GeoTIFF that would otherwise carry that information? (so that the CIS JSON and GeoTIFF encoding are consistent, and can share the same range type information).
If that tag is not present, that would force the CIS JSON encoding to be consistent, so that the values there are also scaled up to 12-bit values by multiplying by 4096.
Thank you!

FYI GeoTIFF includes no information on radiometry, this is part of TIFF, with 3 TIFF tags BitsPerSample and PhotometricInterpretation + SamplesPerPixel.
GeoTIFF is only (this is its scope) CRS definition and georeference of image / grid to ground CRS

Thank you @EmDevys . What I should have said is TIFF of GeoTIFF tags

By the way, I already pointed out that documenting the rangeType in CIS need some rules and illustration by some valid examples. I submitted an proposal for Discussion Paper in 2017, that I presented to Coverage.DWG but there was no solid action following this (see https://portal.ogc.org/files/?artifact_id=72894&version=1).

And the Coverage rangeType should document such information on the coverage data (for all types of coverages including grids/raster/images) in an harmonised way. Ready to contribute any action on this

@EmDevys Please join us next Wednesday morning call 9:00 AM Eastern if you can! I hope we can focus on this, I think it is critical. I suggest we develop clear examples, and perhaps the Coverages API could standardize this, if not CIS 1.2.

I’ll try to arrange to participate, and review this into more details.
Emmanuel Devys
IGN Département Normalisation et référentiels projets| Service des Projets et Prestations
direction des programmes et de l’appui aux politiques publiques
T +33 (0) 1 43 98 85 75
ign.frhttp://www.ign.fr/ – geoportail.gouv.fr
De : Jerome St-Louis<EMAIL_ADDRESS>Envoyé : jeudi 10 décembre 2020 17:51
À : opengeospatial/ogcapi-coverages
Cc : Emmanuel Devys; Mention
Objet : Re: [opengeospatial/ogcapi-coverages] RangeType / SWE Common Confusion (#102)
@EmDevyshttps://github.com/EmDevys Please join us next Wednesday morning call 9:00 AM Eastern if you can! I hope we can focus on this, I think it is critical. I suggest we develop clear examples, and perhaps the Coverages API could standardize this, if not CIS 1.2.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/opengeospatial/ogcapi-coverages/issues/102#issuecomment-742647148, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACDHG2XQ2DMSBTGZMJVM6WLSUD37LANCNFSM4TJVVIGQ.

@jerstlouis In SWE standards, we don't usually use data records to provide encoding level details (bits per sample, etc.). This information is usually provided separately (in a DataStream or DataArray encoding section, see BinaryEncoding for example).
However, I see no issue defining an extension to be embedded inside each SWE Common field to provide such info within the record itself. In that case, one possibility would be to create a JSON object that would contain all encoding information in one place, not only the data type.
For instance, going back to our earlier example, it could be something like:
{
    "name": "band1",
    "type": "Quantity",
    "id": "examples_CIS_RT_F_10_2D",
    "definition": "http://someURI/LANDSAT/Band7",
    "encodingInfo": {
        "dataType": "http://www.opengis.net/def/dataType/OGC/0/unsignedInt",
        "significantBits": 12,
        "scale": 0.000244141,
        "offset": 0
    }
}

Note that dataType and significantBits are properties that are already defined in SWE Common, as part of the BinaryEncoding object I mentioned above. They are used to mean exactly the same thing, although they are used in a different place in the schema. But at least, calling them the same here would keep things semantically consistent.

We could even offer the possibility to provide the scale as a ratio (e.g. "1/4096") if we want to keep the maximum precision.

I must confess that I find it uneasy to elaborate any religion on which semantics should be documented in the rangeType for Coverages, which is an harmonized model for various type of coverage data (raster / grids, but not only) and various types of encoding standards, and in fine various types of values (integers - signed or not, float, double, complex) ... For the semantics of the rangeType would gain to be clarified, as well as the Coverage model and schemas, as identified in this issue.
My impression is that binary encoding is the scope of the encoding standard (as reminded previously in this discussion), and may be unnecessary in the Coverages API (except if restricted intervals of values apply, which is a usage constraint).
The rangeType should document the semantics for the Coverage range, and Coverage values usage, at minimum. For example elevation, radiometry, codelists for thematic imagery ... Should the rangeType go beyond (at the byte or bit level) this is to be discussed / confirmed.
By the way, the DP document on Coverage rangeType semantics referenced above (https://portal.ogc.org/files/?artifact_id=72894&version=1) was applying to CIS 1.0 / GMLCOV, but is still valid on CIS 1.1 and its JSON encoding.
Last point, in addition to coverage data, users may get following information, sometimes overlapping (e.g. in INSPIRE with data such as orthoimagery or elevation), and therefore with some redundant information:

encoding format metadata (e.g in TIFF/ GeoTIFF or netCDF or JPEG 2000/GMLJP2 or PNG or JPEG  - last 2 not geo-enabled)
coverage rangeType, under SWE Common data record
additional metadata (source, quality, content description ...), handled under 19115 set of standards, for discovery and cataloging.

Presumably the Coverages API should be able to handle such information (as available), but I am not sure the Coverage model and corresponding XML or JSON schemas should handle all this. But probably in a similar way as the WCS up to now.
Just some thoughts ...
I apologize, I am not in a position to contribute to this work other than reviewing and commenting when the proposed draft specification is submitted.

@EmDevys you are making a very valid point indeed - great that you remind us of the big picture from time to time!
Having units of measure in the range tpye - we likely agree - is essential. UCUM has been picked not because of its beauty but...well, it is a standard. Injecting all sorts of extra mechanics like styling, transformation into other value systems, etc. establishes some (increasingly complex) microsyntax which less than the majority of tools is likely to implement.
Further, a general mechanism requires very intricate scrutiny about consequences in all possible coverage applications. What about defining metadata records instead that go into the coverage metadata slot? INSPIRE is doing that already, and there is place for more as long as we define own namespaces and manage metadata root elements somewhere (lightweight: on the DWG wiki, heavyweight: OGC-NA). That would allow us to establish building blocks capturing format and application specifics while still going together without friction. Ideally we would offer encodings for each in XML, JSON, and RDF.
Just some wild pre-Xmas idea.

@pebau the trick we did for INSPIRE by shifting their excess baggage to the metadata slot (they'd extended RectifiedGridCoverage by derivation, doesn't work with WCS) should be discussed a bit more if we want to do this correctly, as to my view this type of detailed information on the content of the range would actually be more suited to rangeType (still learning Coverage models by doing ;) )
Please correct me if I'm wrong, but the metadata pertains to the entire range, while the rangeType can reference individual components. For INSPIRE I'm not that worried providing info in metadata as will probably be only one range component, but the moment multiple range components are provided, it becomes unclear what the info in the metadata describes, thus rangeType is more suited.
I do believe that some clean valid examples would be very valuable. In this context I'd also like to explore the potential of utilizing the new O&M V3 ObservationCharacteristics for enhanding the swe:DataRecord (to my reading, this could be added to the extension of the SWE Types provided in the swe:field.) While I don't work with classical coverage topics, I could provide a CIS 1.1 encoded O&M Depth profile to the mix.

@KathiSchleidt
Referring to individual bands in the metadata is no problem, it just needs to be defined properly in the schema. There is precedent with the spatial extension where in a metadata slot you also can indicate regions (in space and time!) to which the particular metadata record applies. EO-WCS uses even polygons to isolate contributing footprints.
Changing the range type per se smells like CIS 2.0 so has a high probability of making existing coverages incompatible - not sure this high price is worth it if alternatives exist.
If however you find a way of circumventing a CIS 2.0 I'd sleep way better - so I'm excited about your vision of a solution that both SWE and CIS enjoy!

Coverages SWG call:
We need to fix and potentially extend the existing examples and want to add further ones based on the comments above.
Current list of examples:

[ ] 00_metadata.json
[ ] 05_2D_index.json
[ ] 10_2D_regular.json
[ ] 11_2D_regular_fileref.json
[ ] 12_2D_regular_fileref_multiband.json
[ ] 15_with-envelope.json
[ ] 20_3D_height.json
[ ] 25_3D_time.json
[ ] 30_4D_height+time.json
[ ] 40_1D_regular.json
[ ] 45_2D_distorted.json
[ ] 46_irregular+distorted.json
[ ] 50_3D_partitioned-1.json
[ ] 55_1D_timeseries-partitioned.json
[ ] 60_3D_timeseries-multipart.json
[ ] 65_1D_timeseries-interleaved.json
[ ] 70_2D_interpolation.json
[ ] 80_sensormodel.json
[ ] 90_point-cloud.json

Proposed additions:

[ ] Classification
[ ] DEM (how to handle both float & int encoding for the same coverage)
[ ] RGB
[ ] RGBA
[ ] Multispectral & Panchromatic, i.e., bands with different resolution
[ ] Hyperspectral

Let me suggest to not just add the examples, but comment them and make clear what is normative, what is convention, what is experimental, etc. - the Coverages.DWG might be a good place to host such background information.

At the 2020/12/16 SWG meeting, I suggested that the range type should provide semantic information about range set values and how to interpret reference real-world values, but not encoding details which may be specific to only some encodings such as binary data types and type sizes

hmm... if rangeType is settled to that point, it may be worth revisiting allowing the provision of the new ObservationCharacteristics type from O&M V3 as an alternative rangeType description - would make for far easier crosswalks between representation types.
Thoughts?

hmm... if rangeType is settled to that point, it may be worth revisiting allowing the provision of the new ObservationCharacteristics type from O&M V3 as an alternative rangeType description - would make for far easier crosswalks between representation types.
Thoughts?


softDelete undefined.patch issue

I just noticed that in the published npm code, the softDelete is compiled from
return this.patch(hook.id, hook.data, hook.params).then(data => {

to
return undefined.patch(hook.id, hook.data, hook.params).then(function (data) {

Don't think that's correct. I'll dig around more tomorrow if I get a minute.

Need to add a proper test.

I'm writing unit tests. get with the above changes returns the soft deleted item.
Minus: subsequent code has to check if item is deleted. Plus: A softDeleted item can be read and 'undeleted' without hacking with hooks.iff().
I'm inclined to leave it as is. What say you @daffl ?

softDelete rewritten with tests in v2.0.2


Question about how to run nfsp?

I noticed that there is "See open_spiel/python/examples/nfsp.py for an usage example." in open_spiel/python/algorithms/nfsp.py.
But I can't find this example.
Is there an example of using nfsp, or how can I quickly create one based on the existing content?

Wow it is the second time in two days that examples that were there before suddenly vanished.
We will catch this example thief and bring them to justice!
(I will look into it and follow-up.)

Found it! https://github.com/deepmind/open_spiel/blob/master/open_spiel/python/examples/kuhn_nfsp.py
But we should indeed fix that comment to point to the correct file name :) So please leave this issue open, I will submit a fix to correct the comment and it will be closed when merged.

@lanctot thanks~I found it. It was not stolen, just changed the name.


Add lesson tags

Links to #62
As part of the lesson tag filter feature, a user should be able to filter lessons by tag(s). This issue should cover just the JS implementation of adding tags to lessons.
Hint: you can add the tag field/property somewhere within addLesson() and you can check to see if this works within renderLessons.

I'll try this issue!

Go for it @MSoup 😄

Closed by #93

Closed by #93


Videos for these audios?

I want to access the videos associated with these audios. I used the VideoIDs from these audio filenames but youtube says " the video is unavailable". Please help. Thanks

Nobody has not yet commented on this thread.

Some of the videos are now unavailable on youtube so it is impossible to get the video associated.

I guess all of the videos for weakly labelled audios are unavailable. BTW thanks for the confirmation.

No most of them are available:
Examples of the two first ones:
https://www.youtube.com/watch?v=KK227gPpRn4
https://www.youtube.com/watch?v=-6p32k2WUCs

I couldn't find the videos you mentioned in the dataset. I guess we are talking about different datasets. I am referring to task 4 of the competition. Following are the sample audios from the weak folder of the audio dataset in task 4:
Y-_A4yJmxFdM_30.000_40.000
Y_CZzm6jUxbo_30.000_40.000
Y_KNIJQJ-CHA_30.000_40.000
As written the dataset description, Y-_A4yJmxFdM will be the VideoID for the video associated with the first audio. So I tried using these videoIDs for almost every audio in this folder and for all of them, the youtube says "Video unavailable".
Let me know where am I failing ?

The Y at the beginning is audioset convention but shouldn't be put in the ID on youtube.
So youtube IDs:
Y-_A4yJmxFdM_30.000_40.000 --> -_A4yJmxFdM
Y_CZzm6jUxbo_30.000_40.000 --> _CZzm6jUxbo
Y_KNIJQJ-CHA_30.000_40.000 --> _KNIJQJ-CHA
You would find it in dataset/download_data.py (line 41):
query_id = filename[1:12]
Then line 60:
https://www.youtube.com/watch?v={query_id}

Gotcha! Thanks so much!


New: Python packaging support

Version is determined from git revision through setuptools-scm.
Is python 2.7 really the minimum supported?

LGTM


✨ Find/Replace in Document

Is your feature request related to a problem? Please describe.
We should be able to find strings of text in a document and browse different occurrences.
Concept

@lukepistrol Work has been done in the past on this. Now that we are working on a new editor view will this need to be done again or can this work translate over?


bug: e2e source cdc test fails in main job

Describe the bug
found here: https://buildkite.com/risingwavelabs/main/builds/3879#01879dd7-9d12-4b77-ab66-c215f4257770
wait for recovery finish
--
  | check mviews after cluster recovery
  | e2e_test/source/cdc/cdc.check_new_rows.slt                   .. [FAILED]
  |
  | failed to run `e2e_test/source/cdc/cdc.check_new_rows.slt`
  |
  | Caused by:
  | query result mismatch:
  | [SQL] select cnt from orders_cnt;
  | [Diff] (-expected\|+actual)
  | -   4
  | +   3
  | at e2e_test/source/cdc/cdc.check_new_rows.slt:7

To Reproduce
No response
Expected behavior
No response
Additional context
No response

@StrikeW Please help to take a look? 🙏


Update dependency @augment-vir/common due to the deprecation of run-time-assertions

Title. Installation of projects dependent to this plugin will show a deprecation warning since run-time-assertions was already deprecated.

resolved in next release

released in v4.0.0


Error with Astream_events and AsyncSqliteSaver after migrating to v0.3

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangGraph/LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangGraph/LangChain rather than my code.
[X] I am sure this is better as an issue rather than a GitHub discussion, since this is a LangGraph bug and not a design question.

Example Code
import sqlite3
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
from langgraph.graph import StateGraph

from typing import Literal
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_core.runnables import ConfigurableField
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from langgraph.prebuilt import ToolNode

@tool
def get_weather(city: Literal["nyc", "sf"]):
    """Use this to get weather information."""
    if city == "nyc":
        return "It might be cloudy in nyc"
    elif city == "sf":
        return "It's always sunny in sf"
    else:
        raise AssertionError("Unknown city")

tools = [get_weather]
model = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
final_model = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

model = model.bind_tools(tools)
# NOTE: this is where we're adding a tag that we'll can use later to filter the model stream events to only the model called in the final node.
# This is not necessary if you call a single LLM but might be important in case you call multiple models within the node and want to filter events
# from only one of them.
final_model = final_model.with_config(tags=["final_node"])
tool_node = ToolNode(tools=tools)
from typing import TypedDict, Annotated

from langgraph.graph import END, StateGraph, START
from langgraph.graph.message import MessagesState
from langchain_core.messages import BaseMessage, SystemMessage, HumanMessage

def should_continue(state: MessagesState) -> Literal["tools", "final"]:
    messages = state["messages"]
    last_message = messages[-1]
    # If the LLM makes a tool call, then we route to the "tools" node
    if last_message.tool_calls:
        return "tools"
    # Otherwise, we stop (reply to the user)
    return "final"

def call_model(state: MessagesState):
    messages = state["messages"]
    response = model.invoke(messages)
    # We return a list, because this will get added to the existing list
    return {"messages": [response]}

def call_final_model(state: MessagesState):
    messages = state["messages"]
    last_ai_message = messages[-1]
    response = final_model.invoke(
        [
            SystemMessage("Rewrite this in the voice of Al Roker"),
            HumanMessage(last_ai_message.content),
        ]
    )
    # overwrite the last AI message from the agent
    response.id = last_ai_message.id
    return {"messages": [response]}

workflow = StateGraph(MessagesState)

workflow.add_node("agent", call_model)
workflow.add_node("tools", tool_node)
# add a separate final node
workflow.add_node("final", call_final_model)

workflow.add_edge(START, "agent")
workflow.add_conditional_edges(
    "agent",
    should_continue,
)

workflow.add_edge("tools", "agent")
workflow.add_edge("final", END)

# Memory
memory = AsyncSqliteSaver.from_conn_string("test.db")
app = workflow.compile(checkpointer=memory)

# Astream_events for token streaming
inputs = {"messages": [("human", "what's the weather in nyc?")]}
config = {"configurable": {"thread_id": "1"}}
async for event in app.astream_events(inputs, config, version="v2"):
    kind = event["event"]
    tags = event.get("tags", [])
    # filter on the langgraph node name
    if kind == "on_chat_model_stream" and event["metadata"].get("langgraph_node") == "final":
        data = event["data"]
        if data["chunk"].content:
            # Empty content in the context of OpenAI or Anthropic usually means
            # that the model is asking for a tool to be invoked.
            # So we only print non-empty content
            print(data["chunk"].content, end="|", flush=True)

### Error Message and Stack Trace (if applicable)

```shell
trace (sync version): https://smith.langchain.com/public/7742def4-35e5-4578-af7f-c1b418c53eee/r

AttributeError                            Traceback (most recent call last)
Cell In[36], line 3
      1 inputs = {"messages": [("human", "what's the weather in nyc?")]}
      2 config = {"configurable": {"thread_id": "1"}}
----> 3 async for event in app.astream_events(inputs, config, version="v2"):
      4     kind = event["event"]
      5     tags = event.get("tags", [])

File /opt/miniconda3/envs/yuichan/lib/python3.12/site-packages/langchain_core/runnables/base.py:1377, in Runnable.astream_events(self, input, config, version, include_names, include_types, include_tags, exclude_names, exclude_types, exclude_tags, **kwargs)
   1372     raise NotImplementedError(
   1373         'Only versions "v1" and "v2" of the schema is currently supported.'
   1374     )
   1376 async with aclosing(event_stream):
-> 1377     async for event in event_stream:
   1378         yield event

File /opt/miniconda3/envs/yuichan/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py:1006, in _astream_events_implementation_v2(runnable, input, config, include_names, include_types, include_tags, exclude_names, exclude_types, exclude_tags, **kwargs)
   1004 # Await it anyway, to run any cleanup code, and propagate any exceptions
   1005 try:
-> 1006     await task
   1007 except asyncio.CancelledError:
   1008     pass

File /opt/miniconda3/envs/yuichan/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py:966, in _astream_events_implementation_v2.<locals>.consume_astream()
...
--> 741     self.checkpointer_get_next_version = checkpointer.get_next_version
    742     self.checkpointer_put_writes = checkpointer.aput_writes
    743 else:

AttributeError: '_AsyncGeneratorContextManager' object has no attribute 'get_next_version'

### Description

I've been using AsyncSqliteSaver with astream_events in my langgraph app for a while. The recent upgrade to langchain v0.3 and langgraph v0.2 broke it. The message is not very descriptive but looks like a bug from the checkpointer class.
The code snippet above can reproduce the error.

### System Info

runtime
langchain_core_version: "0.3.1"

langchain_version: "0.3.0"

library: "langsmith"

platform: "Linux-6.5.0-1021-aws-x86_64-with-glibc2.36"

py_implementation: "CPython"

RENDER_GIT_COMMIT: "16d83d0c987788b782d3c8bb27c8ce34eaba911f"

runtime: "python"

runtime_version: "3.12.2"

sdk: "langsmith-py"

sdk_version: "0.1.122"

Thanks for the question! The issue is with how you are initializing the checkpointer. You can solve this in 2 ways:
Option 1:
async with AsyncSqliteSaver.from_conn_string("test.db") as memory:
    # The rest of your code

Option 2
conn = await aiosqlite.connect("test.db")
memory = AsyncSqliteSaver(conn)
# The rest of your code

Let me know if you still have issues!


WTHIT does not show bee or chicken info (V4.1.0)

WTHIT does not show info that Jade normally does, eg bee breeding. Possible replace?

https://github.com/badasintended/megane/issues/73

Replaced by Jade https://github.com/Sirfirelord1/Petty-s-Cozycraft/issues/13


In RTL mode, padding of iziModal-header works like a counter!

Hi @dolce ,
I used iziModal with rtl: true option but it dose not show the header contents (text and close button) to me. When I inspect the modal in Chrome debugger, encountered this:

Do you have some code in your script that do something like this?
Note: It works nice with rtl: false
Version: 1.5.1

I have the same problem. how should I fix it?

#233
this will fix the issue.


Failed to load resource: the server responded with a status of 405 (Method Not Allowed)

When we click on the "NEW DECK" button we see the following error in the console window (Developer Tools)
"Failed to load resource: the server responded with a status of 405 (Method Not Allowed)"
In order to fix this, edit the index.html file and change the Method from GET to POST

        $scope.newDeck = function () {
            $scope.status = 'Creating a deck';
            $scope.cards = null;
            $http({
                method: 'GET',   --------------------------------> This needs to be POST
                url: $scope.baseUri + '/deck/new',
                headers: {
                    'Content-Type': 'application/json'
                }
            }).success(function (data) {
                $scope.deckId = data;
                $scope.status = 'Deck created';
            });

Can you make a PR for this so we can merge it? There was another PR a couple of days ago changing the HttpGet to a HttpPost on the /deck/new and it broke this.

Made a PR for this. Change is required in index.html in both the AzureCards & AzureCards.WebApp


Notebook showing how to persist/load a optimization result

Persisting the result of an optimisation run is a useful tool to have. Either for plotting later on, creating a history of your research or sharing your results with others.
Having a notebook that gives a clear recommendation on how to do this would be nice. This avoids people doing "crazy" things.
Could be as simple as using joblib.dumps(result)

Opinions on providing skopt.load and skopt.dump as shortcuts to sklearn.externals.joblib.dump/load?

No strong opinion, fine with me.

How should we handle res.specs['args']['func'] when serialising the result object? It might contain weird things that can't be pickled and you might want to share your results with others who don't have the code for it. Sharing pickles between different users/python versions/machines is probably not the smartest thing anyway so should we worry about this? It could be that joblib makes promises on top of pickle??
I'd be interested in a method that allows the user to share their results with others (including their future self which might be on a different python version). This is potentially much harder to solve...

We can have an optional return_specs, which the user should be intelligent enough to set to be True if he wants to pickle or we can backport cloupickle, pyspark does it over here (https://github.com/apache/spark/blob/master/python/pyspark/cloudpickle.py)

Umm. Isn't using joblib.load and dump over-kill for our-case? It says it is useful only for large numpy arrays (http://scikit-learn.org/stable/modules/model_persistence.html) Our largest data structure is a list of lists.

Reason for suggesting joblib is that it is the recommended way to pickle scikit-learn classifiers, which can (in essence) be large numpy arrays. Not sure how applicable that is to our regressors.
I would strongly favour a solution where we don't have to maintain the pickling code ourselves (straight pickle or joblib or some other library).
I would make the "save specs or not" flag an argument of the save() function.

Hmm.. but then how do we serialize lambdas :(

Do we understand why they aren't picklable by default? The "hacks" I've seen to make them (and other common things that aren't picklable) are usually quite short, so it seems like there must be a good reason why the python devs don't just include that no? We should find out.
We should try and balance the (realistic) use cases with maintaining our own pickle substitute.

I guess that saving results is really important to a lot of people and most of them will have weird closures in res.specs["args"]["func"]. My current hack/workaround looks like this:
del res.specs["args"]["func"]

with open('result.pkl', 'wb') as f:
    pickle.dump(res, f)

What do you think of following:

skopt should be able provide a closure-free variant of OptimizeResult.
both pickle and joblib approaches are demonstrated in examples (with pro & contra)
the users should decide themselves which they want to use

Any opinions on dill? Would add an extra dependency.
I probably lean towards providing skopt.save(res) which removes func before saving via pickle.

Haven't used dill before, but looks promising.
The problem is, my lambda is a thin wrapper around a MATLAB engine calling a Simulink model. So I guess there's nothing that can pickle it :smile:.
Just to make it unambiguous: What should skopt.save(res) do exactly? Should it...

return a pickled bytestring?
write to a file using pickle?
return a clone of res without func?

I'd prefer the latter option. I think I can add this function and an appropriate example.

It should save things to a file. Looking back at the discussion and considering the naming convention in other packages it would probably make sense to introduce skopt.dump(obj, fname) and skopt.load(fname) (double checking the pickle interface). Probably could start life as something very similar to what you implemented for yourself using pickle.

Thanks for the clarification! load and dump sound really reasonable. I'll take a look at the joblib's interface as well.
Do you think ./skopt/utils.py is an appropriate module for these functions?

Do you think ./skopt/utils.py is an appropriate module for these functions?

👍

Are you still working on this? Otherwise I might jump on this.

Sorry, I hope I'll have it done till Sunday evening. I'll also comment here in more detail in an hour or so.

So, I'm very sorry for the double delay. I've done some research and tried to find different approaches to this problem.

Idea 1: As discussed, implement skopt.dump() and skopt.load() using pickle. If the objective function is unserialisable, skopt.dump() throws a warning and proceeds by creating a deep copy of the optimisation results object, removing the objfun from the copied object and storing it into a file.
Unfortunately, one has to use a deep copy, which is not so good for very large objects (but hey, one should use joblib in this case anyway). This is PR #245 (WIP, waiting for your feedback before proceeding).
Idea 2: Do not invent the wheel and just provide a notebook with some examples. Advantages: Fewer functions in skopt (= fewer tests required, fewer things that can go wrong) This is PR #246
Idea 3: Provide a __getstate__(self) method so that pickle or joblib know what to serialise. This proved to be a total tarpit. First, one cannot detect from inside of __getstate__ if the objfun is serialisable or not. Second, it is impossible to monkey patch an instance of OptimizationResult with this method afterwards. One can do it, but then it causes an infinite recursion when actually calling __getstate__. So the only possibility is to subclass OptimizationResult in skopt, which is IMHO a bad idea.
Idea 4: Use pickle's persistance_id. This seemed to be quite an elegant solution, pickle everything except for callables. However, I couldn't make the unpickler work. I suppose that removing all the callables during pickling somehow corrupts the resulting files.

Anyway, feel free to close one of either PRs; I'll try to finish the approved one without delays this time. :grinning:


How to stop the job?

Used example in PyCharm 2020.1 on Windows 10
def my_job(*args, **kwargs):
    print('args={} kwargs={} running at {}'.format(args, kwargs, datetime.datetime.now()))

Cron().schedule(
        Tab(name='run_my_job').every(seconds=5).run(my_job, 'my_arg', my_kwarg='hello')
    ).go()

After trying stopping a console it looks like the job is still running but just without calling my_job

Did you ever figure out what's happening here?  I only intended to support linux-based systems with crontabs, but if you've gotten it to work with windows, I'm interested.  I cannot reproduce your issues on my mac.

With jupyterlab on manjaro(an arch-based linux distro), I interrupted the kernel at 2021-07-02T19:21:24.800226, and things went as Berezhnyk described.

But everything works fine under a terminal, so it doesn't hurt much if not fixed.


Fix clock connections in InferReadWrite

SHOULD BE MERGED AFTER #306 AND CHANGE BASE TO MASTER

This fixes #307, and should be merged quickly due to ucb-bar/chisel3#288.

Thanks @donggyukim, this works for me

is this safe to merge?

@aswaterman Yes.

thanks, i'll merge once tests pass


Experimental fix for [issue 2312]

This pr is an experimental fix for https://github.com/DioxusLabs/dioxus/issues/2312
it doesn't change the underlying logic, just the events that's tracked.
(this is just me adding random events to the track list until it fixes the hot-reload issue)
This works under helix and vscode, not sure if neovim has the same behaviour
@Andrew15-5 mind testing?

If we have an image or something called img.png and then we want to rename it. If the file name is changed to image.png first in the Rust code, then the dx will show the unknown file error. After renaming the file (mv img.png image.png) the watcher won't see it and the workflow will be broken. Maybe re-saving the Rust file will fix this.
This is just an example that I come up with on the spot. This means that the potential FS manipulations can degrade the DX for everyone just so that you can use the default saving behavior in Neovim. I don't think this is worth it, if what I describe above is true. I don't really even know why other watchers (although I haven't used much of them, or I just can't remember them) don't complain about the Neovim's backup saving shenanigans. I only faced this problem with Rust/notify-based watchers (Dioxus/Typst CLI).
I can test this later (in a few days?), now I'm mostly busy.

In our recent CLI overhaul I incorporated your changes into our watcher - will test against the original issue.

https://github.com/DioxusLabs/dioxus/issues/2312#issuecomment-2247186938
So, for that commit this isn't even necessary.


docs(networking-manager): added "Default Player Prefab" infos

:tada: This PR is included in version 8.2.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update nf-winuser-setfocus.md

Documentation should specify what causes different errors. For example, during debugging of large app, I got ERROR_INVALID_PARAMETER, it it's really not obvious why.
Adding comment:
Extended error ERROR_INVALID_PARAMETER (0x57) means that window is in disabled state.

@NikoAri : Thanks for your contribution! The author(s) have been notified to review your proposed change.


[New chain]: Haqq Mainnet

Chain Name
Haqq Mainnet
RPC URL
https://rpc.eth.haqq.network
The chain must be added to https://chainlist.org/

[X] I confirm that the chain is added to chainlist

After creating the issue, a bot will estimate the required pre-fund and post it in the comments. Please check this checkbox after you send the pre-fund.

[x] I sent the pre-fund in accordance with Github Action's comment

Relevant information
No response

0xd3f71485f8c417c7f37cf352d0ba0ab555d9f3aa53174979fa0e43bc2652abd0


Fixing sas9 tests

Issue
#468
Intent
Fixing the 2 failing tests on SAS9
Implementation

Request with debug on - failed because of sufficient condition check in the code and parsing webout that is already parsed.
Make err and parse log - order of execution was wrong.

Checks
No PR (that involves a non-trivial code change) should be merged, unless all four of the items below are confirmed!  If an urgent fix is needed - use a tar file.

[x] Code is formatted correctly (npm run lint:fix).
[x] All unit tests are passing (npm test).

[ ] All sasjs-cli unit tests are passing (npm test).
[ ] All sasjs-tests are passing (instructions available here).

It might be good to bring in the changes from your sasjs-tests automation branch in here, so you can check if they pass automatically.


CASMINST-3509: Correct wipe_ncn_disks_for_reinstallation

Summary and Scope
CASMINST-3509: Correct wipe_ncn_disks_for_reinstallation
Several commands were added that had been used on the test on hela and the overall flow of the steps was adjusted to be more explicit upon which type of node the command should be run.
Also, indicate that install/create_ncn_metadata_csv.md will be pulling data from the SHCD HMN tab.
Issues and Related PRs
Testing
Tested on:
Test description:
Risks and Mitigations
Pull Request Checklist

/backport main

/backport release/1.2

Made new PRs for 1.2 and main


Editing --> "Border" - has to be an option "outer" instead of "inner"

How about to make an option to allow for a border be outer (enlarging width and height of the gif) instead of inner space?
Because, for example, I want to make a border 10 pixels wide on every side... and it covers gif so bad.

Done! It will be available with the next version (v2.19), in a few days.
You can simply put negative values to expand the border outwards.

NOTE: When expanding the frame size, the border will be applied to all frames, since the project must have a constant frame size.

By the way, you'll be able to add border as a task:


Not installable on PHP 7.2

There is a set of uninstallable dependencies:
$ composer global require edgedesign/phpqa --update-no-dev

Results in
Using version ^1.18 for edgedesign/phpqa
./composer.json has been created
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - fabpot/php-cs-fixer v2.0.0 requires php ^5.3.6 || >=7.0 <7.2 -> your PHP version (7.2.2) does not satisfy that requirement.
    - fabpot/php-cs-fixer v2.0.0 requires php ^5.3.6 || >=7.0 <7.2 -> your PHP version (7.2.2) does not satisfy that requirement.
    - fabpot/php-cs-fixer v2.0.0 requires php ^5.3.6 || >=7.0 <7.2 -> your PHP version (7.2.2) does not satisfy that requirement.
    - Installation request for fabpot/php-cs-fixer (locked at v2.0.0) -> satisfiable by fabpot/php-cs-fixer[v2.0.0].

Running update with --no-dev does not mean require-dev is ignored, it just means the packages will not be installed. If dev requirements are blocking the update you have to resolve those problems.

Installation failed, deleting ./composer.json.

That's not an issue that can be fixed in phpqa.
You have some conflict in your global composer.json (php-cs-fixer is not even required dependency).
Docker image is proof enough of phpqa + php 7.2:

Sorry, my bad, I did have some conflicts. Thanks!


Larger images are still visible when not active.

I am trying to use SimpleSlider (Polymer version) with different sized images, with the following CSS:
simple-slider {
	width: 100%;
	height: 500px;
	background: #fff;
}
simple-slider img {
	height: 500px;
}

And when smaller images are displayed, you can still see larger images in the background. This is because the slider relies on z-index only and doesn't set the other slides' visible properties.

A workaround should now be possible using the onChange and onChangeEnd callback added on v0.6.0
Although it might not be the perfect solution, it allows for fixing the issue if needed - there might be also honest use cases for wanting to have different sized images displayed at the same time.


Fixed the intrinsic model to make the dimension of the prior correct

Hi Max,
You had a small error in the intrinsic part of the model.  Basically when you set alpha=1 the dimension of the support of the prior goes from n to n-1 in this case, so the log density wasn't quite right.
I added a little bit of explanation to the text as well.
I needed to add a gcblibPermit() statement as well, because otherwise it wouldn't knit on my machine.
Best wishes!
Dan

Thanks Dan. Nice catch, and a very nice write-up! Appreciate your time on this - open source at its best!


Remove defunct make target for conformance tests

Conformance tests are loaded via the "normal" integration tests anyway. This is defunct as the package in question isn't even vendored.

/lgtm

I think test-integration stopped working. I think it'd be nice to have a makefile that runs both the integration and the conformance tests locally using the cluster created by make cluster-local. That was the original idea, but not sure if it's can still be done easily.

@davidor Right, things are fairly broken in that regard. It's on my TODO to fix as well FWIW but feel free to take a crack at it too.


Preformatted code blocks are ignored

The code (below) between triple backquotes is not output in the bootprint HTML file:
swagger: "2.0"
info:
    title: XXX
    description: XXX
    version: "0.1"
host:
    api.xxx.org
schemes:
    - https
produces:
    - application/json
paths:

    /foo/bar:
      get:
        summary: Foo bar
        description: |
              Lorem ipsum dolor sit amet
              ```{
              id   : "xxx",
              xxx:["xxx"
              name : {
              en:" xxxx",
              fr:""
              }
              ```
        responses:
          200:
            description: |
              Lorem ipsum
            schema:
              type: integer

I guess it should be
```
{
instead of
right? I've verified the bug.

I've fixed it and possible #17 as well.
Could you please verify using<EMAIL_ADDRESS>and<EMAIL_ADDRESS>

Closing. Please comment if anything still does not work

I confirm this is now fixed, many thanks!


Deprecate the plugin

This deprecates the plugin, by making it clear that it is no longer needed once version 5+ of govuk-frontend has been released.
See also:

adding the task list template to the common-templates plugin
Decide what to do with the Prototype Kit Task List plugin

This shouldn't be merged until v5 of govuk-frontend has been released, and then the Prototype Kit updated.

@nataliecarey does this make sense to you?

@colinrotherham @stevenjmesser this can be merged and released now. It deprecates the plugin and marks it as non-compatible with govuk-frontend v5.
Once merged and released the repo can probably be archived too?

If you're happy to give this a 2nd approval @querkmachine we can figure out how to publish it
Shall we save time and do a version bump (see example) in this PR?

I've created https://github.com/alphagov/design-system-team-internal/issues/851 so we can archive the repo. I'm not 100% sure we have the full access to do it but we can find out and ping whoever needs to do it.


TimeoutException: com.microsoft.cognitiveservices.speech.util.SafeHandle.finalize() timed out after 10 seconds

09-04 08:57:17.109  3437  3449 E AndroidRuntime: FATAL EXCEPTION: FinalizerWatchdogDaemon
09-04 08:57:17.109  3437  3449 E AndroidRuntime: Process: com.xxx.xxx, PID: 3437
09-04 08:57:17.109  3437  3449 E AndroidRuntime: java.util.concurrent.TimeoutException: com.microsoft.cognitiveservices.speech.util.SafeHandle.finalize() timed out after 10 seconds
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at com.microsoft.cognitiveservices.speech.util.SafeHandle.releaseRecognizerHandle(Native Method)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at com.microsoft.cognitiveservices.speech.util.SafeHandle.close(Unknown Source:29)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at com.microsoft.cognitiveservices.speech.util.SafeHandle.finalize(Unknown Source:0)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:319)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:306)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at java.lang.Daemons$Daemon.run(Daemons.java:140)
09-04 08:57:17.109  3437  3449 E AndroidRuntime: 	at java.lang.Thread.run(Thread.java:1012)

when start recognize use recognizeOnceAsync , but sometimes this error occur. please help
	pushStream = AudioInputStream.createPushStream();
	// customize audio record
        AudioRecordManager.getInstance().startRecord(audioChunk -> {
            pushStream.write(audioChunk.toBytes());
        });
        AudioConfig audioConfig = AudioConfig.fromStreamInput(pushStream);
        SpeechRecognizer speechRecognizer = new SpeechRecognizer(speechConfig, audioConfig);
        task = speechRecognizer.recognizeOnceAsync();

when stop recognize call  pushStream.close()
        try {
            AudioRecordManager.getInstance().stopRecord();
	    if (pushStream != null) {
                pushStream.close();
	    }
            if (task != null) {
                try {
                    SpeechRecognitionResult result = task.get(15, TimeUnit.SECONDS);
                } catch (Exception ignored) {
                }
            }
        } catch (Exception ignored) {
        }

logfile.txt

Hi @mackwu828 this is Darren from the Speech SDK team. Please try the following change in your code: BEFORE the line pushStream.close(), add this line: speechRecognizer.stopContinuousRecognitionAsync().get();. And yes, I see that you use recognizeOnceAsync() and not startContinuousRecognitionAsync()... calling stopContinuousRecognitionAsync should work to stop both.
Also note that you need to explicilty call .close() on  audioConfig, speechRecognizer when you are done using the recognizer, or define them in a try-with-resources block such that .close() is automatically called.

thanks! i will try this

Hi @mackwu828 this is Darren from the Speech SDK team. Please try the following change in your code: BEFORE the line pushStream.close(), add this line: speechRecognizer.stopContinuousRecognitionAsync().get();. And yes, I see that you use recognizeOnceAsync() and not startContinuousRecognitionAsync()... calling stopContinuousRecognitionAsync should work to stop both.
Also note that you need to explicilty call .close() on audioConfig and speechRecognizer when you are done using the recognizer, or define them in a try-with-resources block such that .close() is automatically called.

Can you help me? I have the same question in my app.
Fatal Exception: java.util.concurrent.TimeoutException: com.microsoft.cognitiveservices.speech.util.SafeHandle.finalize() timed out after 60 seconds
at com.microsoft.cognitiveservices.speech.util.SafeHandle.releasePropertyHandle()
at com.microsoft.cognitiveservices.speech.util.SafeHandle.close(:55)
at com.microsoft.cognitiveservices.speech.util.SafeHandle.finalize()
at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:339)
at java.lang.Daemons$FinalizerDaemon.processReference(Daemons.java:324)
at java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:300)
at java.lang.Daemons$Daemon.run(Daemons.java:145)
at java.lang.Thread.run(Thread.java:1012)
   And my code:
  withContext(Dispatchers.IO) {
         SpeechSynthesizer synthesizer = new SpeechSynthesizer(speechConfig, audioConfig);
         SpeechSynthesisResult result = result = synthesizer.SpeakSsmlAsync(ssml).get();
         result.close();
  }


chore: removing duplicate folder

chore: removing duplicate folder

@jyothishkjames hi, there. Many thanks for your contributions. We've just merged your previous PR yesterday. Would you please merge master into your branch and solve the conflicts?

@nan-wang Hi, I have resolved the conflicts.


chore: commit build files so that report works

@nobbynguyen would you be able to approve this please? It' just a bunch of build files that i've had to move to /docs folder so that the report can build!


any

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Issue Number: N/A
What is the new behavior?

Other information

Codecov Report
Patch coverage: 100.00% and project coverage change: -0.43 :warning:

Comparison is base (dcd2055) 64.95% compared to head (1d8bbb5) 64.53%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #688      +/-   ##
==========================================
- Coverage   64.95%   64.53%   -0.43%
==========================================
  Files         228      221       -7
  Lines       22242    22094     -148
==========================================
- Hits        14448    14259     -189
- Misses       7033     7062      +29
- Partials      761      773      +12

Flag
Coverage Δ

51.19% <96.29%> (-0.60%)
:arrow_down:

1.17.x
?

1.19.x
?

1.20.x
64.53% <100.00%> (-0.10%)
:arrow_down:

integration
51.19% <96.29%> (-0.60%)
:arrow_down:

macOS
?

secure
51.19% <96.29%> (-0.60%)
:arrow_down:

ubuntu
46.37% <92.30%> (+0.09%)
:arrow_up:

unit
46.37% <92.30%> (-0.02%)
:arrow_down:

windows
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

internal/xsql/context.go
61.90% <ø> (-4.20%)
:arrow_down:

internal/bind/auto_declare.go
91.42% <100.00%> (+0.51%)
:arrow_up:

internal/bind/bind.go
65.38% <100.00%> (+8.24%)
:arrow_up:

internal/bind/numeric_args.go
91.66% <100.00%> (+0.14%)
:arrow_up:

internal/bind/positional_args.go
90.00% <100.00%> (+0.25%)
:arrow_up:

internal/bind/table_path_prefix.go
100.00% <100.00%> (ø)

internal/logger/log.go
51.96% <100.00%> (ø)

internal/xsql/connector.go
54.81% <100.00%> (+4.00%)
:arrow_up:

internal/xsql/dsn.go
51.56% <100.00%> (ø)

sql.go
44.57% <100.00%> (ø)

... and 21 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Fix broken link

It looks like the link for Thoth.Json changed at some point. Here is an updated link.

@0x6a62 thank you!


Surahs nav visible above footer

@mmahalwy I don't seem to be having this problem in Firefox...

@mmahalwy I had previously posted this wasn't happening for me, I was wrong I looked at it incorrectly, afwaan.
I'm currently working on this.
Would it be good to just pull the footer over also, covering the nav as well?

Footer should go over everything @muhsinali

going to close this


Fix activity diagram for schedule command

Coverage remained the same at 88.937% when pulling 3a46ff157a14627bf1135c962dd28f2c3cf1de99 on AyushChatto:fixActivityDiagram into 81ac5fa6f2c8eb26c68e36521c5596825e79e78a on CS2103-AY1819S1-W13-1:master.


Setting a Foreground on a SplitButton does not apply to the dropdown half

re: https://github.com/microsoft/terminal/pull/13871
See source:
https://github.com/microsoft/microsoft-ui-xaml/blob/33732458ede38166f9127675898e4760c82b89bb/dev/SplitButton/SplitButton.xaml#L290-L293
We were setting Foreground on the SplitButton we use as our new tab button. However, the visual styles for the secondary button use Foreground="{ThemeResource SplitButtonForegroundSecondary}" , so they'll just always ignore the Foreground property (even if the primary button respects it).
Also, this seems to have regressed in #3887, which first shipped in 2.6.1
https://github.com/microsoft/microsoft-ui-xaml/blame/33732458ede38166f9127675898e4760c82b89bb/dev/SplitButton/SplitButton.xaml#L292
If this is by design, that's fine - but it should be documented as such.

Nothing has changed here ; still an issue.

Hey @zadjii-msft, this is by design, since SplitView's primary and secondary texts uses different brush colours. Templatebinding both elements to SplitView.Foreground will not allow them to consume different brushes, and ultimately remove flexibility in customization.
I agree much work should be done in terms of documentation since it's not immediately obvious what element Foreground (nor Background) sets. Happy to discuss further for ideas.


add a few hints

email hints for testing. + no good, but use dots to test easily

Closing because there was no follow up on feedback points within several months.


Add Snyk Docker scan for the frontend Docker image

Is the issue open?

Yes, it is, @Rajdeep1311. You can also use Trivy or Docker Scout.
Good hunting.

Sir, I raised a PR please do check @aerabi

The PR only has updates in the Dockerfile's comments. The CI job didn't end up there. Maybe caused by git merge or something? 🤔

But I added lines to scan the file sir

Where did you add it, @Rajdeep1311?
The PR has only changes to the Dockerfile.

Oh okay sir, can you please suggest what could be done to scan the image sir?


Handling ECS host autoscaling policies

Good work on this module.
I had a question. How do you handle autoscaling of ECS hosts?

Thank you.
I don't have any code to show you because currently I can predict our load and don't need autoscale. But I did have a look in to it. You have different options, depending on how fancy or fast you need to be able to scale up or down.
Assuming you don't need to scale up/down very often, let's say once a month. I wouldn't make anything fancy but create a trigger on CPU or Memory usage and when that exceeds the threshold manually scale your EC2 and your containers.
If you need to scale your containers up/down constantly then AWS already can do that. As you can see here https://aws.amazon.com/blogs/compute/automatic-scaling-with-amazon-ecs/
You can provide scaling options for the EC2 instances as well as your services, based on CPU.
Instead of using the CPU metric you could use your own metric like: (average processing time above X, more than X amount of messages in queue, etc...)
Maybe @nathanpeck has some tips as well?
Hope this helps.


Force normalization in convolution

The deconvolved beam kernels should be normalized when convolving. See #388

Coverage remained the same at 74.723% when pulling 7b76399ecde2a622f8b1b79309dbe0292544926b on e-koch:force_normalization into 827bdca68762ecc5296e6f2a2b8c028f4f331a2b on radio-astro-tools:master.

Coverage remained the same at 74.723% when pulling 8928d968afea2a207c4e50fc19a484f873562111 on e-koch:force_normalization into 827bdca68762ecc5296e6f2a2b8c028f4f331a2b on radio-astro-tools:master.


Updated error condition for invalid tokens

Description
After merging this PR: #451, the canaries broke due to the invalid token check changing. This PR will check to determine if  the string contains invalid token before deleting the DiscoveredCluster resources.
Related Issue
If applicable, please reference the issue(s) that this pull request addresses.
Changes Made
Updated operator to check if the error contains invalid token.
Screenshots (if applicable)
Add screenshots or GIFs that demonstrate the changes visually, if relevant.
Checklist

[x] I have tested the changes locally and they are functioning as expected.
[ ] I have updated the documentation (if necessary) to reflect the changes.
[x] I have added/updated relevant unit tests (if applicable).
[x] I have ensured that my code follows the project's coding standards.
[x] I have checked for any potential security issues and addressed them.
[x] I have added necessary comments to the code, especially in complex or unclear sections.
[x] I have rebased my branch on top of the latest main/master branch.

Additional Notes
Add any additional notes, context, or information that might be helpful for reviewers.
Reviewers
Tag the appropriate reviewers who should review this pull request. To add reviewers, please add the following line: /cc @reviewer1 @reviewer2
/cc @cameronmwall @ngraham20
Definition of Done

[ ] Code is reviewed.
[ ] Code is tested.
[ ] Documentation is updated.
[ ] All checks and tests pass.
[ ] Approved by at least one reviewer.
[ ] Merged into the main/master branch.

/lgtm


Pushbutton long press function triggers on first normal button press

I believe there is a small logic error in the abutton.Pushbutton.buttoncheck coro:
ticks_diff(ticks_ms, ticks_change) returns an erroneous result on the first button press, because t_change has not been changed from None to a ticks value. This causes the long press function to be triggered directly, when the button is pressed for the first time (i.e. before the release and even before the long press timeout has passed).
The solution is, I believe to set t_change before the while loop is entered to a sufficiently high number, e.g.
t_change = time.ticks_ms() + time.ticks_add(0, -1)

See the example for http://docs.micropython.org/en/latest/library/utime.html#utime.ticks_add
# Find out TICKS_MAX used by this port
print(ticks_add(0, -1))

Interesting. In my tests (when I implemented it) I did not find any bugs and even though I am using the library, it didn't throw any errors or unintended activations either. However, I would expect time.ticks_diff(0,None) to throw an exception, which it doesn't, it somehow calculates with None.. So that's a bit weird.
Were you able to reproduce the problem or is it "just" a code analysis?
Your proposed solution could work around the issue but it wouldn't need t_change to be TICKS_MAX, it would be enough to set t_change to the current time.ticks_ms() because the difference from the current time has to be bigger than self.long_press_ms for it to trigger.

Turns out I was using the the Pushbutton class wrongly. I didn't realize that __init__ already schedules a buttoncheck task, so I scheduled it again, which triggered the observed behaviour:
import uasyncio as aio

from pyb import Pin
from abutton import Pushbutton

def cb():
    print("Long press triggered.")

async def button_coro(pin):
    print('Button coro started.')
    btn = Pushbutton(Pin(pin, Pin.IN, Pin.PULL_UP), suppress=True)
    btn.long_func(cb)

    print("Entering button coro loop.")
    await btn.buttoncheck()
    print('Display coro done')

async def main():
    print("Starting button coro...")
    button_task = aio.create_task(button_coro('A6'))

    print("Entering wait loop.")
    while True:
        await aio.sleep(1)

aio.run(main())

The correct way would presumably be just:
import uasyncio as aio

from pyb import Pin
from abutton import Pushbutton

def cb():
    print("Long press triggered.")

async def main():
    btn = Pushbutton(Pin('A6', Pin.IN, Pin.PULL_UP), suppress=True)
    btn.long_func(cb)

    print("Entering wait loop.")
    while True:
        await aio.sleep(1)

aio.run(main())

I find it a bit odd, that the buttoncheck task is scheduled before you have a chance to attach the callback(s). I'll probably remove that in the copy of abutton.py in my project and start the coro manually.
Anyway, seems this can be closed as invalid, though it might still be prudent to set t_change correctly at the start of the loop.

This library was built to be a drop-in replacement for Peter Hinch's aswitch library: https://github.com/peterhinch/micropython-async/blob/master/aswitch.py
Therefore his documentation about the usage is valid for my library.
(The goal was to reduce coroutine usage and work around a cancellation bug in the old uasyncio v2 version. Now that is not important anymore with v3)

I find it a bit odd, that the buttoncheck task is scheduled before you have a chance to attach the callback(s). I'll probably remove that in the copy of abutton.py in my project and start the coro manually.

Where's the problem with that? The task isn't executed until you yield to the scheduler, so until then you can safely modify the callbacks without any button checks being executed.
I might still give it some thoughts to check if there is a possibility of running into an error if t_change is None.

Where's the problem with that?

Well, it's imaginable (though probably unusual) that the scheduler is already running, when Pushbutton is instantiated, isn't it?
Apart from that, it's just a matter of preference / object semantics, I guess. I think it's unexpected for an object initialization call to do that kind of thing, i.e. potentially trigger some ongoing task. But I realize that this is done elsewhere too, e.g. blah = SomeService(host, port) triggering a connection.

Yes the scheduler might already be running, which is mostly the case in my applications. But the task still doesn't get executed until the current task yields (calls await asyncio.sleep(x)) so you would still be safe to change everything in your current coroutine/task.
Hmm I guess it does depend on preferences and application design.


Bug: bump nuget package versions

Expected Behaviour
Update all nuget package versions to latest releases
Current Behaviour
NA
Code snippet
NA

Possible Solution
No response
Steps to Reproduce
NA
AWS Lambda Powertools for .NET version
latest
AWS Lambda function runtime
dotnet6
Debugging logs
No response

Released v0.0.2-preview.


Merge remote-tracking branch 'CenturyLinkCloud/master'

If you are trying to sync your local fork to the CenturyLinkCloud/master you have the pull request backwards.

@DLM1 @chrislittle is correct.  Please see https://github.com/CenturyLinkCloud/PublicKB#merging-an-upstream-repository-into-your-fork for instructions.
There are no changes here.  Closing PR.


ClientAddr does not correspond to Client IP when proxy protocol enabled

Do you want to request a feature or report a bug?
Bug
Did you try using a 1.7.x configuration for the version 2.0?

[ ] Yes
[X] No

 ELB AWS (proxy protocol enabled via annotation service.beta.kubernetes.io/aws-load-balancer-proxy-protocol=*) --> Ingress controller Traefik (proxy protocol enabled via trustedIPs in [entryPoints] in toml).
Checking with tcpdump command in traefik pods, we can see that proxy protocol header is propagated as expected: ...6..cTCP4 91.XXX.XXX.XXXX 10.41.XXX.XXX 33668 443 but when we check traefik logs in ClientAddr we see the internal elb ip not the client real one:{"BackendAddr":"XXX.XXX.XXX.XXX:8000","BackendName":"XXX.XXX.XXX.XXX","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"XXX.XXX.XXX.XXX:8000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"10.41.XXX.XXX:61714","ClientHost":"10.41.XXX.XXX","ClientPort":"61714","ClientUsername":"-","DownstreamContentSize":108,"DownstreamStatus":202,"DownstreamStatusLine":"202 Accepted","Duration":6529152,"FrontendName":"XXXXXXXXXX","OriginContentSize":108,"OriginDuration":6397011,"OriginStatus":202,"OriginStatusLine":"202 Accepted","Overhead":132141,"RequestAddr":"XXXXXXXXXXX","RequestContentSize":176,"RequestCount":20681965,"RequestHost":"XXXXXXXX","RequestLine":"POST /ob-events/v2/invoice-payment-due HTTP/1.1","RequestMethod":"POST","RequestPath":"/ob-events/v2/invoice-payment-due","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2019-05-20T08:15:29.727365143Z","StartUTC":"2019-05-20T08:15:29.727365143Z","downstream_Access-Control-Allow-Credentials":"true","downstream_Access-Control-Allow-Origin":"https://XXXXXXXXXXXXXXXX"...
Output of traefik version: traefik:1.6.0-alpine
What is your environment & configuration (arguments, toml, provider, platform, ...)? K8s with kops in AWS
checkNewVersion = false
maxIdleConnsPerHost = 1024
defaultEntryPoints = [ "http", "https" ]
logLevel = "INFO"

[traefikLog]
  format = "json"

[accessLog]
  format = "json"
  [accessLog.fields]
    defaultMode = "keep"
    [accessLog.fields.headers]
      defaultMode = "keep"
      [accessLog.fields.headers.names]
        "Authorization" = "drop"

[entryPoints]
  [entryPoints.http]
    address = "[::]:80"
    [entryPoints.http.redirect]
      entryPoint = "https"
    [entryPoints.http.proxyProtocol]
      trustedIPs = ["<IP_ADDRESS>/22"]
  [entryPoints.https]
    address = "[::]:443"
    [entryPoints.https.tls]
      MinVersion = "VersionTLS11"
      CipherSuites = ["TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_RSA_WITH_AES_128_GCM_SHA256", "TLS_RSA_WITH_AES_256_GCM_SHA384"]
      trustedIPs = ["<IP_ADDRESS>/22"]

[kubernetes]
namespaces = ["xxxxxx", "xxxxxx"]

[api]
  dashboard = true

[metrics]
  [metrics.statistics]
    RecentErrors = 50
  [metrics.prometheus]
    Buckets=[0.1, 0.3, 1.2, 5.0]

[acme]
email = "XXXXXXXX"
storage = "traefik/acme/account"
acmeLogging = true
entryPoint = "https"
onHostRule = false

[acme.dnsChallenge]
  provider = "route53"
        delayBeforeCheck = 15

[[acme.domains]]
  main = "XXXXXXXXXXXX"

[consul]
endpoint = "consul:8500"
watch = true

@esierrapena have you managed to fix this?

Ok, nevermind, this is expected. For anyone getting here from the search. If you want to filter based on forwarded headers (and, or proxy), you need proper annotation, for example:
ingress.kubernetes.io/whitelist-x-forwarded-for: "true"


Refactor chainComponents (bonus: reduce bundle size)

Minor refactor of chainComponents. results is now recursively accumulated rather than dependent on scope.
Reduces bundle size by roughly 18% going from 524B to 429B as compared via gzip-size dist/react-composer.min.js
See discussion leading up this here: https://github.com/jmeas/react-composer/pull/33#issuecomment-364484063

:memo: Beyond the refactor, further bundle size reduction was thanks to using less [not none] ES6/ES7 features that require Babel polyfilling. This reduced the bundle size by an additional ~14% with subjectively little cost to readability.

Inline unnecessary interim variables
Favor defaultProps over providing defaults in destructured arguments
Remove [next, ...remaining] destructuring in favor of components[0] and components.slice(1)

Bonus: Add a few comments for clarity

Thanks @erikthedeveloper ! I'm going to spend a bit more time comparing the minified files before I merge this, but the PR looks great. I'll have time to do this tonight.

@jmeas Considering that we were aiming to reduce total bundle size, that got me thinking so now I'm playing with it at lunch a bit 🤔 ... TL;DR by using less ES6/ES7 features we require less Babel polyfilling/etc... with very little cost to readability IMO.
I've got another very minor commit I'm working on pushing up that brings this down further to now shaving off ~18% from total bundle size from 524B to 429B.

That sounds amazing @erikthedeveloper :+1:

Thanks 😄  Ok, pushed. I'm calling this one good as far as I know. I've also updated the description to better reflect the changes. Let me know if you run into concerns.

This PR was 👌. Really, really nice work @erikthedeveloper :v:

Ya know, I've never cut a release just to decrease the file size. What do you think, should that be a minor semver bump? 4.1.0?

@jmeas I imagine this could be a patch. From https://semver.org/
Given a version number MAJOR.MINOR.PATCH, increment
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.

I guess neither really lines up perfectly. The old file size was definitely not a bug, and the reduced file size is not exactly new functionality.
I’m leaning toward a minor bump. 4.0.1 indicates to me that something is wrong with 4.0.0 (the documentation may not describe the behavior of the library, for instance), whereas 4.1.0 suggests to me that it’s an improved 4.0.0. A big file size drop lines up more with the latter sentiment in my mind.

Yeah I see what you are saying about patch == bug. Minor makes sense 👍🏼

4.1.0 has been released! Thanks @erikthedeveloper :)


Can't run test in Azure with 17.6 latest VS test platform

Description

Steps to reproduce

Azure Taskgroup is using latest stable VsTest installer

Run the test suite
It can't identify test scripts and can't run any automated test scripts  and I had to cancel run after waiting for 30 mins

Expected behavior

Actual behavior
Can't identify tests and run is stuck until cancelled

Diagnostic logs
Working run with older version

Environment

Hi @pshilpa1,
Could you please let us know if you moved from 17.5 to 17.6 or if you did a bigger jump? Also, would it be possible to know what is the target framework of your test projects?

We se tit up in Azure to use latest stable so , 17.5 to 17.6
Target framework is .NET Framework4.8 for all test projects.
Thanks

If you try to run vstest.console.exe 17.6 locally, are you reproducing the issue or is it only reproducing through the vstest task on AzDo?

Locally works fine , issue is only through Azure.

I'm having a similar issue, and the build fails with the following trace:
##[error]DiscoveryMessage : System.InvalidOperationException: The provided manager was not found in any slot.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ParallelOperationManager`3.ClearCompletedSlot(TManager completedManager)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ParallelOperationManager`3.RunNextWork(TManager completedManager)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.Parallel.ParallelProxyDiscoveryManager.HandlePartialDiscoveryComplete(IProxyDiscoveryManager proxyDiscoveryManager, Int64 totalTests, IEnumerable`1 lastChunk, Boolean isAborted)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.Parallel.ParallelDiscoveryEventsHandler.HandleDiscoveryComplete(DiscoveryCompleteEventArgs discoveryCompleteEventArgs, IEnumerable`1 lastChunk)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.InitializeDiscovery(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler, Boolean skipDefaultAdapters)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.DiscoverTests(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler)

The issue started when moving from 17.5 to 17.6 two days ago.

@pshilpa1 is that a distributed run?

I've tried multiple approaches but I can't repro. What is the configuration of your vstest task?

Ok can repro even locally now.

Workaround should be to make the dll selection narrower by changing the assembly filter to be specific to your test assembly names. Eg if all your tests are suffixed with UnitTests.dll or Tests.dll then you select just those from bin: https://github.com/microsoft/vstest/issues/3939

I can also confirm this is working in 17.5

Merged into 17.6 and main. Insertion into dotnet/sdk 7.0.3xx that is shipping 17.6 is pending here: https://github.com/dotnet/sdk/pull/32755.

#4516 workarounds summarized here.


Problemlist ExtractFilesTest.java and MultipleManifestTest.java in jdk8u

Problemlist tools/jar/ExtractFilesTest.java and tools/jar/MultipleManifestTest.java in jdk8u before the upstream test-fix PR has been integrated. The two tests fails with jtreg5.1.
Fixes: #5831

Thanks all for the quickly review and merged.


Adding PCA rest calls and local projections

Waiting for some fixes to go to production for the tests to run.


Add inline insertion points

When working on annotating method return type values in this PR in validation I've faced the limitation of the current insertion mechanism: it does not allow to add the code inside existing line.
The thing is that type annotation of a method return value must be placed before the short type name. Like this:
@java.lang.Override
    public io.spine.validation.test.money. @io.spine.validate.Validated AlwaysValidTime build() {

It cannot be done like this:
    @io.spine.validate.Validated  // WON'T WORK: the Java compiler won't accept this.
    @java.lang.Override
    public io.spine.validation.test.money.AlwaysValidTime build() {

Because of this we need a mechanism that would allow inserting (or even replacing) the code inside existing lines.

In the Chat discussion, @armiol suggested to have API for updating method signatures, instead of inline insertion points. It would be easier to understand.
It may be easier to understand, but it would make ProtoData not a language neutral tool. So far it looks like it does not know much about the languages it serves. @dmdashenkov, do you agree?

@dmdashenkov, fixed?

Yeap.


Merging variables recursively

At present, top level Project.variables override top-level Module.variables. This is reasonable and allows a module to define defaults that can be overridden by the Projects in which the module is being used.
Top-level Target.variables override top level Project.variables. This is also reasonable and allows less-specific values to be defined in a Project while the Targets within can override the values with more-specific ones.
It should be possible for variables to follow this pattern at a deeper level so that, for example, a Project can override a subset of a Module's hierarchy of defaults:-
variables: # Project
    connect_params:
        host: "myhost.example.com"

variables: # Module (defaults)
    connect_params:
        host: "localhost"
        port: "31337"

At present, the above example would result in the loss of the port variable; that is, the Project.variables need to define all of the ones defined by the Module in order to override a subset of them.

@boite indeed, experienced this happening. Is this a shortcoming of array_merge? if we'd like to get this to work as expected, do you know of any options? (same goes for host-level variables, and possibly in the future group-level variables)

I believe this would be achieved with the use of array_replace_recursive in place of array_merge.


UI: Loading screen for Grading View

Currently, there is a brief delay when clicking the "Show all groups" button in Grading View, between the button getting selected and the actual content appearing:

As the semester progresses and more submissions are recorded, this wait will increase, and it is not immediately obvious if the page failed to load the results, or is currently still loading them. There should be a loading screen to reflect this.

With pagination coming, the load time will be consistently short.
I disagree on the use of loading screens. We should actually figure out ways to load content faster, increase responsiveness and minimize features.

With pagination coming, the load time will be consistently short.
I disagree on the use of loading screens. We should actually figure out ways to load content faster, increase responsiveness and minimize unnecessary features.

Ok, I guess that is a way to solve the issue at hand

Duplication: similar to this cadet issue: https://github.com/source-academy/cadet/issues/257


Make feedback form accessible to external users

Issue
As an external user, when I am prompted to visit the link circled below, I'm then required to log in to Google... and I can't, because I'm not a MC employee.

This is the form I'm trying to get to -- https://docs.google.com/forms/d/e/1FAIpQLSclm2PAT9TJ9q23TT6YWQg6hj3EndUAdyF5k6kvNJHu6njNtw/viewform

Acceptance criteria
We need to update the document permissions and/or link in such a way that I'm not required to be a MC employee to access the form.

[ ] 1. As a user logging in via Google or Django username and Password, I can access the feedback form.

Housekeeping

[ ]  Unit test/s written wherever applicable
[ ]  i18n tags included wherever applicable
[ ]  Functionality is accessible to people using screen readers

From Slack thread https://mercycorps.slack.com/archives/GJ5KDBU1K/p1556148434030600

We should EPICify this into CMS-adjacent problems. Right now we direct users to:

a Google form for feedback
FAQ on the Hub
Documentation on the (old?) wiki
A few stray “Form Guidances” (see .../admin/workflow/formguidance/) which are mostly links to Google docs(?)
mailto: links for admins

The first 4 items are available only to (certain!) MC users. They’ll be useless for partners. And they are a nightmare to manage.
We could solve these problems in various ways:

Porting documentation and FAQs to a public space like Github wiki
Adding a feedback form
Flatpages or static pages for documentation
Building or bolting on a CMS


chore: Adopt src/ layout and tox

@mgxd if you could have a look, I'd appreciate it.

The tests for the docker image are still there on Circle, which is where we've had actual problems AFAIR. Building the wheels for PyPI is generally less problematic, since you clone the whole repo and there isn't anything like .dockerignore to get in the way.