Common type wrappers for gemoetry/crs formats

Dealing with formats for geometry and crs data has been annoying me for a while. We end up with a lot of importWKT/toEPSG methods that mean that calling method/package needs to know about all the formats instead of using generic approaches. We also need to somehow track what format a string is holding we are passing it between packages.
Using the type system can resolve this pretty neatly by wrapping a format once when it is loaded, avoiding any future validation or tracking. We can then leverage multiple dispatch to handle everything after that.
In GeoFormatTypes.jl I'm defining WellKnownText, ProjString, EPGScode and other wrappers that classify geometries and metadata, see https://github.com/yeesian/ArchGDAL.jl/pull/97 for an example use case.
Once formats are wrapped we can define pretty powerful generic tools in a few lines - like this reproject method that transforms coordinates between projections in any format such as WellKnownText-> EPSGcode(4326) without the calling method/package having to know about any of the formats.
Any thoughts? It would be good to move GeoFormatTypes.jl to JuliaGeo and have it adopted in the ecosystem.

A big +1 from my side for moving GeoFormatTypes to JuliaGeo. One candidate to use the interface would be Shapefiles.jl as discussed here. So your plan would be the the crs function from GeoInterface.jl would return a GeoFormat? Sounds good to me.

Thanks for starting the discussion. I agree it would be nice to improve our integration regarding coordinate reference systems. And I guess we would not want that to depend on Proj4.jl for the case we just want to carry it but not reproject. Although for converting between different GeoFormat types, I'm guessing we will need Proj4.jl in almost every case. For example, say we want to support writing the .prj files in Shapefile.jl by defining the EPSG code. We cannot do this without Proj4.jl, since we have to look up the EPSG code in the proj.db.
Why is this issue both about geometry and crs formats? Isn't geometry already addressed as part of GeoInterface? Different geometry formats of course sometimes have different crs formats, but I'd consider them separately.
@meggart regarding the crs function of GeoInterface, it is currently not part of https://github.com/yeesian/GeoInterfaceRFC.jl. I guess it could be added there, haven't thought about it much.
Some more quick comments. Perhaps we can generalize the EPSG to SRID, which holds both an authority, such as EPSG, as well as an identifier. I see GeoFormatTypes also has GeoJSON and KML. The latest specifications for both define these formats always to be EPSG:4326, so I think we can leave them out. I'm not so familiar with GML.
So SRID's are nice and should be used when possible, but of course they can only be used if the projection you use is defined in a database. So we still need generic formats like PROJ string and WKT. Using PROJ strings is now discouraged by PROJ itself (https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems). Although we probably need some support there to be able to handle Shapefiles for instance. For WKT it is important to track which version we mean, see https://proj.org/development/reference/cpp/cpp_general.html?#_CPPv44WKT2. Ideally I think we'd want to have a struct for WKT2, such that we can serialize it to different representations such as https://proj.org/usage/projjson.html.

Yes @visr there are two issues here: geometry formats and CRS formats. It would be nice if they were separate things entirely but unfortunately they aren't... That's one reason I changed the package name. Well known text and GML both contain geometries and CRS, which can be in the same string. We could use the type system to distinguish CRS only, CRS/Geometry, and Geometry only formats, but that's something that will need to be worked through with implementation problems.
The fact that KML is always 4326 could also be encoded in this package so no-one else has to specify that. So crs(::KML) = EPSGcode(4326) or something.
Also, geometric formats also suffer from the problem of method permutations. GDAL/ArchGDAL does conversion between geometries in different formats and each conversion has it's own named method each way. That means the calling code has to know all the format names and methods, which is what I'm trying to avoid. GeoInterface.jl defines the spatial types but not the formats they come in, so doesn't really help here.
@meggart I'll move GeoFormatTypes.jl here if someone wants to make me an owner of JuliaGeo. Your use case and questions about WKT in the other thread  were similar to mine. It seems that GDAL/ArchGDAL are the way to convert WKT to other projections currently? If we wrap the types the conversions can at least be easy. And eventually we can define an all julia type-system driven format for internal use and some day julia-based parsers.

@visr Those are good points about well known text.
I was thinking we could define AbstractWellKnownText and have various specific versions that inherit from it. WellKnownText2 <: AbstractWellKnownText,ESRIWellKnownText <: AbstractWellKnownText, etc (or whatever they are).
Can you also explain a little more what you mean by having both an authority and an identifier for SRID? How would we use method dispatch on this general type?
I would also like more details on why we don't need wrappers for string geometry formats? They can definately simplify some of the methods in ArchGDAL.

We debated SRIDs and CRS a lot at my previous job.

Can you also explain a little more what you mean by having both an authority and an identifier for SRID?

You may find the following documentation somewhat helpful for terminology: https://github.com/JuliaGeo/Geodesy.jl#coordinate-reference-systems-and-spatial-reference-identifiers (disclaimer: I wrote this, but I am not a Geodecist)

I mean how do we implement that generalisation in the context of dispatch and a type heirarchy. Having EPSGcode(1234) lets us easily use dispatch to send to methods that accept epsg and look them up in the epsg database.
I was imagining other authorities that matter would get their own types, so you could do reproject(x, EPSGcode(4326), SomeAuthority(5678)) and would dispatch to methods that can do the conversion.

Ah I misunderstood. Might it make sense to have SRID{:Authority}(code) as the generalization of EPSGcode? It seems likely that the typical julia function would manipulate SRIDs from only one authority at a time. The code is not really independent from the authority (ultimately it's an index into the authority's database), so I'd put them together into a composite type by preference.

geometry formats and CRS formats. It would be nice if they were separate things entirely but unfortunately they aren't...

They are different concepts and I believe we should treat them seperately in our APIs. I see a CRS as a property of any geospatial data (vector/raster/mesh/...) that may be present, at different levels. Since you often want to store the CRS with your data, they are encoded with that data in different forms. But instead of having a CRS type for each format, I'd rather just have the basic CRS types such as WKT, SRID, PROJ string. So Shapefile.jl would as you suggest return a PROJ string type for GeoInterface.crs (or define an interface on it, if we can). And when writing Shapefile.jl has the logic of how to encode a PROJ string to make a valid Shapefile (simply writing it to a .prj in this case).

The fact that KML is always 4326 could also be encoded

Agree, good suggestion. Though to me this would be most natural in a potential KML package rather than in this package.

I was thinking we could define AbstractWellKnownText and have various specific versions that inherit from it.

Something like that could work. Haven't looked much into the actual WKT format or the differences between versions, so it's hard to say much at this point.

I would also like more details on why we don't need wrappers for string geometry formats? They can definately simplify some of the methods in ArchGDAL.

I can see how it would simplify this in the case of GDAL and its API that just has strings going in and out, so you currently need to select the right method for the format. I wouldn't say we don't need it, or that we cannot improve the interoperability here. So if I understand correctly, this concerns specifically geometries serialized to strings in different formats, and not the parsed versions, which already would be a ArchGDAL.Polygon or Shapefile.Polygon, both implementing the GeoInterface? I do think I see value about having interoperable wrapped strings around the ecosystem, but I'm not yet sure about the benefit to confusion ratio of having this as a second wing of interoperability next to the GeoInterface. Say if I have a GeoJSON string, I'd want to parse it with GeoJSON.jl or ArchGDAL.jl for it to become useful to me. Or if I just want to pass it along, it can stay a string. What would wrapping it in a GeoJSON string wrapper mean? I already knew it was a GeoJSON string, so GeoJSON.read works and actually ArchGDAL.read is smart enough to figure out it is GeoJSON as well (if not, we can specify the format). I guess I need a concrete use case :)

Might it make sense to have SRID{:Authority}(code) as the generalization of EPSGcode?

Yeah indeed, I was thinking exactly that. We could always add a constructor EPSG(::Integer) such that EPSG(4326) -> SRID{:EPSG}(4326).

So if I understand correctly, this concerns specifically geometries serialized to strings in different formats, and not the parsed versions,

Yeah mostly everything here concerns handling strings or numbers representing some kind of geospatial data that is otherwise unidentifiable or requires parsing to do so, and for whatever reason isn't parsed immediately. With one caveat that if something is parsed then put in a Dict it would also need a wrapper... because we can't reliably dispatch on Dict or know what it contains, as was discussed in the other thread by @meggart.

Or if I just want to pass it along, it can stay a string

This is really the problem. Some intermediate package, or just higher-level methods in the receiving package, will still need to know that it's a GeoJSON string and send it to a specific method if it wants to convert it. And what if it was KML or WellKnownText? we will need lists of custom methods like in ArchGDAL and that requirement will propagate through the ecosystem.
If the strings are wrapped the method permutations are only necessary at the point of conversion -  say in Base.convert(Geometry, x::GML) or some other general method we define. Outside that, things can be general.

At some point, I'll like to have an interface for projections and reference systems in GeoInterface, and I'm not entirely convinced it needs to be outside of GeoInterface.jl just based on https://github.com/yeesian/ArchGDAL.jl/issues/95#issuecomment-552257103. Nonetheless,

I've been slow to participate of late since I'm still ramping up in my full-time job, so things on GeoInterfaceRFC has been slower moving,
There is a lot of activity in Geodesy.jl that's worth considering (c.f., https://github.com/JuliaGeo/meta/issues/7#issuecomment-553731558), and
The proposal in https://github.com/yeesian/GeoInterfaceRFC.jl is for GeoInterface.jl to move from a type-based dispatch mechanism to a trait-based dispatch mechanism. Given that GeoFormatTypes.jl is a type-based system, some thought needs to be had about how that should work out.

So I'm okay with GeoInterface being slower in the pace of development (w.r.t. GeoFormatTypes), and for GeoFormatTypes.jl to move to JuliaGeo and for packages to start adopting it (given the endorsement in https://github.com/JuliaGeo/meta/issues/7#issuecomment-553420831).

Say if I have a GeoJSON string, I'd want to parse it with GeoJSON.jl or ArchGDAL.jl for it to become useful to me. Or if I just want to pass it along, it can stay a string. What would wrapping it in a GeoJSON string wrapper mean? I already knew it was a GeoJSON string, so GeoJSON.read works and actually ArchGDAL.read is smart enough to figure out it is GeoJSON as well (if not, we can specify the format). I guess I need a concrete use case :)

Sorry I've been slow to the discussion. I also agree that I prefer to operate from understanding concrete use-cases. From what I understood in https://github.com/yeesian/ArchGDAL.jl/issues/95, it allows us to have functions like
transform(sourceproj::WellKnownText, targetproj::EPSGcode, val::Geometry)

rather than having the type in the function names like what ArchGDAL currently does:
julia> ArchGDAL.from
fromGML  fromJSON  fromWKB   fromWKT

julia> ArchGDAL.import
importEPSG   importEPSGA!  importPROJ4   importURL!    importXML
importEPSG!  importESRI    importPROJ4!  importWKT     importXML!
importEPSGA  importESRI!   importURL     importWKT!

julia> ArchGDAL.to
toColorTable toISOWKT      toMICoordSys  toWKB
toGML        toJSON        toPROJ4       toWKT
toISOWKB     toKML         toRGBA        toXML

@yeesian, to reply to some of your points:

There is a lot of activity in Geodesy.jl that's worth considering (c.f., #7 (comment)), and

As far as I can tell the datums etc in Geodesy.jl are mostly orthogonal to this package?

The proposal in https://github.com/yeesian/GeoInterfaceRFC.jl is for GeoInterface.jl to move from a type-based dispatch mechanism to a trait-based dispatch mechanism. Given that GeoFormatTypes.jl is a type-based system, some thought needs to be had about how that should work out.

These wrappers can't be trait-based as the objects are mostly just String - we have nothing to dispatch traits on.
And yes being able to write that transform function and other things like it is the reason for this package.
GeoFormatTypes is pretty much finished for now:
https://github.com/rafaqz/GeoFormatTypes.jl
I'm keen to register it as it's one of the last things blocking GeoData.jl from having coherent load/save between NetCDF/tiff/grd/hdf5 files, which will be great.
I would like it to be part of JuliaGeo, but I'll need to work on it further so I need to be an owner here to want to transfer it here. My pull request at DimensionalArrayTraits is sitting dormant because I can't merge it even though @meggart reviewed it, so I'm hesitant to base my infrastructure on packages here.

I have made you an owner of JuliaGeo, sorry I took so long to get around to it -- I like how you continue to push through friction where you encounter it, and escalate it if you feel things are not moving for you.
Friction does exist in JuliaGeo, not always without reason, and should not be taken as a passive-aggressive display of people's intentions. (Volunteers work on different schedules, and with varying levels of commitment, etc. There have been multi-year initiatives and discussions that did not pan out, but they often provide learning points for the next round of packages and initiatives.)

Thanks! And no problem at all. I have a bunch of other projects I barely respond to at all due to priorities, and people here are comparatively pretty responsive to ideas. These current changes are just central to my paid work and other research moving forward right now, so there is a tension between wanting to work collectively and integrate (which is always better in the long run), and getting things done now. Apologies if that ever appears pushy or rushed.

there is a tension between wanting to work collectively and integrate (which is always better in the long run), and getting things done now

Well said! Collaborating with mixed priorities is always difficult.
I skimmed through GeoFormatTypes quickly, it looks useful. A few thoughts which may or may not be helpful:

The CRS is a combination of coordinate system (could be a map projection or something else) and the datum. (Talking about datum is as confusing as it is hard to escape!) Using the term projection to refer to both of these — as has historically been done by proj.4 — can be very confusing IMO. coordinate_reference_system or some shortening of that would be more transparent.
The CRS for some geoformats may not be embedded within them. Eg, you can have WKT without the CRS. At least this was valid last I knew about it. So projection is not well defined for WKT without a full WKT parser, I would think.
If an add-on package was to provide a parser for GML it could implement projection(::GML) but this would override the definition at https://github.com/JuliaGeo/GeoFormatTypes.jl/blob/285866a5a4d75df7df3d516bf49d51b59a5c07e2/src/GeoFormatTypes.jl#L157.  May be good to just have a generic fallback for one of the abstract geo format types which returns some kind of "no parser exists for this GeoFromat" so that individual parser packages can define the parsers on demand.

As far as I can tell the datums etc in Geodesy.jl are mostly orthogonal to this package?

Unfortunately, not really! Datum is inescapable (at least for high accuracy ~decimeter level measurements). When you measure a point using a GNSS receiver, what are you really measuring with respect to? It's not the satellite constellation! The satellites are just a mechanism for transferring the reference frame defined by an ensemble of ground stations to the handset position. That is, the satellite positions themselves are defined relative to this ensemble of ground stations. If you choose different ground stations, you have a different datum. WGS84 is the datum defined by a set of ground stations maintained by the US department of defense (eg, see here: https://www.ga.gov.au/scientific-topics/positioning-navigation/wgs84 ). The ITRF datums (and regional refinements like ETRS) are defined by a different set of ground stations.
Of course, these are just some thoughts from a quick read and may not be applicable to you right now. So feel free to do with them what you will.

By orthogonal I mean the datum is a property of the data, and this package simply labels the format of the data and knows very little about what is contained in it. It's just explicitly categorising format standards to make it easy to write combinatorial method dispatch, it doesn't actually do anything :)
I think the main conflict is that I included the method projection and probably shouldn't have!!

The CRS for some geoformats may not be embedded within them. Eg, you can have WKT without the CRS. At least this was valid last I knew about it. So projection is not well defined for WKT without a full WKT parser, I would think.

Absolutely. Same goes for GML as you say - that's how I imagined it would work. This kind of organised type pyracy seems not that uncommon in Base type packages, especially for Julia Base methods dispatching on shared types.
But I think having the projection method might be overstepping the purpose of this package, it just seems like a very cheap bonus to having things explicitly wrapped, but maybe should be deleted. I was also throwing up wether to use projection or crs and swapped crs for projection because crs breaks the "avoid vague acronyms" rule. I actually didn't know what it meant the first time I read it in a package, but projection is obvious, though as you say somewhat erroneous.
I kind of assumed someone would pull me up on it too, so projection can totally go back to crs, or just get deleted, especially seeing it's already complicated discussion of what is otherwise a dead simple package.

This kind of organised type pyracy seems not that uncommon in Base type packages, especially for Julia Base methods dispatching on shared types.

Yeah, I don't mean to suggest that this form of type piracy is bad. It's a very reasonable way of splitting code into a lightweight core where the detail can be implemented elsewhere. All I mean is that implementing some foo(::GML) for concrete type GML should only be done in one package. If you want a fallback which says "not implemented", that's good to implement on an abstract type. That way, when loading a package which does implement foo(::GML), it adds to the method table rather than deleting and replacing an existing method.

Edit: the acronym thing is a larger question here too. I used KML because I don't think people know or care that the K stands for Keyhole, and they know what kml is.

Agreed. The larger goal of the "avoid acronyms" thing is to improve clarity and reduce unnecessary jargon so it's definitely subjective. But people will only learn that the company Keyhole existed via kml, not the other way around. At that stage, kml has become the proper name for this thing.
(Side note, I've always thought "well known text" was a really terrible name for a format. The expansion says virtually nothing about the format or the domain in which it's used.)


Discussion for hull algorithms

While studying and examining the relevant hull algorithms, apart from the bugs, it seems that the results of the different algorithms greatly vary, especially given some 'edge cases': co-linear points, duplicate points, zero area 'polygons' etc.
I was thinking that it would be nice to have some defined consistent description of what should happen in all those cases and then (try to) re-implement the algorithms to adhere to those rules.
Then 1) the user can easily change algorithms (e.g. for performance) without sudden strange results and also 2) we can have more meaningful and complete unit tests + use each algorithm to validate the others.
From my current tests it seems that the MonotoneChain algorithm is a good quality benchmark: de-duplicates points, graciously removes co-linear points, handles zero area polygons in a meaningful way. Maybe it's behavior could be the basis of the expected result? (correct me if I'm wrong - if there are cases where the above do not hold for this algorithm).
I have not examined how feasible it is to make all algorithms have the same properties.
I was working on some fix for the hull bugs but because most of them have to do with those edge cases it felt like there should be some expected behavior to follow before making another implementation.

With the work on GiftWrap and DivideAndConquer I'm making progress towards that, but also finding that each of these has their own drawbacks wrt. robustness.  As I fix the issues I'm seeing them get closer to a unified operation.  There still might be differences though.  One example in my mind is the order of the vertices of the hull.  For example, for a hull with 20 points, GiftWrap may choose point 4 to be index 0 in the resulting hull and DivideAndConquer might choose point 10.
Let's revisit once the bugs are cleaned up.

Alright, fair enough. I'm leaving the bugs to you then since you made good progress.
Point order should not be standard, yeah, since this will probably impose a non-trivial performance penalty from some algorithms. At the end we could just document the expected behavior on a per-algorithm basis.


Upgrade installer to ship with jdk 17.0.4+8

Impact

[ ] Bug fix (non-breaking change which fixes expected existing functionality)
[X] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description
Upgrades the jdk installed with the windows/mac installers to 17.0.4 vs. 17.0.3
Things to be aware of

Only impacts the installers
Builds always use "latest" 17.x so already upgraded

Things to worry about

Nothing

Additional Context
No security issues with 17.0.3, just upgrades and bugfixes in 17.0.4

Looks like the jdk is Java v17.0.4
Java Home C:\Users\*myUser*\liquibase\jre (Version 17.0.4)


Bump peer dependency range for Relay v0.10.x

Currently the peer dep range requires 0.9.x.
I've actually moved all of my packages to use >= ranges instead of ^ ranges for Relay, but I'm not sure that's safe with Relay 2 coming soon... but then again it's better than having to remember to re-publish every time Relay bumps its version.

1.0.1 is published now. let me know if you find any issues! Thanks for reporting

Thanks!


Individual transform properties from MDN

I'd love to add detection for individual transform properties to doiuse, however these features are absent from caniuse.
Do y'all have any qualifications for which features get imported from MDN? I'm willing to write the PR to add these if it helps.

There is no solution right now. We contain only Can I Use data. MDN data can be packed to this package only as a big exception, and we need a really good reasons for that.
The best way is to use MDN data in doiuse.


Export defaults from PooledByteBufAllocator static fields - tuning pa…

…rameters for org.apache.spark.network.util.NettyUtils Now, Spark must modify access to private fields.

@slachiewicz can you also squash the commits into one and sign the ICLA:
http://netty.io/s/icla
And please update the commit message to follow our template:
http://netty.io/wiki/writing-a-commit-message.html

@slachiewicz ping ;)

Done, with lots of pain with git ;-)

Cherry-picked into 4.0 (ad008f6831b1d233d9669a99fcec74ccf44a9c72), 4.1(a18416df60809d5ec9c703bfbc15658a8d4443fc) and master (a6816bd59ef2923d82af5b7a7f3b722507a56a3a)

@slachiewicz thanks!


2.4.7-amzn-1' is considered invalid by version checker

  When running splink on EMR cluster, the spark version value of '2.4.7-amzn-1' is considered invalid by version checker.  Any work around to this on the splink side?  If looks like the "-amzn-1" portion is what's causing failure, and I can't change what spark returns for version.

An error was encountered:
Invalid version: '2.4.7-amzn-1'
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/splink/spark/spark_linker.py", line 154, in init
if version.parse(self.spark.version) >= threshold and not self.catalog:
File "/usr/local/lib/python3.7/site-packages/packaging/version.py", line 52, in parse
return Version(version)
File "/usr/local/lib/python3.7/site-packages/packaging/version.py", line 197, in init
raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '2.4.7-amzn-1'
Originally posted by @reygvasquez in https://github.com/moj-analytical-services/splink/discussions/954

A few notes:
The packaging module does not appear to be a builtin.  I get ModuleNotFoundError: No module named 'packaging' with a ModuleNotFoundError: No module named 'packaging'from from packaging import version
I don't think we have included it as a requirement or version controlled it (since i assumed it was a builtin).
I'm tempted to work with string comparisons (e.g. "2.4.7-amzn-1" < "3.4.0"), since this test should work in any plausible custom spark version

Could us this as a backup https://stackoverflow.com/a/28568003/1779128


Add disableEditing, enableEditing

Per the 0.4.0 epic, we would like an API for disabling the ability for a user to enter and edit existing text. This is useful in conjunction with the programmatic API- It allows us to build editing interfaces that output mobiledoc and cards, but without needing to allow freeform content.
The APIs are:

editor.disableEditing() - stop the user from being able to edit the
current post with their cursor. Programmatic edits are still allowed.
editor.enableEditing() - allow the user to make direct edits directly
to a post's text.

@bantic could use a review here!

@mixonic :+1: I had a small comment on one of the tests. It would be nice if there were a single bottleneck that event listeners ran through — that way there'd only be a single point where you would ignore events — like a handle(Xevent), but that's a separate concern. This looks good

Phantom does not respect toggling contenteditable off :-/ Left the skip in with a comment. Confirmed to pass locally.


Add support for live documentation editing

This patch adds the docs-live make target. This make target uses
sphinx-autobuild to rebuild and reload docuemtnation content in the
browser.
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running

The plugins, roles and utils are not getting build for live version.
/home/jpodivin/repos/ci-framework/docs/source/index.rst:95: WARNING: toctree glob pattern 'plugins/*' didn't match any documents
/home/jpodivin/repos/ci-framework/docs/source/index.rst:102: WARNING: toctree glob pattern 'module_utils/*' didn't match any documents
/home/jpodivin/repos/ci-framework/docs/source/index.rst:109: WARNING: toctree glob pattern 'roles/*' didn't match any documents

You can work around it by building the standard version first, and using the `sphinx-autobuild` just to serve it. But I'm not sure if the changes are going to propagate that way.

Thanks @jpodivin , I have added a new docs make target to build the auto generated content. I then pass that to sphinx-autobuild as a --pre-build` command. I also added the source plugins directory as one to watch for changes and excluded the generated content from the watch as it causes a loop.
Please test and let me know if that helps the issues you discovered!

The plugins, roles and utils are not getting build for live version.
/home/jpodivin/repos/ci-framework/docs/source/index.rst:95: WARNING: toctree glob pattern 'plugins/*' didn't match any documents
/home/jpodivin/repos/ci-framework/docs/source/index.rst:102: WARNING: toctree glob pattern 'module_utils/*' didn't match any documents
/home/jpodivin/repos/ci-framework/docs/source/index.rst:109: WARNING: toctree glob pattern 'roles/*' didn't match any documents

You can work around it by building the standard version first, and using the `sphinx-autobuild` just to serve it. But I'm not sure if the changes are going to propagate that way.

Thanks @jpodivin , I have added a new docs make target to build the auto generated content. I then pass that to sphinx-autobuild as a --pre-build command. I also added the source plugins directory as one to watch for changes and excluded the generated content from the watch as it causes a loop.
Please test and let me know if that helps the issues you discovered!

Thanks. I've verified that the documentation includes all parts of the CIFMW and that it's possible to edit it live. Great work.


Example of including ipython notebooks

This PR is an example of using nbviewer to include ipython notebooks.
Requirements:

Install nbviewer
Install pandoc (I used apt-get install pandoc)
All notebooks must include a title using a markdown cell with H1 title
There may be a way to achieve this without moving the notebooks


Disabling the folder loader bar?

Is your feature request related to a problem?
Files just looks great, especially compared to the aging default explorer. I just have one major issue, the loading bar at the top makes me feel like the app is sluggish. I have only SSD so everything is quick, but there is that bar that shows for like one tenth of a second and makes you feel, in my opinion, like you're waiting much longer than you are actually waiting.
Describe the solution
Add an setting to allow the users to disable the loader bar and have the same behavior as Windows, just don't show anything while it is loading.
Scope

Capability
Priority

Option to disable the loading bar
Must

Disabling the loading bar isn't going to make the app any faster, it's used to indicate that the app is loading your Files. We have made several improvements in main in regards to loading items faster and you'll see the improvements of this in the next update.

Sorry but i strongly disagree and it's a general UX matter that i faced many times in web developement. My complain is not that the load is too slow, it's that the loader makes you feel like it is. Why show it for less than a second? It just makes the small load time noticeable.

@Yolo-plop You have a good point about the bright flash making it seem slow to the users, that's why we plan to make the loading indicator less prominent with #3922.

Perhaps only display the progress bar after a certain amount of milliseconds?

@mdtauk That's something to look at, can you add your feedbac to #3922?


[Frontend] Fix a small unique_ptr array access.

When the size of the array accessed here is zero, retrieving the address of the zero-th element here is undefined. When the frontend is linked against a libc++ that has the unique_ptr hardening in this commit enabled, it traps here.
Instead, simply call .get() to retrieve the address of the array, which works even when it is a zero-byte allocation.

@swift-ci please smoke test


chore: classify was rw porting notes

Classifies by adding issue number (#10691) to porting notes claiming was rw.

maintainer merge

Whilst I would prefer issues which covered hundreds of porting notes rather than just 8, several of the examples here are where rw has become erw and in my mind this indicates that something is wrong, so my feeling is that this is a good category even if it's a small one.

bors merge


HTML

HTML
User Story
As a bootcamp student
I want the prework notes to be structured on a webpage
so that I can easily find and read information
Acceptance Criteria
GIVEN a Prework Study Guide website
WHEN I visit the website in my browser
THEN I see four boxes titled HTML, CSS, Git and JavaScript with associated notes listed

Pull Request: added code to the HTML file #7


Configuration (brightness value) not getting saved after sending command via HTTP

After connecting my AC Dimmer with wemos mini I am unable to save the configuration via HTTP GET command. I tried using this: 192.168.0.x/?b=0
This makes the output voltage to drop and then it rises again and when I open the page 192.168.0.x to check the configuration it shows b=255. Which means it is not getting saved. Any ideas?

Try setting buttonActiveOn = 1 at the top. Depending on how this is set and your pullup/pulldown button configuration, it may think you're holding down the button, which will quickly increase the brightness until it reaches 255 again.

I just removed the buttons part and configured it according to my needs and its working... thank you anyway for replying


not able to download python extension in visual studio on mac

Hi @pareekrahul
Visual Studio for Mac doesn't have Python language support, unfortunately. The extensibility model for Visual Studio for Mac is completely different from Visual Studio (for Windows).
For Mac users we recommend using Visual Studio Code, with the Python extension (which is developed by our team).


Add AppVeyor tests for Windows

@jzaefferer: this should not be merged yet. You will need to add Appveyor yourself, then ping me to update the patch to include your project instead of mine.

So did you figure out the Windows failure? Or is that still pending as well?

Nope, that is still pending, unfortunately. I will be able to try again later to solve the failure, but it's a good practice to test on Windows too since (unfortunately) Windows is different than *nix.

Fixed one, 3 left. Anyway, will you add an AppVeyor project yourself in order to use it here?

Until then https://ci.appveyor.com/project/XhmikosR/grunt-html/build/job/xg6gjfog2imememg

I don't get at all why those 3 tests have [] for actual. Shouldn't we get an error or something?

Maybe this line is the culprit? Does exec work differently on Windows (i.e. is stderr empty here)?

This is what I get for those 3
Testing html_test.js
>>>> stderr:  {"messages":[]}

F
>>>> stderr:  {"messages":[]}

F
>>>> stderr:  {"messages":[]}

F

Ah, we're not normalizing the paths in config.files before handing them off to chunkify. Try this:
var files = config.files.map(path.normalize);
async.mapSeries(chunkify(files, maxChars), function(chunk, cb) {
...

I think, spawn should be used instead.

But we aren't indeed normalizing the paths, so that is a valid change too. Unfortunately, it doesn't fix those 3 tests.

Damn, I thought the Unix path separators were screwing things up. If that's not it, I haven't a clue.

I still think that change is useful, and I'll apply it. But something else is causing those failures.

Added your patch too. https://ci.appveyor.com/project/XhmikosR/grunt-html/build/13

{"messages":[]} seems to be returned only if there were no errors or the path given does not exist. Must be the latter, since we know invalid.html contains errors. Maybe using path.resolve rather than path.normalize would do the trick?

path.resolve doesn't work at all in lib/htmllint.js.

This works fine for me:
var files = config.files.map(function(file) {
  return path.resolve(file);
});
async.mapSeries(chunkify(files, maxChars), function(chunk, cb) {
...

Without any changes in tests?  I'll try that later or tomorrow and let you
know.
On Mar 11, 2015 9:41 PM, "Josh Hagins"<EMAIL_ADDRESS>wrote:

This works fine for me:
var files = config.files.map(function(file) {
return path.resolve(file);
});
async.mapSeries(chunkify(files, maxChars), function(chunk, cb) {
...
—
Reply to this email directly or view it on GitHub
https://github.com/jzaefferer/grunt-html/pull/60#issuecomment-78354213.

Figured it out, the problem is in chunkify. Batch apparently treats single quotes as any other character, so the path passed to vnu.jar is actually 'test/invalid.html', not test/invalid.html. We just need to use double quotes in chunkify rather than single.

@jawshooah: cherry picked your patch and it indeed fixes the other failing tests :+1:
@jzaefferer: so only you adding AppVeyor to the repo is left.

Alright, added: https://ci.appveyor.com/project/jzaefferer/grunt-html

Cool, I'll rebase when I'm home and ping you.
On Mar 13, 2015 12:11 PM, "Jörn Zaefferer"<EMAIL_ADDRESS>wrote:

Alright, added: https://ci.appveyor.com/project/jzaefferer/grunt-html
—
Reply to this email directly or view it on GitHub
https://github.com/jzaefferer/grunt-html/pull/60#issuecomment-78899489.

@jzaefferer: https://ci.appveyor.com/project/jzaefferer/grunt-html/build/1
So, everything looks good, you can merge this PR.
PS. Enable in AppVeyor settings, to skip branches without appveyor.yml.

Thanks! Squashed into a single commit (sorry @jawshooah for dropping you as author there), updated commit message.

So wrong, in so many aspects. Why on earth do you change the patch author? Just so you have a patch in "one commit"? Well, guess what, it's not meant to be one commit, otherwise it would have been.
There's a nice merge PR button you should start using, or update the contribution guidelines, that you will do whatever you want with one's patch.

Whoa. It's his repo, dude. And it's pretty common for people to squash the changes in a PR to keep the history succinct.


CNDB-11563 move and rename SAI iterators and postings as in ASF

What is the issue
It is not natural to have SAI iterator classes in utils package, furthermore, Apache Cassandra has them in a separate package iterators. Also ASF C* has consistent naming for iterator classes. Similar to SAI listing classes.
What does this PR fix and why was it fixed
Fixes https://github.com/riptano/cndb/issues/11563
This PR:

Moves SAI iterator classes from utils package to new package iterators
Renames SAI iterator classes to be consistent and equivalent to ASF C* for the same classes.
Renames AbstractIterator into AbstractGuavaIterator, moves it into the general utils package, and replaces the identical copy of it in disk package. This is in accordance with ASF C*.
Renames and moves SAI postigns classes from utils package to new package postings, which is in accordance with ASF C*.
Removed unused imports in the affected classes as done for free by Interllij during the above refactoring.
Fixed few typos in the affected classes.

This will help with rebasing CC on ASF C* 5.0.
Checklist before you submit for review

[ ] Make sure there is a PR in the CNDB project updating the Converged Cassandra version
[x] Use NoSpamLogger for log lines that may appear frequently in the logs
[ ] Verify test results on Butler
[ ] Test coverage for new/modified code is > 80%
[x] Proper code formatting
[ ] Proper title for each commit staring with the project-issue number, like CNDB-1234
[ ] Each commit has a meaningful description
[ ] Each commit is not very long and contains related changes
[ ] Renames, moves and reformatting are in distinct commits

The corresponding build in CNDB: https://github.com/riptano/cndb/pull/11580


declare global ignored

Hi! Trying to make ts library with d.ts bundle. The problem that my global typing ignoring by plugin.
Have tried to import global.ts explicitly export * from './types/global' by index.ts entry file but still – just ignored this block:
declare global {
    namespace ... {
    ...
    }
}

config that I used for plugin:
config.plugins.push(new BundleDeclarationsWebpackPlugin({
    entry: ['src/index.ts', 'src/types/global.ts'],
    outFile<EMAIL_ADDRESS>    compilationOptions: {
        preferredConfigPath: './tsconfig.json',
    },
    removeEmptyLines: false,
    removeEmptyExports: false,
    removeRelativeReExport: false,
}));

Any ideas here?

From the dts-bundle-generator config docs:
           output: {
                /**
                 * Enables inlining of `declare global` statements contained in files which should be inlined (all local files and packages from inlined libraries).
                 * Optional. Default value is `false`.
                 */
                inlineDeclareGlobals: false,

^ might be worth giving this a try?


Add labels on charts' axis

General
Every chart that contains axis should display them just like the following example:
https://bl.ocks.org/d3noob/23e42c8f67210ac6c678db2cd07a747e
The axis labels should be passed through options variable.
Components affected:

line
bar
area
band
multiline

DoD

charts with X, Y axis can receive labels through options
labels displayed correctly

Blocked By

#96

Won't resolve the area chart in this issue.


Add IReadOnlyObservableList interface

Related to #45, IReadOnlyReactiveCollection<T> in UniRx has an indexer definition, but IObservableCollection<T> does not have an equivalent definition.
Therefore, I added IReadOnlyObservableList<T> as an interface to handle both IReadOnlyList<T> and IObservableCollection<T>.

ok, thanks!


[KOGITO-2567] Restore root classloader in DMNKogito

https://issues.redhat.com/browse/KOGITO-2567

@tarilabs I am not sure what you mean. The CL was set to null already, this is just removing that line. Where's the difference? Anyhow, I don't necessarily want to merge this

If you remove the line, as this PR is proposing, the DMNRuntime internal WILL try to use the Classloader.
My understanding in native scenario, we cannot use it to load class dynamically.
So my perspective is that this PR should not be merged, as otherwise would just be rugging a problem under the carpet; we should leave master code as-is, setting the classloader as null, so we make SURE there is NO use of the classloader by the dmnruntime internals.

oh ok I was missing that detail. @danielezonca is there any particular reason why you wanted to restore this?

I synced with @danielezonca on this immediately after I commented on the PR.
Daniele was looking to have the CL which is also used by the DMNRuntime internals for service load, but the solution, imho for Kogito, is not passing through that mechanism.

Let me add a bit of background: we tried to test jPMML-DMN integration in Kogito (JVM mode) and it failed with a NPE, the exception came from this code in DMNCompilerImpl during DMN import compilation because the classloader is null. Using this patch we were able to run an example (here).
We do not aim to make jPMML native compile (actually I'm not sure it can because it performs a lot of XML work at runtime) so we tested only JVM mode and for now this is the only PMML impl we can use because the old PMML impl uses 7.x RuleUnit while the new one is not ready yet.
I expect we will have to change some code in kie-dmn to be sure the new PMML impl can native compile and work together with DMN but we will look at that later (probably the line of code that throws the NPE).
The build is green so this change is not breaking native compilation of DMN that's why I proposed to merge it but I'm open to discuss.
Side note: I cannot create a DMN-jPMML integration test here because of license limitation so I would probably need a completely new repo only for that because kie-jpmml-integration repo is in 7.x stream so we cannot create kie -> kogito dependency.

The build is green so this change is not breaking native compilation of DMN that's why I proposed to merge it but I'm open to discuss.

well the CI build currently does not test native builds; I am not sure I follow you here :)

@evacchi
Ok fairpoint, I was not aware that CI is not running native tests. Ok I run them manually and it is still working :)

please Close this, as it's superseeded by the enseble:
https://github.com/kiegroup/drools/pull/2985
https://github.com/kiegroup/kogito-runtimes/pull/622


I have a question about LiferayJspDebuggerPositionManager class.

@dmarks2 I have a question about LiferayJspDebuggerPositionManager line 56. Do we need to override the method locationsOfLine in LiferayJspDebuggerPositionManager class because we have a compatible issue wtih Intellij EAP versioni about the ReferenceType.  Can we discuss this when you have free time? We can discuss in liferay slack community.
Package not found (1 problem)
Package 'com.sun.jdi' is not found (1 problem)
Package 'com.sun.jdi' is not found along with its 4 classes.
Probably the package 'com.sun.jdi' belongs to a library or dependency that is not resolved by the checker.
It is also possible, however, that this package was actually removed from a dependency causing the detected problems. Access to unresolved classes at runtime may lead to NoSuchClassError.
The following classes of 'com.sun.jdi' are not resolved:
Class ReferenceType is referenced in
LiferayJspDebuggerPositionManager._locationsOfClassAt(...)
LiferayJspDebuggerPositionManager.access$000(...)
LiferayJspDebuggerPositionManager.access$400(...)
LiferayJspDebuggerPositionManager$1.(...)
LiferayJspDebuggerPositionManager$1._getSourceName(...)
.

closing the issue as we already discussed about it


all: Only load Opt in the system class loader

We can detect whether we are running on Android via system properties and thus don't need to have this bit passed in via a flag.
As a result, the runtime no longer depends on Opt, which is thus no longer loaded in the bootstrap class loader in addition to the system class loader.

CC @marktefftech

Overall this change LGTM-- @TheCoryBarker and myself will need to update our open PR's (PR#737, PR#743) with similar changes. Cory, please confirm.
FWIW I think it makes sense to submit this change before the above PR's so we can rebase accordingly.

LGTM


Confusing error message on type argument mismatch

I'm getting this error from the analyzer:
[ARGUMENT_TYPE_NOT_ASSIGNABLE] The argument type 'List<MapEntry> (.../sdk/lib/core/list.dart)' can't be assigned to the parameter type 'List<MapEntry> (.../sdk/lib/core/list.dart)'.

Note that I'm formatting the machine readable format differently from how the analyzer normally prints messages.
The problem is the MapEntry argument. One version is from package:kernel/ast.dart the other is from dart:core.

Duplicate of #29058, which has more examples.


Test Cilium on AWS - Test Release KubeOne 1.7

We upgraded Cilium as part of https://github.com/kubermatic/kubeone/pull/2860
### Provisioning
- [x] /test pull-kubeone-e2e-aws-default-cilium-containerd-external-v1.27.3
- [x] /test pull-kubeone-e2e-aws-flatcar-cilium-containerd-external-v1.27.3
- [x] /test pull-kubeone-e2e-aws-rhel-cilium-containerd-external-v1.27.3
- [x] /test pull-kubeone-e2e-aws-rockylinux-cilium-containerd-external-v1.27.3

### Upgrading
- [ ] /test pull-kubeone-e2e-aws-default-upgrade-cilium-containerd-external-from-v1.24.15-to-v1.25.11
- [ ] /test pull-kubeone-e2e-aws-flatcar-upgrade-cilium-containerd-external-from-v1.24.15-to-v1.25.11
- [ ] /test pull-kubeone-e2e-aws-rockylinux-upgrade-cilium-containerd-external-from-v1.24.15-to-v1.25.11
- [ ] /test pull-kubeone-e2e-aws-rhel-upgrade-cilium-containerd-external-from-v1.24.15-to-v1.25.11

failing conformance tests
[
 {
 "name": "[It] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]",
 "status": "failed",
 "details": {}
 },

 {
 "name": "[It] [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]",
 "status": "failed",
 "details": {}
 }
 ]

This looks like a known issue: https://github.com/cilium/cilium/issues/14287

CentOS and Amazon Linux 2 tests have been removed from the list because the kernel version doesn't support Cilium.

Testing is done as part of #2894


adding mDriver library

@ArduinoBot

@ArduinoBot

@ArduinoBot

Hello @ahmedosama07. The automated system here only operates on open pull requests. Since your submission was already accepted and the pull request merged, the bot won't pay any more attention to this pull request.
But this human is happy to provide assistance. Was there something you needed from the bot?


Newsreader CLI

A REPL-like Newsreader CLI would make for a cute example

Hello found this library searching for nntp/usenet libraries, I currently use slrn, but I would like to create a client, are you going to keep developing on this library?


missing mathjax as a requirement, how do we know this module has it in the window namespace?

          missing mathjax as a requirement, how do we know this module has it in the window namespace?

Originally posted by @kpsherva in https://github.com/inveniosoftware/invenio-requests/pull/403#discussion_r1763237384

After checking, found out that in case MathJax is missing from the window, the mathematical content won't be rendered, but no error or warning will be thrown.


Implement Tagging for Recorders in Pulse

This pull request introduces a new feature to the Pulse package that allows developers to tag all types of records. This feature enhances the tracking capabilities of Pulse, providing more context and granularity when analysing records.
The implementation involves the addition of a tag method to the Pulse facade, which accepts a callback function. The callback function is expected to return a string that represents the tag. These tags are then included in the record data that Pulse records.
This feature does not break any existing features of the Pulse package. It is an optional enhancement that developers can choose to use if they find it beneficial.
The benefit to end users is the ability to categorise and filter records based on custom criteria. For example, a developer could tag records that occur within a specific module of their application, making it easier to identify patterns or recurring issues within that module.
Tests have been added to ensure the correct functioning of this feature. The tests cover the basic usage of the tag method, as well as more complex scenarios involving multiple tags and records.
This feature makes building web applications easier by providing developers with more control and flexibility when it comes to tracking. It allows developers to tailor the tracking capabilities of Pulse to better suit their specific needs and the needs of their application.
This is an initial implementation for tagging all types of records. If this approach is deemed beneficial, I plan to extend this functionality to include tags for the rest of the recorders in the Pulse package.
Thank you for considering this contribution. I look forward to your feedback.

Hello @timacdonald & @jessarcher,
I'm following up on this PR. Could you let me know if it's feasible? Additionally, do you think it would be a valuable feature to add? I'm eager to proceed with the rest of the PR, but I want to ensure we're moving in the right direction with a solid solution.

Hey @andre-hoffmann - going to hold off on this one for now. Thanks.

Hey @andre-hoffmann - going to hold off on this one for now. Thanks.

Hi @taylorotwell, May I ask the reason behind this decision? I know a lot of people request this feature. Its main purpose is for multi-tenancy to identify what tenant is behind the recording.


Randomly get connection reset

Describe what are you trying to do
I simply run KrakenD on k8s and connect to backend service and when running I'll get a random connection reset like
Logs
If applicable, any logs you saw in the console and debugging information
read tcp <IP_ADDRESS>:43510-><IP_ADDRESS>:9000: read: connection reset by peer

this might be related to my issue (https://github.com/devopsfaith/krakend-ce/issues/396). Eventually, on querying client side, I am also getting
('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
However, I do not know whether the root cause is the same, as this symptom can be caused by a lot of things.

I don't think this is related to KrakenD...
as commented here: https://github.com/golang/go/issues/20960#issuecomment-348405178

If you get broken network connections, they might be real (a peer really did disconnect), or they might be from hitting kernel limits on one of the sides.

check your networking and kernel configuration (ulimits, etc) and maybe your k8s instances (free disk space, etc)


duration for api.get_prices - Is not working. Only 30mins interval of data is being returned

duration for api.get_prices - Is not working. Only 30mins interval of data is being returned. Will be good if i can get some sample code to achieve 5mins interval data - I tried - api.get_prices(site_id, duration=5)

It’s resolution, not duration


修复 HeartbeatProvider/HeartbeatConsumer 单元测试错误

修复 HeartbeatProvider/HeartbeatConsumer 单元测试错误

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: chenniufly:x: huaming.chen<EMAIL_ADDRESS>seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

怎么退订
在2017年10月11日 17:18，CLAassistant 写道：
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.
❌ chenniufly
<EMAIL_ADDRESS><EMAIL_ADDRESS>seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

提交PR前请Sign The License Aggrement
已参照修改：02bdd04cde1b336d34279cca7c0538e7094459fc


[question] cat inner puzzle

I am currently investigating the creation of cats and would like to write my own inner puzzle.
I have seen the cat_v2.clsp and noticed that it requires a INNER_PUZZLE to curry in. I cannot find a inner puzzle reference anywhere though. Neither in the documentation, the chia github repo nor in this repo.
I would like to ask how this is handled and if you have a reference for the default inner_puzzle as a starting point

some update, is it correct, the inner puzzle is hardcoded here?
 # Construct the intermediate puzzle
    p2_puzzle = Program.to(
        (1, [[51, 0, -113, curried_tail, solution], [51, address, amount, [address]]])
    )

    # Wrap the intermediate puzzle in a CAT wrapper
    cat_puzzle = construct_cat_puzzle(CAT_MOD, curried_tail.get_tree_hash(), p2_puzzle)
    cat_ph = cat_puzzle.get_tree_hash()

Yes that is the inner puzzle that this repo uses for minting.  It's not a very effective inner puzzle for other purposes though. See here for more https://docs.chia.net/guides/chialisp-inner-puzzles?_highlight=inner&_highlight=puzzle


Specifying types or changing a column's type

Hello,
Is it possible to specify the columns types when loading data (e.g. csv) or changing a column's type after creating a dataframe?
Thanks!

Can you try this -https://github.com/databricks/koalas/pull/604?

It seems to work, although users would need to specify all columns since it uses selectExpr. I guess this does the job for now, but an implementation of dtype={‘a’: np.float64, ‘b’: np.int32, ‘c’: ‘Int64’} and to_numeric or as_type would be great!

I don't know why sometimes, the result is full of None

Do you mind if I ask your codes to reproduce and investigate further?

Btw, I'm working on adding dtype argument as well as other options for read_csv.


Sparklens UI Not working

the UI seems to be down
http://sparklens.qubole.com/

Still UI is down. Is there any alternative way to view the reports?


dosbox: 0.74-2 -> 0.74-3, fixing CVE-2019-7165, CVE-2019-12594

Motivation for this change
https://nvd.nist.gov/vuln/detail/CVE-2019-7165
https://nvd.nist.gov/vuln/detail/CVE-2019-12594
See also #64729
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

$ nix-review pr 64729 passes on NixOS
diff LGTM
dosbox and keen4 binaries work
[7 built, 114 copied (297.1 MiB), 63.9 MiB DL]
https://github.com/NixOS/nixpkgs/pull/64729
2 package were build:
dosbox keen4


Add an option to hide errors

Running scan_log4j_versions.py against the root directory generates tonnes of output for unopened files.
/usr/bin/python3 scan_log4j_versions.py /
Scanning /
snap/core18/2246/usr/share/doc/apparmor/changelog.Debian.gz: file could not be opened successfully
[Errno 2] No such file or directory: '/snap/core18/2246/usr/share/doc/bash/README.md.bash_completion.gz'
snap/core18/2246/usr/share/doc/bash/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/bash-completion/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/bsdutils/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/bzip2/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/cloud-guest-utils/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/cloud-init/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/coreutils/changelog.Debian.gz: file could not be opened successfully
snap/core18/2246/usr/share/doc/dash/changelog.Debian.gz: file could not be opened successfully
...

Please add an option to hide/disable errors on output.

Added -quiet flag in the latest commit.
Also changed the output format, so that you can filter out the interesting lines only with | grep ">>"

Thanks, works fine except on a few systems which return error:
Traceback (most recent call last):
  File "scan_log4j_versions.py", line 4, in <module>
    from enum import Enum, IntEnum, auto
ImportError: cannot import name 'auto'

Could you run python --version on those systems?
In any case, one quick fix you can try is using scan_log4j_version_p2.py instead (it's a python 2 port, but it also works with python 3). Would not recommend working with it always, it may be slower in some cases.

python3 --version
Python 3.6.9

Have to say it's a little puzzling, but I'll test with these versions precisely.
Did you try
python3 scan_log4j_versions_p2.py --quiet
?

python3 scan_log4j_versions_p2.py --quiet
Traceback (most recent call last):
  File "scan_log4j_versions_p2.py", line 5, in <module>
    from typing import IO, Set
ImportError: No module named 'typing'

Hi @kaipee .
Great, thanks for your input!
Fixed the dependence on newer language features in latest commit, tested with python3.4.10. Let me know if it helps (both scan_log4j_versions.py and scan_log4j_versions_p2.py should work with python3.4 now). Running _p2 script in python2.7.6 is not resolved for now (requires more extensive changes)

Seems to be working, many thanks!
wget https://raw.githubusercontent.com/jfrog/log4j-tools/main/scan_log4j_versions.py
chmod +x scan_log4j_versions.py
python3 scan_log4j_versions.py / -quiet
Scanning /


Support Cubbyhole to pass tokens

Spring Cloud Vault Config allows configuration of a static token spring.cloud.vault.token. Storing the token allows lookup attacks: The static key can be retrieved from the configuration by unauthorized users and can be used to access Vault.
Implementing a cubbyhole authentication for a short-lived (temporary) and long-lived (permanent) token can take advantage from token expiry. A short-lived token that allows a limited number of usages can be used by the application to obtain secrets from cubbyhole. The cubbyhole holds a long-lived token (or other credentials) which is used then to access Vault secrets.
Users could configure the short-lived token in spring.cloud.vault.token to obtain secrets from cubbyhole.
This approach prevents lookup attacks once the short-term token expires and reveals configuration issues/accesses to Vault is the short-lived token was used to access Vault because an application is no longer able to start up using the temporary token.

+1 That would be awesome!

I'd like to point to the new response wrapping feature which might simplify solving this issue: https://www.vaultproject.io/docs/concepts/response-wrapping.html

Response wrapping is using cubbyhole to store responses at v1/cubbyhole/response and generates a token with a TTL. Currently, I think the approach could look like:

Operator generates two tokens. A short and a long lived one
The long lived is stored in a path below cubbyhole/…
The short lived token and the path are configured in the app (spring.cloud.vault.cubbyhole.path)
App is started and then you're good to go

Does this make sense?
I will also open a ticket to support wrapped cubbyhole responses but that's different from cubbyhole usage for authentication.

Yes, that makes perfect sense. I'm not sure if it's relevant, but perhaps this method could be useful in determining whether the token has expired or already used:

A solution that works for me:
when creating a response-wrapped token you get:
1.token creation time
2. token's time to live
Adding these two and passing the result to the target-application helps you (in case of getting denial of token) determine which of the 2 occurred.

http://stackoverflow.com/a/38778650/801334


Genny

Changes
This builds on #98 to change action registration into an opt-in on a per-org basis, with repos being able to override org-level settings.

🎁 Refactors repos.yaml, *-exclude.yaml and *-omitted.yaml into a single file
No behavior changes in this PR (it does not change config.json).

/kind cleanup
/kind enhancement

Related #94

/hold
Don't look at this until #98 lands

closing due to staleness
cc @kvmware


[Driver] Use -platform_version to specify platform versions for the Darwin linker

Standardize the way in which we pass platform version information to
the Darwin linker, using the -platform_version option. In the case
of Mac Catalyst, there may be two such platform arguments.
-platform_version supports both the deployment target (subsuming options like -macosx_version_min) and the SDK version (which wasn't previously available
to the linker). Populate the SDK version by reading the SDKSettings.json that is
part of recent-ish Darwin SDKs, which can also contain macOS <-> iOS version
mappings for Catalyst.
Implements rdar://problem/55972144.

@swift-ci please test

@swift-ci please test source compatibility

Build failed
Swift Test OS X Platform
Git Sha - 970ff6b818945581306e38d4307493995675c113

@swift-ci please test

@swift-ci please test source compatibility

Build failed
Swift Test Linux Platform
Git Sha - 970ff6b818945581306e38d4307493995675c113

Build failed
Swift Test OS X Platform
Git Sha - 970ff6b818945581306e38d4307493995675c113


NodeJS Client - new DockerComposeEnvironment results in "build contains unsupported option: 'secrets'"

Expected Behaviour
DockerComposeEnvironment should support compose v2 where secrets are allowed under build. E.g.
version: '3.8'
services:
  server:
    container_name: server
    command: npm run start:debug
    env_file:
      - .env
    volumes:
      - .:/usr/src/app
      - node_modules:/usr/src/app/node_modules:consistent
    build:
      context: .
      target: development
      secrets:
        - npmrc

Actual Behaviour
Right now the above fails with
Error: Jest: Got error running globalSetup - /home/runner/work/server/test/service/setup.ts, reason: The Compose file './docker-compose.yaml' is invalid because:
services.server.build contains unsupported option: 'secrets'

Locally running docker compose up --build works fine and secrets being under build is supported. It seems like testcontainers aren't supporting compose v2 in the node js client?
Testcontainer Logs
N/A
Steps to Reproduce
See description above
Environment Information

Operating System: Mac Air m2
Docker Version: Docker version 24.0.2, build cb74dfc
Node version: v18.17.0
Testcontainers version: "testcontainers": "^10.3.2",

Hi @anton-prehendo, do you have both docker compose v1 and v2 installed on your machine?

I don't think so
% docker compose version
Docker Compose version v2.19.1

The error I posted is actually from tests running in CI, I assume the docker compose version in there would come from the testcontainers itself?

@anton-prehendo please try with version 10.4.0, where the docker-compose preference has changed from v1 -> v2


Modifications for gpu testing framework

Modified the Makefile to be more flexible

options include: FC, USE_MPI, USE_OPENMP, USE_DOUBLE, USE_OPENACC, DEBUG
example: FC=pgf90 USE_MPI=false USE_OPENMP=false USE_DOUBLE=false USE_OPENACC=false DEBUG=true

Added the ability to read in a specified namelist file.  This is done through an optional command line argument --namelist The default is "namelist.input".  Example of how to read in a different namelist: --namelist namelist.asd04-128x1k.input

These changes are designed to work with
https://github.com/sherimickelson/CM1_benchmarkingWF
To run the test suite, modify the top of the runit file and then execute that file.

Hi @sherimickelson , thanks for your update. I checked out the latest gpu-opt branch of CM1 Repo (commit: c5e2be3) and merged your changes. Then I checked out your Repo (https://github.com/sherimickelson/CM1_benchmarkingWF) and revised the runit script for some paths/ID (no changes to the compiler options). Then I ran the runit script by ./runit. The output contained the message Success!!! Job is in the queue. but it is followed by the error message immediately:

@sjsprecious, thanks for trying the code out.  Could you create a directory called "logs" in your CM1_benchmarkingWF directory and see if the error still appears?

Thanks @sherimickelson. After manually generating the logs folder under the CM1_benchmarkingWF directory, I did not see the previous error but encountered the following one:
nvlink error   : Undefined reference to 'parcel_module_find_vertical_location_index_' in 'parcel.o'
nvlink error   : Undefined reference to 'parcel_module_find_horizontal_location_index_' in 'parcel.o'
pgacclnk: child process exit status 2: /glade/u/apps/opt/nvhpc/21.11/Linux_x86_64/21.11/compilers/bin/tools/nvdd
make: *** [cm1] Error 2
mv: cannot stat ‘../run/cm1.exe’: No such file or directory
ERROR: Problem while building

With this error message, I could find the executables cm1-debug.exe and cm1-mpi-openacc-debug.exe under the run directory, but not the cm1-openacc-debug.exe.

And the cm1-mpi-openacc-debug.exe test run fails with the following error message:
FATAL ERROR: variable in data clause is partially present on the device: name=tmass
 file:/glade/scratch/sunjian/CM1/src/misclibs.f90 totmois line:3524

compared 295 blocks, 898427260 elements,<PHONE_NUMBER> bytes
1443305 errors found in 111 blocks
 absolute tolerance = 1.00000000000000008e-15, abs=15
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[61595,1],0]
  Exit code:    1

@sjsprecious I'm glad that fixed the first problem.  As for the build and run problems, those were the same problems I was seeing.  Maybe we can talk about them during our meeting this afternoon?

This appears to be an issue that is triggered when the _B4B flag is enabled.  I have done most of my testing lately without the _B4B flag which explains why I have not encountered this issue.

This appears to be an issue that is triggered when the _B4B flag is enabled. I have done most of my testing lately without the _B4B flag which explains why I have not encountered this issue.

Oh, my bad. I checked out the latest gpu-opt branch but forgot to comment out the _B4B flag. I would do another test with this flag turned off. Thanks @johnmauff for pointing this out!

After I commented out the _B4B flag and rebuilt/reran the runit script, I got the following runtime error for the cm1-mpi-openacc-debug.exe test:
call to cuStreamSynchronize returned error 700: Illegal address during kernel execution

compared 623 blocks,<PHONE_NUMBER> elements,<PHONE_NUMBER>6 bytes
195118334 errors found in 264 blocks
614 errors tolerated in 6 blocks
 absolute tolerance = 1.00000000000000008e-15, abs=15

And the same build error for the cm1-openacc-debug.exe test before still existed.

Disabling the DEBUG flag (DEBUG=FALSE) in the runit script will resolve the build error for cm1-openacc-debug.exe.

@sherimickelson @johnmauff Ok, I have a fix to the multi-MPI-GPU runtime error in this PR https://github.com/george-bryan/CM1/pull/26. After revising the source code, I am able to use this verification framework to run the MPI, one GPU and multi-MPI-GPU tests with different namelist input. The only failed test now is the multi-MPI-GPU one with the "namelist.sas.input" input. The error message says "Courant number has exceeded 1.5". It looks like a namelist issue rather than a code bug?

And one more minor comment: when we have more tests running, the root directory of CM1_benchmarkingWF becomes messy. Is it better to move all the shell scripts and PBS output to the logs folder?

Add verification test capabilitiy


Send Downlink Data with MQTT not working

Hello brocaar,
I try to send downlink message via MQTT. I follow the instruction and publish the json data to "application/[applicationID]/node/[devEUI]/tx" but I didn't see my message in "/api/nodes/{devEUI}/queue".
If I use HTTP Post to send, It works.
Thank you in advance,
Movsun

Are you sure you replaced the [applicationID] and [devEUI] with the correct application id and (lowercase) devEUI? Could you also check the lora-server-logs?

Just checked, but I don't see any problems with the latest version. When enqueueing data over MQTT, I get the following logs (lora-app-server):
INFO[8237] handler/mqtt: data-down payload received      topic="application/1/node/0101010101010101/tx"
INFO[8237] downlink queue item enqueued                  dev_eui=0101010101010101 id=1


Workflow-level option support

The structure of process currently supports task-level options, or the fixed set of workflow-level options defined for upload_options. I've run into a circumstance where tasks run in a pipeline's workflows need to access a common option, and it would be super helpful when building the payload not to have to be concerned with what tasks in a given workflow might need to access that option, instead making it available to all tasks.
As it stands, all task-level options are passed into the task's process method as kwargs. Being able to leverage that same mechanism for workflow-level options would be fantastic: each task in a workflow would get a reference to the option via the kwargs whether it requires it or not. In the case of duplicate keys between a task's options and the workflow options, it seems that the value from the task options should be given precedence.
Could we add a new (optional) object to the process object, say workflow_options? Perhaps an example would be helpful:
{
    "type": "FeatureCollection",
    "features": [
        ...
    ],
    "process": {
        "upload_options": {
            ...
        },
        "tasks": {
            "some-task": {
                "opt1": "bananas",
                "opt2": 42
            },
            "other-task": {}
        },
        "workflow_options": {
            "global_opt": ["a", "b", "c", "d"],
            "opt1": "apples"
        }
    }
}

If we passed this payload into a workflow with the tasks some-task and other-task, then the kwargs passed into the process method for each would be like the following:
# some-task
kwargs = {
    "global_opt": ["a", "b", "c", "d"],
    "opt1": "bananas",
    "opt2": 42,
}

# other-task
kwargs = {
    "global_opt": ["a", "b", "c", "d"],
    "opt1": "apples",
}

Thoughts?

I considered the fact that stac workflow implementations like cirrus have workflow already defined in the process definition. Currently that is a string parameter that needs to equal the workflow name for which the payload is to be dispatched. I was thinking that it could make sense to support workflow as an object instead of a string, which would then have name and options keys:
"workflow": {
    "name": "some-workflow",
    "options":  {
        "global_opt": ["a", "b", "c", "d"],
        "opt1": "apples"
    }
}

Except 1) workflow might not be supported by all stac workflow implementations (SWOOP maybe? I'm remembering a concern about having the workflow name in the payload, at least); 2) such a change would require updates on the cirrus side to support; and 3) process could be viewed, in some sense, as the workflow configuration object. With regard to the latter, it might then make sense not to name this workflow_options but just options. I think, though, given that the object is named process and that we already have upload_options that we need a more descriptive name to disambiguate it and make it clear what it is for. Other such names might be global_task_options, or just global_options, or something along those lines.

After discussion we have decided we are good proceed with the proposed plan. We do have reservations about the lack of clarity on the arguments passed into a task process method because of the use of kwargs, but that seems orthogonal to the concern here and we can address that with better parameter definition/validation in the future.

Agreed on the way forward for now. One comment about the process block field names should we ever look at reworking this block and updating dependent libraries (cirrus, swoop). When I look at this:
{
    "type": "FeatureCollection",
    "features": [
        ...
    ],
    "process": {
        "upload_options": {
            ...
        },
        "tasks": {
            "some-task": {
                "opt1": "bananas",
                "opt2": 42
            },
            "other-task": {}
        },
        "workflow_options": {
            "global_opt": ["a", "b", "c", "d"],
            "opt1": "apples"
        }
    }
}

...it seems that the tasks field is really a task_options field. Renaming tasks to task_options would harmonize the three top-level field names such that their names match their intended use. Makes things clearer, IMO.

Resolved by #167


Update vuln reporting language to match reality

We don't need to be the dispatch for all vulns, now that other projects are starting to have their own processes. But we don't want to discourage reports about stuff that isn't directly in k/k either.
Saying that we usually disclose vuln reports in 7 days is just not true. But, I think it's still good to aim for 7 days when we aren't blocked on and coordinating release of patches.

@philips @mayakacz @tallclair @jonpulsifer

@cjcullen is it appropriate to label this committee/product-security?
(I'm not too well up on the committee and organizational structure for Kubernetes)

@sftim yes
/lgtm

/committee product-security

lgtm

/approve


Support kerberos keytab file

What would you like to be added:
It is not possible to authenticate with a Kerberos keytab file when I use a kerberized hdfs as object storage.
JuiceFS support authenticating with a credential cache file only.
It would be good if support a keytab file as well.
Why is this needed:
Authenticating with a credential cache file is tricky under CSI Driver.
https://github.com/juicedata/juicefs-csi-driver/issues/572
Currently, we have to do like the below whenever mount a pod. because the ccache file will be expired at the end.

generate my ccache file by kinit
create a kubernetes secret with my ccache file.

I wrote the POC code, and It works well.
I can try to contribute if you are interested in this
thanks!

Thanks, we are interested in this.

@kyungwan-nam Hello, could you submit a pr for this issue. Some other users also encountered this problem.

@tangyoupeng
Sorry for the delay.
Users can set the environment variables KRB5KEYTAB, KRB5PRINCIPAL for keytab instead of KRB5CCNAME.

done by #3517


CMake Design for Faster Build Times

CMake usage
Introduction
CMake is the build system used in Hyperledger Iroha. It is a complex system, with no single way to achieve the desired result. Some ways are newer and more preferable, however a lot of options that CMake provides have different trade offs and it is not clear which one to use.
The aim of this document is to analyze what methods of organizing the project exist, and which one achieves our goals, which are: fast build and separation of components from each other.
Build performance measurement
Any discussion of build time without measurement is meaningless. In this case we are interested in build time of the whole project. Simple measurement can be obtained with time utility like this:
gtime -f "%S\t%U" cmake --build build --target -- -j6 > /dev/null
This will output system and user time of the build.
Notice that gtime is OS X specific. For Linux use /usr/bin/time (not builtin time!)
Of course, ccache must be disabled for the measurement since we are interested in compilation from scratch.
How precise is this measurement? Running this script 10 times and then counting average build time on a Macbook yields standard deviation of the measurement around 1%. This means that this method is precise enough to be used. We should consider incorporating it into our CI.
Project Structure
Projects are organized in subdirectories with source files separated into physical components. The question is how to organize CMake to build these components
Single Library
This approach consists of adding all source files of the project, into a single library, and then linking that library to an executable or tests.
This has several advantages:

Less linking required
No dependencies between libraries, which means more parallelism

However there are severe disadvantages to this method:

CMake file gets big
Every executable must be linked with the whole library
Every time change occurs, whole library must be relinked
Configuration cannot be different for different modules

This approach is used by a lot of projects, one of the examples is RocksDB database.
https://github.com/facebook/rocksdb/blob/da1c64b6e75048068bf8599f198b8fe0e54eedfc/CMakeLists.txt#L462 - RocksDB CMakelists.txt where sources are defined.
Multiple Libraries
This approach constructs each module as a separate library, which are then linked together for the final executable.
Advantages:

Libraries can be defined in their own directories CMakeLists.
Tests only need to link with required libraries.
Each library can have its own configuration (defines, compile options)

Disadvantages:

More linking required
More space for build directory

https://github.com/ethereum/aleth/blob/master/CMakeLists.txt - Ethereum C++ Client
Difference in build time
In both cases, you must compile all object files for the resulting executable. So there is no difference in compilation time. However single library requires only linking a single library and then linking it to each executable. The multiple libraries option requires a lot more linking.
The total build time depends on how slow is the linking of the libraries. Usually linking is a just a fraction of compilation time.
What to use in Iroha
Currently iroha is built as a set of libraries which are then linked into irohad executable and various tests. The impact of this on a performance is not very clear, and we have to measure link and compile time to get the picture of where most of the build time is spent.
If we want to reduce link time, it could be more beneficial to create a single target for all source files, and then add all executables to it. This however may be problematic since now we cannot separately configure concrete parts of the system.
Test Organization
How to organize tests is an important question, since usually test code is bigger than application code. The general question is how granular test executables must be. We might have a single binary with all the tests from the whole project, executable for one module or one executable for each test file.
When we have single binary for a test, we must build everything in order to run a single test. However this compilation must only be done once, and all subsequent recompiles are as cheap as the other options.
On the other hand, having many executables increases total size of the build files, and as in the case with multiple libraries, more linking needs to be done.
At the moment, almost each test file is compiled into a separate executable. This leads to about 100+ executables being generated, all of which statically link necessary components. This leads to a huge code duplication, and a large amount of linking required.
To mitigate that, we can reorganize tests to have a single executable per module. This would lead to faster build time, less space wasted. However this might make it harder to profile tests execution time, since they will be joined together and hard to separate.
To do that reorganization, some code changes must be done in addition to CMake rewrite. One of the issues that I have encountered trying to do that, is the fact that a lot of functions in header files used in tests are not declared inline, which leads to link errors because of duplicate symbols. This is easily solvable, but may slow down the refactoring.
Possible Build Optimizations
Now that we've seen basic information about project structure, we need to examine CMake and C++ specific options which can improve build time of the project.
Unity build
Unity build is a C++ "technique" to improve compile times. It involves combining all .cpp files into a single big file, which is then compiled only once.
This has a huge impact on the performance of the compilation for several reasons:

file IO is drastically reduced single every header is included only once.
Every template is instantiated only once.

However unity build has major drawbacks, which prevent it from being just a drop-in solution:

Every symbol must be unique, since they will all end up in the same file.
Memory usage during compilation may increase significantly, since the whole combined file is compiled at once.

Effect of compilation speed varies, depending on the project and has to be tested.
Unity build can break correct code, for example when the same symbol is declared in separate files. This means that if we want to implement this thing for iroha, some substantial refactoring might be required.
Cotire plugin for cmake can create unity build targets by itself, and we should try that and see the benefit.
https://buffered.io/posts/the-magic-of-unity-builds/ - about unity builds
Also as an example of Unity build we can check out Ripple project:
https://github.com/ripple/rippled/blob/96c949a99791fdd077b091f682139a050cd1034b/CMakeLists.txt#L1331
Precompiled Headers
Precompiled headers does not solve problem of the clean build (for example for CI), however for development, they can significantly speed up the build. The idea is to precompile headers that rarely change (such as third party or standard library), and then use these precompiled headers when compiling targets. This significantly improves compilation time. However it has some limitations, such as only one header for a source file can be precompiled.
Both unity build and precompiled headers techniques are supported in cmake by cotire plugin.
https://github.com/sakra/cotire
Precompiled headers require us to carefully separate rarely changed header files into a separate header, which then gets included everywhere. Cotire seems to be able to extract 3rd party headers into a precompiled header, which needs to be checked for iroha project.
https://clang.llvm.org/docs/UsersManual.html#usersmanual-precompiled-headers - CLang user manual precompiled headers
Modules Isolation
Even though performance is an important factor, we must consider also correctness of build configuration. To ensure that, modules must be as isolated as possible, to prevent unwanted dependencies. One of the big issues in this regardis isolation of header files. Currently, I can include every header from irohad into every file. This means that dependencies are not transparent, since we do not state in CMake, that this module depends on other module, even though I can include header file from that module.
This happens because we define global include_directories() for the whole irohad directory. This issue can be mitigated, if each library defines its own include directories. The drawback of that, with current directory layout we would have to change include paths, for example "ametsuchi/storage.hpp" will become "storage.hpp" etc. Whether or not that is a good thing, is a subject of discussion.
Conclusion
As a main thing, I would like to get total build time statistics into CI as soon as possible.
As it stands, the easiest thing we can do to improve build times, is to reduce number of test executables. All other changes need to be investigated, since they may require some substantial refactoring.
As for the project structure, combining all components into a single library seems to present more issues than benefits, considering that the link time is not substantial part of the build.

If we want to reduce link time, it could be more beneficial to create a single target for all source files, and then add all executables to it. This however may be problematic since now we cannot separately configure concrete parts of the system.

if you do this, you will slow down average compilation time, because single change will lead to the full recompilation

if you do this, you will slow down average compilation time, because single change will lead to the full recompilation
Not really, since only the changed file will be recompiled.


[ENG-5329] Phase 2: Dep-group: Celery / Kombu

Purpose
Python and dependencies upgrade
Changes
Bumped celery, kombu and their helper libs
QA Notes
Please make verification statements inspired by your code and what your code touches.

Verify
Verify

What are the areas of risk?
Any concerns/considerations/questions that development raised?
Documentation

Side Effects

Ticket
ENG-5329

Nice! Merged.


application not compiling, .net 6.0

how to fix these issues? complains path is larger than 256 characters,

The path where your project is located exceeds the maximum character of 256 characters of the Windows computer path, please move to the disk and directory path

does it compile to .net 6 for .net 5?
On Wed, Feb 23, 2022 at 3:40 PM tchivs @.***> wrote:

Closed #3 https://github.com/tchivs/Abp.AspNetCore.Blazor.Theme/issues/3
.
—
Reply to this email directly, view it on GitHub
https://github.com/tchivs/Abp.AspNetCore.Blazor.Theme/issues/3#event-6124697154,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXEVFKHECIJHXZHKWWJOYEDU4TISXANCNFSM5PD4HNOA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***
com>


input.password doesn't have scale prop

Bug report 🐞
Version & Environment

Version of browser
Version of geist-ui/react latest

Expected Behaviour
No type errors
Actual results (or Errors)
I got an error:
<Input.Password scale={4/3} />

It throws a type error

Fixed in v2.3.0.


Fetch libsodium binary from github

High Level Description
Server download.libsodium.org has rather strict SSL configuration as with newer SSL libraries build fails with
python 3.5 [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)

as python requests fail to download the binary.
The library version stays the same, only fetched from different mirror.
$ sha1sum libsodium-1.0.9.tar.gz
6e886fa6c7b0c0dc8a9039f49e1b04532401a6ae  libsodium-1.0.9.tar.gz

Related Issues

no related issue so far

Checklist for all PR's

[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

I am okay with this. It's from the library developer's github repo.

@mesosphere-mergebot label ready for review

@mesosphere-mergebot bump-ee

Thanks for your contribution, @deric. We'll merge this PR.

@deric sorry somehow I closed this unintentionally and it was never merged! would you mind re-opening against the 1.10 branch so we can get it in? Sorry for the mixup.

@lingmann  - as an admin, if you re-target it, we could be fine with it too. (And could possibly merge it as this PR satisfies all the required criteria).

Do we also need to create a pull request against the master branch?

mergebot/enterprise/build-status/aggregate failed as of a flake. Applying the ship-it label.

Is there a JIRA ticket that can be closed now that this has been merged?

@jgehrcke None that I would know of.

Tracked and closed it here: https://jira.mesosphere.com/browse/DCOS_OSS-1694


ARTEMIS-4349 Replace Guava cache with Caffeine

based on benchmark https://github.com/ben-manes/caffeine/wiki/Benchmarks

any idea how active is Caffeine? more specifically how good are them with fixing CVEs?
there was a couple of CVEs in the past with guava.  https://www.cvedetails.com/product/52274/Google-Guava.html?vendor_id=1224 and we had to update the version before.
my concern with the replacement is how good would be fix things like these?

well.. at least some initial googling shows that they are active on fixing this:
https://security.snyk.io/package/maven/com.github.ben-manes.caffeine:caffeine
any opinions?

I'm no fan of guava but my first question on any change like this is what benefit if any does it actually impart? Is there some measurable performance improvement worth changing it now vs at some later stage like a major version bump where these types of changes are more expected? (e.g. Artemis 3.0)

Caffeine is essentially the successor to Google's Guava Cache. It's used by a lot of projects. Spring, for example, started moving away from Guava to Caffeine in 2015. I considered using Caffeine when I first started using Guava Cache on ARTEMIS-2886, but I decided on Guava mainly because we already had a dependency on it. This PR completely removes Guava from the code-base in lieu of Caffeine which is fine by me.
My only real concern is essentially the same as Tim's which I put on ARTEMIS-4349. Changing Caffeine for the sake of performance doesn't make much sense without clear data that changing will actually make any meaningful difference. At this point we don't have that data.

@jbertram based on what you said it sounds like caffeine has a bigger community than guava? that would be a good reason for the replacement by itself.

I'm running the whole test suite with these changes (on a separate CI).
If caffeine has a bigger community, the jar is smaller, I think we should bring this right away (if tests are ok of course).
my initial concern  was about the community support and CVEs.. but from what I later gathered this is actually better.
performance wise I think the differences would be impossible to measure. We currently use some login changes from Security as Justin mentioned... but you would have to use the cache quite a lot to have any significant result.
I think we should merge this, but not much for the performance reasons.
anyone has anything against that?

Guava itself is much more than just the cache, but if you're comparing Guava Cache vs. Caffeine then I think more folks are using Caffeine.
Also, since Guava is a lot more than just a cache that means Caffeine is much smaller and will likely reduce risk of CVEs.

apparently there are a lot of test failures if running this. leave it with me for another day please so I can check if it was a real failure or a build issue.

Hi,
If you do not have a performance reason to switch then staying with Guava's is perfectly fine. I would not use their asMap().compute as a wonky implementation given it was added so late (and you don't use it, so that's fine). Guava's is robust but not actively maintained (CacheBuilder's JavaDoc recommends Caffeine).
The most common cause for tests to fail during a migration is that Caffeine switches eviction and listeners to be executed async by default, whereas Guava piggybacks on the calling thread. You can emulate that using Caffeine.executor(Runnable::run) if desired. The problem is that often tests assumed no concurrency (immediate eviction or callbacks invoked) and that is no longer a valid assumption. In both cache's their internal logic is very inexpensive (e.g. LRU reordering), but we do not know the cost of the user's callback. Since the JVM now offers shared threads and Caffeine offers an AsyncCache, defaulting to async to minimize user-facing latencies seemed like a reasonable choice. There are other subtle differences, such as if assuming lru eviction order, so you might want to review this migration page.
Whatever you decide to do is fine with me. Feel welcome to reach out if you have any questions or concerns.

All the arguments seems to indicate we should do the replacement.
However this is still failing to build. I have a pipeline where I execute the whole test suite:
"[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project artemis-quorum-ri: Compilation failure
[ERROR] An unknown compilation problem occurred
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :artemis-quorum-ri
[Pipeline] }"
This is also translated as errors on the build here.
to test this locally you could do:
mvn clean; mvn install -DskipTests=true -Pdev -Derrorprone

still failing to build

without the latest commit build is successful

It may completes but as per earlier comments it isn't really 'successfull'; the compiled code then doesnt actually work because all the code wasnt generated, since the change stopped the logging annotation processor from running at all, meaning there are missing classes that stop various modules (client, broker, etc) from working.

stopped the logging annotation processor

let me check this regression

still not building:
[INFO] Compiling 6 source files with javac [debug target 11] to target/classes
compiler message file broken: key=compiler.misc.msg.bug arguments=11.0.8, {1}, {2}, {3}, {4}, {5}, {6}, {7}
java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'
at com.google.errorprone.scanner.ScannerSupplier.defaultSeverities(ScannerSupplier.java:67)
at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerInfos(ScannerSupplier.java:88)
at com.google.errorprone.scanner.BuiltInCheckerSuppliers.allChecks(BuiltInCheckerSuppliers.java:600)
at com.google.errorprone.scanner.BuiltInCheckerSuppliers.defaultChecks(BuiltInCheckerSuppliers.java:608)
at com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:35)
at jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:215)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.prepareCompiler(JavacTaskImpl.java:199)

still not building

yep, going to do https://github.com/apache/activemq-artemis/pull/4545 first

want the guava dependencyManagement restored

for the moment - yes, keep annotationProcessorPaths for further change

want the guava dependencyManagement restored

for the moment - yes, keep annotationProcessorPaths for further change

Regardless of the annotationProcessorPaths situation. Guava will still be getting used in the project after this change, no matter which way the ErrorProne etc annotation processors are being configured (or even if we removed ErrorProne entirely).

Guava will still be getting used in the project

directly or as transitive dependency?

Both (once you restore the console bit you have removed)

FWIW I dont see #4545 as a pre-requisite to this PR. It may get rid of the build error, in a different way, but it doesnt change the fixes/restorations still needing made here, which would also get rid of the build problem even without the other PR.

pre-requisite to this PR

its not a blocker but allow to implement current changes easy way

I raised a new PR, #4584, to address the various trivial commented build etc issues on this PR which were preventing this progressing. Unfortunately it turns out this PR was still failing a test (MQTTSecurityManagerTest.testSecurityManagerModifyClientIDAndStealConnection) in its current state, and so the new PR is also and also cant be merged without more work first to identify+fix that issue.