Feature Request: Hierarchical storage and processing in xarray

I am using xarray for processing geospatial data and have encountered two major challenges with existing data structures in xarray:

Data arrays stored in an xarray Dataset cannot be grouped into hierarchical levels/logical subsets to reflect the internal organisation of the data. This makes it difficult to identify and process a subset of the data variables that pertain to a specific problem.

When two data arrays having a shared dimension but different coordinate values along the dimension are merged into a Dataset, the union of coordinate values from the 2 data arrays becomes the new coordinate set corresponding to that dimension. Consequently, when the value of a variable in the dataset corresponding to a coordinate value is unknown, nan is used as a substitute which results in memory wastage.

I would like to suggest a tree-based data structure for xarray in which the leaves store individual data arrays and the other nodes store the hierarchical information. Since data arrays are stored independently, each dimension only needs to be associated with coordinate values that are valid for that data array.
To meet these requirements, I have implemented a data structure that also supports the below capabilities:

Standard xarray methods can be applied to the tree at all hierarchical levels, i.e., when a function is called at a hierarchical level, it is mapped over all data arrays that occur at the leaves under the corresponding node. For example, say I have a tree object (lets call it dt) with child nodes: weather, satellite image and population. Each of these nodes has data arrays/subtrees under it.

The mean over time of all data variables associated with weather can be obtained using dt.weather.mean('time') which applies the function to sea_surface_temperature, dew_point_temperature, wind_speed and pressure.

It can be encoded into the netCDF format, like xarray Datasets.
It supports item assignment at all hierarchical levels.

I would like to know of the possibility of introducing such a data structure in xarray and the challenges involved in the same.

@emilbiju - thanks for opening an issue here. You may want to take a look at the conversation in #1092.

Thanks @jhamman for sharing the link. Here are my thoughts on the same:
For use-cases similar to the one I have mentioned, I think it would be more meaningful to allow the tree structure (calling it Datatree further) to exist as a separate data structure instead of residing within the Dataset. From what I understand, the xarray Dataset would enforce all its component variables to share the same coordinate set for a given dimension name. This would again result in memory wastage with nan values when the value corresponding to a coordinate is unknown.
Besides, xarray only allows attribute access for getting (and not setting) values, but a separate data structure can allow attribute access for setting values as well. For example, the data structure that I have implemented would allow something like dt.weather = dt.weather.mean('time') to alter all the data arrays under the weather node.
I am currently using attribute-based access for accessing child nodes/data arrays in the Datatree as it appears to reflect the tree structure better, but as @shoyer has pointed out, tuple-based access might be easier to use programmatically.
Instead of using netCDF4 groups for encoding the Datatree, I am currently following a simple 3-step process:

Combine all the data arrays at the leaves of a Datatree object into a dataset.
Add an additional data array to the dataset that would contain an ancestor matrix (or any other array-like representation) that can encode the hierarchical structure with a coordinate set containing names of the tree nodes.
Use the xarray.Dataset.to_netcdf method to store it in a netCDF file.

Therefore, within the netCDF file, it would exist just as a Dataset. A specially implemented Datatree.open_datatree method can open the dataset, detect this additional array and recreate the tree structure to instantiate the object. I would like to know if using netCDF4 groups instead provide any advantages over this approach?

Thanks for writing this up @emilbiju . These are very interesting ideas

The nice thing about using NetCDF groups (or HDF5?) is that it is a standard and your data files are readable using other software.

So far, xarray has been reluctant to add "groups" or this kind of hierarchical organization because of all the additional complexity involved (#1092)

That said, there is definitely interest in a package that provides a high-level object composed of multiple xarray datasets (again #1092). So I encourage you to post your code online so others can try it out and iterate.
a. For example, our friends over at Arviz have a InferenceData structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html

I would be open to exploring adding a hierarchical data structure into xarray (on an experimental basis, to start), but it would need someone with serious interest and time to make it happen. Certainly there are plenty of use cases across various fields.

The data model you sketch out here looks very similar to what we discussed in #1092. I agree that the semantics are well defined.
The main question in my mind is whether it would make more sense to make an entirely new data structure (e.g., xarray.TreeDataset) or add in a new feature like groups to the existing `xarray.Dataset.
Probably a new data structure would be easier at this point, because would keep Dataset simpler and wouldn't break existing code that works on xarray.Dataset.

@joshmoore - based on https://github.com/pangeo-forge/pangeo-forge/pull/27#issuecomment-755397835, you may be interested in this issue. One way to do multiscale datasets in Xarray would be to use hierarchical groups (one group per scale).

@joshmoore - based on https://github.com/pangeo-forge/pangeo-forge/pull/27#issuecomment-755397835, you may be interested in this issue. One way to do multiscale datasets in Xarray would be to use hierarchical groups (one group per scale).

a. For example, our friends over at Arviz have a InferenceData structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html

Just a note that this link has moved to: https://arviz-devs.github.io/arviz/getting_started/XarrayforArviZ.html

a. For example, our friends over at Arviz have a InferenceData structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html

Just a note that this link has moved to: https://arviz-devs.github.io/arviz/getting_started/XarrayforArviZ.html

Thanks for the link, @jhamman. The most immediate issue I ran into when trying to use xarray with OME-Zarr data does seem similar. A rough representation of one multiscale image is:
image_pyramid:
  |_ zyx_array_high_res
  |_ zyx_array_mid_res
  |_ zyx_array_low_res

but of course the x, y and z dimensions are of different sizes in each volume.

Thanks for the link, @jhamman. The most immediate issue I ran into when trying to use xarray with OME-Zarr data does seem similar. A rough representation of one multiscale image is:
image_pyramid:
  |_ zyx_array_high_res
  |_ zyx_array_mid_res
  |_ zyx_array_low_res

but of course the x, y and z dimensions are of different sizes in each volume.

@jhamman @joshmoore a prototype to bring together XArray and OME-Zarr/NGFF with multiple groups:
https://github.com/OpenImaging/miqa/blob/master/server/scripts/compress_encode.py

On today's Xarray dev call, we discussed pursuing another CZI grant to support this feature in Xarray. The image pyramid use case would provide a strong link to the bioimaging community. @alexamici and the B-open folks seem enthusiastic.
I had to leave the meeting early, so I didn't hear the end of the conversation. But did we decide who might serve as PI for such a proposal?

FWIW, a while ago I wrote a mock-up (and probably outdated) DatasetNode class:
https://gist.github.com/benbovy/92e7c76220af1aaa4b3a0b65374e233a  (nbviewer link)

This is related to some very recent work we have been doing at NSLS-II, primarily lead by @danielballan .

Not really sure if there is anything we can do from ArviZ to help with that,  if there is let us know and we'll do our best cc @percygautam

I think @alexamici and I can write the technical part of the proposal.

Happy to provide assistance on the image pyramid (i.e. "multiscale") use case.

So we have:

A technical team who can write the proposal and execute the proposed work (@aurghs & @alexamici of B-open)
Numerous supporting use cases from the bioimaging (@joshmoore), condensed matter (@tacaswell), and bayesian modeling (ArviZ; @OriolAbril) domains

We are just missing a PI, someone who is willing to put their name on top of the proposal and click submit. I have gone on record as committed to not leading any new proposals this year. And in any case, this is a good opportunity for someone else from the @pydata/xarray core dev team to try on a leadership role.

I volunteer to contribute writing to this from the condensed matter / sychrotron user facility perspective.

I can shoulder part of the load and help is definitely needed. LOI is due on Tuesday. I'll take a stab this evening and post a link.

Here are some biomedical papers that are using ArviZ and therefore xarray even if most don't cite xarray and some don't cite ArviZ either. Topics are quite disperse: covid, psychology, biomolecules, oncology...
Some ArviZ recent biomedical citations

Arroyuelo, A., Vila, J., & Martin, O. A. (2020). Exploring the quality of protein structural models from a Bayesian perspective. bioRxiv.
Axen, S. D. (2020). Representing Ensembles of Molecules (Doctoral dissertation, UCSF).
Brauner, J. M., Mindermann, S., Sharma, M., Johnston, D., Salvatier, J., Gavenčiak, T., ... & Kulveit, J. (2021). Inferring the effectiveness of government interventions against COVID-19. Science, 371(6531).
Busch-Moreno, S., Tuomainen, J., & Vinson, D. (2020). Trait Anxiety Effects on Late Phase Threatening Speech Processing: Evidence from EEG. bioRxiv.
Busch-Moreno, S., Tuomainen, J., & Vinson, D. (2021). Semantic and prosodic threat processing in trait anxiety: is repetitive thinking influencing responses?. Cognition and Emotion, 35(1), 50-70.
Dehning, J., Zierenberg, J., Spitzner, F. P., Wibral, M., Neto, J. P., Wilczek, M., & Priesemann, V. (2020). Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions. Science, 369(6500).
Heilbron, E., Martìn, O., & Fumagalli, E. (2020). Efectos protectores de los alimentos andinos contra el daño producido por el alcohol a nivel del epitelio intestinal, una aproximación estadística. Ciencia, Docencia y Tecnología, 31(61 nov-mar).
Legrand, N., Nikolova, N., Correa, C., Brændholt, M., Stuckert, A., Kildahl, N., ... & Allen, M. (2021). The heart rate discrimination task: a psychophysical method to estimate the accuracy and precision of interoceptive beliefs. bioRxiv.
Wang, Y. (2020, September). Data Analysis of Psychological Measurement of Intelligent Internet-assisted Sports Training based on Bio-Sensors. In 2020 International Conference on Smart Electronics and Communication (ICOSEC) (pp. 474-477). IEEE.
WASSERMAN, A., SHRAGER, J., & SHAPIRO, M. A Multilevel Bayesian Model for Precision Oncology.
Weindel, G., Anders, R., Alario, F. X., & Burle, B. (2020). Assessing model-based inferences in decision making with single-trial response time decomposition. Journal of Experimental Psychology: General.
Yamagata, Y. (2020). Simultaneous estimation of the effective reproducing number and the detection rate of COVID-19. arXiv e-prints, arXiv-2005.

I'm excited to see this coming together! I would be happy to advise as well...
Side note: at some point, this would probably be worth adding to Xarray's official roadmap.

We could also provide a use-case in remote sensing: it would be really useful in the interferometric processing for managing Sentinel-1 IW and EW SLC data, which has multiple tiles (burts) partially overlapping in one direction (azimuth).

This sounds like an interesting project - I'm also about to be able to work on xarray much more directly (thanks @rabernat ).
Should I add this as another xarray project board alongside explicit indexes and so on?
I wonder if this could find another domain use case in plasmapy as part of the overall plasma object @StanczakDominik? At the very least this would allow you to store all the various equilibrium and diagnostics information that goes in an EFIT file.

Whoa, that sounds awesome! Thanks for the heads up :) Definitely could be quite handy, looking forward to seeing how this develops. @rocco8773 this should be interesting for you as well :)

For scientific imaging, i.e. biomicroscopy, medical imaging, where xarray compatibility is being considered in the NGFF, it would be helpful to avoid unnecessary divergence by ensuring the proposed hierarchical storage is compatible. This would mean:

Each scale / group can be independently treated as an xarray.Dataset.
They are organized in such a way that the collection of scales can be referenced as it is now, i.e. as a collection of paths,

  “multiscales”: [
    {
      “datasets” : [
          {"path": "0"},
          {"path": "1"},
          {"path": "2"},
          {"path": "3"},
          {"path": "4"}
        ]
      “version” : “0.1”
    }
  ]
}

Picking up on @dcherian's https://github.com/pydata/xarray/issues/4118#issuecomment-806954634 and @rabernat's https://github.com/ome/ngff/issues/48#issuecomment-833456889, Zarr was also accepted to the second round and certainly references this issue in case we want to sync up. (Apologies if I missed where that discussion moved.)

A simple comment/question:
In xarray.Dataset, why not just use the Unix-path notation into a "flat" dict model?
Actually, netCDF4 implements this Unix-like path access to groups and variables: /path/to/group/variable.
All of the hierarchical stuff (e.g., getting a sub-Dataset from a random group) and conventions (e.g., dimensions scoping rule) would then be driven by the parsing of strings only. It's all about symbolic names (like in a file system right?) and there would be not any hierarchical data in memory anymore.
My question is then: Are there some tricky points for xarray.Dataset not to go this simple way?
Some related remarks:

About the attribute access to variables: I don't really know why this exist at all since it is all about mixing unrelated namespaces: (1) the class internals and (2) the user's variables one. Mixing namespaces seems very bad to me: it makes some variable names forbidden in order to avoid any collision between the two namespaces, it usually imply unnecessarily complex code with corner cases to deal with.
About netCDF4 being a self-described format: xarray API has open_dataset(filepath), but this function is unable to read the whole file in memory without getting help from a priori file content description, i.e., the names of the groups if you follow me. Considering xarray for simple tasks like geographical-selection-croppping, it seems to ignore the self-describing nature of netCDF4 format. As far as I can understand the situation, a "flat" model could be a good way to go.

cc @d-v-b and https://github.com/JaneliaSciComp/xarray-multiscale

Flagging another possible use case, this time in Magnetic Confinement Fusion: representing the IMAS data model.
IMAS is currently closed-source (being part of the ITER project), but there is a big push to make it open-source and the standard data model for tokamak plasma data.
I'm not very familiar with IMAS (@smithsp and @orso82 are more so), but it is hierarchical. There is some more information in appendix A3 of this paper, which talks about "taking advantage of the homogeneity of grid sizes that is commonly found across arrays of structures", which sounds very closely related to the DataTree proposal.
This might allow the xarray.DataTree to do more of the heavy-lifting within OMAS (which already uses xarray, and is intended to be compatible with IMAS).

@martinitus raises a really interesting point about tags vs hierarchical structures over in https://github.com/pydata/xarray/issues/1092#issuecomment-868324949

However, one point I didn't see in the discussion is the following:
Hierarchical structures often force a user to come up with some arbitrary order of hierarchy levels. The classical example is document filing: do you put your health insurance documents under /insurance/health/2021, 2021/health/insurance,....?
One solution to that is a tagging of documents instead of putting them into a hierarchy. This would give the full flexibility to retrieve any flat DataSet out of a TaggedDataSet by specifying the set of tags that the individual DataArrays must be listed under.

I think using tags is a really interesting alternative to hierarchies. I don't have a clear sense of the overall tradeoffs, though.

I think using tags is a really interesting alternative to hierarchies. I don't have a clear sense of the overall tradeoffs, though.

That is interesting. I think there is an argument for using a hierarchical model to map onto the full netCDF data model with groups, but perhaps methods to select elements via tags could be included too, for the best of both?

@shoyer if you used tags wouldn't you lose the ability to round-trip a netCDF file with groups? When you read in the groups from the file you would be throwing information away by going from a hierarchy A/B to simply tags A&B, and there wouldn't be a way to restore that before calling .to_netcdf() would there?

if you used tags wouldn't you lose the ability to round-trip a netCDF file with groups?

That sounds right to me -- a downside of tags is that they can't be (uniquely) expressed in a hierarchical arrangement like those found in HDF5/netCDF4 files.
But if this is a better way to organize data in memory, we could consider how to make an adapter layer for on disk storage.

Some other thoughts about tags:

Does the definition of tags include variable names of DataArrays? I think it should.

As @martinitus mentioned, a DataTree containing only leaves with only 1 tag each is effectively a Dataset. I wonder if Dataset could be refactored to be a special case of a more general DataTree, possibly as a subclass?

Selecting via tags would need to allow a distinction between "get me all leaves with these exact tags" and "get me all leaves whose tags include these ones". Maybe dt.choose_only(tags) and dt.choose_all(tags)?

The latter type of tag-based access would make plotting different leaves against one another easier too - given a multi-resolution (or multi-model) datatree like this:

dt
|-- high_res
|    |-- temperature
|    |-- CO2
|-- medium_res
|    |-- temperature
|    |-- CO2
|-- low_res
|    |-- temperature
|    |-- CO2

then assuming that the definition of tags included the DataArray variable names, then
dt.choose_all('temperature').plot.line(x='time')
would select all leaves with a tempature tag, check that the temperature DataArrays had the same dimensions (but no need for any time coordinates to share size or values), and then plot them against one another on the same axes. This would be so useful - I would say this use case is 90% of the reason users iterate over dictionaries of datasets currently.

With a tag-based system you can create cycles of tags, like A&B, B&C, C&A, which you can't really do with hierarchical trees. I don't think that actually causes any problems though...

That sounds right to me -- a downside of tags is that they can't be (uniquely) expressed in a hierarchical arrangement like those found in HDF5/netCDF4 files.

hdf5 allows for internal links so a datasets and groups can appear in multiple places in the tree.  You can even make cycles where groups are in them selves (or their children).  The NeXuS format (the xray/neutron one) makes heavy use of this to let data appear both where it "makes sense" from a science point of view from an instrumentation point of view.
I think it is reasonable to expect that netcdf -> xarray -> netcdf always , however I think it is unreasonable to ask that xarray -> netcdf -> xarray will always work.  I think it is OK if xarray can express more complex relationship and structures that you can in netcdf (or hdf5 or any existing at-rest format).   In an extreme case, consider an interface to a database that returns xarrays 😈 .

As a user who (so far) does not use any netCDF or HDF5 features of xarray I obviously would not like to have a otherwise potentially useful feature blocked by restrictions imposed by netCDF or HDF5 ;-).
That said - I think @tacaswell comment about round trips is very reasonable and such invariants should be maintained! It would be extremely confusing for users if netcdf -> xarray-> netcdf is not a  "no-op". The same obviously holds true for any other storage format. As a user I would generally expect something like the following:
a1= xarray.load("foo.myformat")
xarray.save( a1, "bar.myformat")
a2= xarray.load("bar.myformat")
assert a1 == a2, "Why should they not be exactly equal?!?"

I think that xarray's current use of both dict-like access and attribute-like access for variables makes representing a general netCDF file in a single DataTree incompatible with the nice syntax that @emilbiju originally suggested.
Consider a tree with a node structure for a hypothetical DataTree object dt that looks something like
DataTree("root")
|-- DatasetNode("weather")
|   |-- DatasetNode("temperature")
|   |   |-- DataArrayNode("sea_surface_temperature")
|   |   |-- DataArrayNode("dew_point_temperature")
|   |-- DataArrayNode("wind_speed")
|-- DataArrayNode("population")

We ideally want to be able to seamlessly access both subtrees and individual variables via chains of keys, e.g.
weather_subtree = dt['weather'], and wind_speed_da = dt['weather']['wind_speed']. (We want that so that each subtree behaves as much like an xarray.Dataset as possible, with respect to mapping functions over all its child nodes and so on.)
This particular example is fine, and would correspond to a netCDF file with groups "root", "root/weather", and "root/weather/temperature", plus the four stored DataArray variables.
However, if one of the variables has the same name as one of the groups (which I think is permitted in the netCDF format), then there is no easy way to access all the elements whilst retaining the nice syntax. For example consider
DataTree("root")
|-- DatasetNode("A")
|   |-- DatasetNode("B")
|   |   |-- DataArrayNode("foo")
|   |   |-- DataArrayNode("bar")
|   |-- DataArrayNode("B")
|-- DataArrayNode("C")

Now we have a key collision between the group named "B" and the DataArray named "B", i.e. dt['A']['B'] is ambiguous.
We can't just forbid this type of tree because then there would be netCDF files that we couldn't represent as a DataTree, so we would not have the property netCDF -> xarray.DataTree -> netCDF in general.
We can't use different types of access (e.g. subtree = dt.A.B for the subtree and da = dt.A['B'] for the variable, because we've already given up the .B namespace to also point to the variable (i.e. same location as ['B']). If we break that convention it's going to be very confusing for users who are expecting the root of the DataTree to behave like xarray.Dataset currently does.
(We could divide access through __call__ like ds['A']('B') but that wouldn't be very pythonic).
The only way I can see around this is to hide a node's data variables behind a .ds property (i.e. da = dt['A'].ds['B']), or get groups via a dedicated method (i.e. subtree = dt.get_child('A')), but those are so much more ugly and less intuitive that it feels like a shame to have to do that.
It sounds like @emilbiju avoided this by not satisfying netCDF -> xarray.DataTree -> netCDF:

(Instead of using netCDF4 groups for encoding the Datatree ... within the netCDF file, it would exist just as a Dataset)

so I'm wondering if anyone else has other suggestions or thoughts?

However, if one of the variables has the same name as one of the groups (which I think is permitted in the netCDF format), then there is no easy way to access all the elements whilst retaining the nice syntax.

NetCDF does not allow variables and groups with the same name, e..g,
import netCDF4

nc = netCDF4.Dataset('testing.nc', 'w')
nc.createVariable('foo', float)
nc.createGroup('foo')
# RuntimeError: NetCDF: String match to name in use

I'm pretty sure this is also prohibited for all HDF5 files, just like how you can't have a directory and file with the same name on most filesystems.

Oh excellent, thanks for the clarification Stephan!
On Thu, 19 Aug 2021, 00:23 Stephan Hoyer, @.***> wrote:

However, if one of the variables has the same name as one of the groups
(which I think is permitted in the netCDF format), then there is no easy
way to access all the elements whilst retaining the nice syntax.
NetCDF does not allow variables and groups with the same name, e..g,
import netCDF4
nc = netCDF4.Dataset('testing.nc', 'w')nc.createVariable('foo', float)nc.createGroup('foo')# RuntimeError: NetCDF: String match to name in use
I'm pretty sure this is also prohibited for all HDF5 files, just like how
you can't have a directory and file with the same name on most filesystems.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/4118#issuecomment-901598698, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AISNPI4WWZ3ZBJNKK4HMLWDT5SBMDANCNFSM4NQEIKFQ
.

So I had a crack at making a full DataTree class - you can find it in this repo.
It's based on @benbovy's DatasetNode example - the basic idea is that each tree node wraps a single Dataset. The differences are that this effort:

Uses a NodeMixin from anytree for the tree structure,
Implements path-like and tag-like getting and setting,
Has functions for mapping user-supplied functions over every node in the tree,
Automatically dispatches some of xarray.Dataset's API over every node in the tree (such as .isel),
Has a bunch of tests,
Has a printable representation that currently looks like this:

Some limitations of the approach I used are:

Each dataset in the tree is entirely separate, so doing something like dt.sel(time=50) would require each Dataset in that subtree to have it's own coordinate called 'time'. (That's normally useful though),
While you can access nodes via tags, the underlying implementation is in terms of paths, so ('folder1', 'folder2') points to a different node than ('folder2', 'folder1'),
There's no support for symbolic nodes yet, and I'm unsure if this design can allow for loops or not.

You can create a DataTree object in 3 ways:

Load from a netCDF file that has groups via open_datatree(),
Using the init method of DataTree, which accepts a nested dictionary of Datasets,
Manually create individual nodes with DataNode() and specify their relationships to each other, either by setting .parent and .chlldren attributes, or through __get/setitem__ access, e.g.
dt['path/to/node'] = xr.Dataset()

It's about 70% working, but some things I could do with some help with are:

Fundamental design questions about the class structure, such as whether DataTree should be a subclass of Dataset?
Getting arithmetic and ufuncs to act properly on the whole tree,
Saving a tree to a single netCDF file.

There will definitely be many bugs, but any thoughts or input appreciated!

Thanks @TomNicholas! I've just been starting to look into this. I'm going to give it a spin and would be happy to help with your numbers 3 and 4.

Thanks @jhamman - expect things to break as I keep realizing certain methods have to be defined differently from in Dataset for things to work.
Help with 3 would be especially appreciated, as at the moment I can open and alter a file with groups, but I can't save my resulting tree :sweat_smile:

Great work @TomNicholas!
For rich/html reprs, I think that we could take much inspiration from some of the dask reprs shown in this blog post.
I haven't looked at your repository in detail yet, but I have one general question about the design: what is the rationale of having two separate classes DataTree and DataNode? Could those classes be merged somehow? To me it seems that any node could be the root of a (sub)tree. h5py has a separate File object that does double duty as the HDF5 root group, and serves as entry point into the file, but in the case of Xarray maybe this is less relevant since Xarray object are abstract data containers? Zarr (abstract data store) has no such separate class and uses a regular zarr.hierarchy.Group as the root.

Thanks @benbovy !

For rich/html reprs, I think that we could take much inspiration from some of the dask reprs shown in this blog post.

I don't know much about HTML, but graphs where you can mouseover nodes to see node information sound awesome!

what is the rationale of having two separate classes DataTree and DataNode?

They aren't separate: DataNode is merely a (perhaps badly-named) pointer to a second init method for the same DataTree class.
The idea was that creating a single node of a tree by specifying only its (name, dataset, parent, children) attributes was conceptually different to "I have loads of datasets, and I want to arrange them all into one big tree using path-like addresses", so I made two different init methods on DataTree to cover that. The idea was from the xarray.Dataset._construct_direct() classmethod, which creates a new instance of a Dataset by directly setting attributes like (variables, coord_names, dims, attrs). That is an internal classmethod though, and isn't externally exposed like DataNode().
We could just merge the two signatures into one __init__ method though, or use a less confusing name (I just didn't want DataTree._init_single_node(name, data, parent) everywhere in my tests.) Also internally it's nice to have a separate ._init_single_node() method because that's (a) closer to the super().__init__() defined by TreeNode, and (b) doesn't require calling the fairly complex getting and setting methods.

Could those classes be merged somehow?

They were originally separate (I had DataTree and DatasetNode, where the former was a subclass of the latter), but then I merged them together in Condense DatasetNode and DataTree into a single DataTree class #11.

Zarr (abstract data store) has no such separate class and uses a regular zarr.hierarchy.Group as the root.

Good to know that other nested structures took a similar approach.

Thanks for launching this discussion @TomNicholas !
I'm a core dev of pytroll/satpy which handles earth observing satellite data. I got interested in DataTree because we have data from the same instruments available at mulitple resolution, hence not fitting into a single Dataset.
For use Option 1 is probably feeling better. Even when having data at  multiple resolutions, it is still a limited number of resolutions and hence splitting them in groups is the natural way of going I would say.
We do not use the features you mention in Zarr or GRIB, as a majority of the satellite data we use is provided in netcdf nowadays.
Don't hesitate to ask if you want to know more or if something is unclear, we are really interesting into these development, so if we can help that way...

@TomNicholas (cc @mraspaud)

Do you have use cases which one of these designs could handle but the other couldn't?

The two main classes of on-disk formats that, I know of, which cannot be always represented in the "group is a Dataset" approach are:

in netCDF following the CF conventions for groups, it is legal for an array to refer to a dimension or a coordinate in a different group and so arrays in the same group may have dimensions with the same name, but different size / coordinate values,
the current spec for the Next-generation file formats (NGFF) for bio-imaging has all scales of the same 5D data in the same group.

I don't have an example at hand, but my impression is that satellite products that use HDF5 file format also place arrays with inconsistent dimensions / coordinates in the same group.

One thing that came up in our discussion about this in the developer
meeting today is that we could also pretty easily expose a "low level" API
for IO using dictionaries of xarray.Variable objects. This intermediate
representation could be useful for cleaning up data into a form suitable
for conversion into Dataset objects.
On Wed, Feb 16, 2022 at 11:39 PM Alessandro Amici @.***>
wrote:

@TomNicholas https://github.com/TomNicholas (cc @mraspaud
https://github.com/mraspaud)
Do you have use cases which one of these designs could handle but the
other couldn't?
The two main classes of on-disk formats that, I know of, which cannot be
always represented in the "group is a Dataset" approach are:

in netCDF following the CF conventions for groups
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#groups,
it is legal for an array to refer to a dimension or a coordinate in a
different group and so arrays in the same group may have dimensions with
the same name, but different size / coordinate values,
the current spec for the Next-generation file formats (NGFF)
https://ngff.openmicroscopy.org for bio-imaging has all scales of
the same 5D data in the same group.

I don't have an example at hand, but my impression is that satellite
products that use HDF5 file format also place arrays with inconsistent
dimensions / coordinates in the same group.
—
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/4118#issuecomment-1042656377,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJJFVT27QD4RQDYZ2N4W7TU3SQ3BANCNFSM4NQEIKFQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

@TomNicholas I also have a few comments on the comparison:

Option (1) - Each group is a Dataset

Model maps more directly onto netCDF (though still not exactly, because netCDF has dimensions as separate objects)

This is only true for flat netCDF files, once you introduce groups in a netCDF AND accept CF conventions the DataGroup approach can map 100% of the files, while the DataTree approach fails on a (admittedly small) class of them.

Enforcing consistency between variables guarantees certain operations are always well-defined (in particular selection via an integer index like in .isel).
Guarantees that all valid operations on a Dataset are also valid operations on a single group of a DataTree - so API can be essentially identical to Dataset.

Both points are only true for the DataArray in a single group, once you broadcast any operation to subgroups the two implementations would share the same limitations (dimensions in subgroups can be inconsistent in both cases).
In my opinion the advantage for the DataTree is minimal.

Metadata (i.e. .attrs) are arguably most useful when set at this level

The two approach are identical in this respect, group attributes are mapped in the same way to DataTree and DataGroup
We agree on all other points.

@alexamici

in netCDF following the CF conventions for groups, it is legal for an array to refer to a dimension or a coordinate in a different group and so arrays in the same group may have dimensions with the same name, but different size / coordinate values, (this was the orginal motivation to explore the DataGroup approach)

I'm having difficulties to understand your above point wrt to the scoping rules from the above CF document. Shouldn't it be impossible to create two arrays (in the same group) having dimensions with exactly the same name from different groups? Looking at the example here https://github.com/alexamici/xarray-datagroup there are coordinates with name "/lat" vs "lat". Aren't that two different names? Maybe I'm missing something essential here.

@kmuehlbauer in the representation I use the fully qualified name for the dimension / coordinate, but the corresponding DataArray will use the basename, e.g. both array will have lat as a coordinate. Sorry for te confusion, I need to add more context to the README.

in the representation I use the fully qualified name for the dimension / coordinate, but the corresponding DataArray will use the basename, e.g. both array will have lat as a coordinate. Sorry for te confusion, I need to add more context to the README.

Thanks for clarifying. I'm wondering if that can be a source of misunderstanding. How should the user differentiate that? I mean finally those dimensions which have the same name lat are different entities and it should be possible to tell them apart somehow. I think I'm slowly getting to the bottom of this (representation in dictionaries, duplicate keys) and I really need to look into the implementation. I'll open an issue over at xarray-datagroup if I have more questions to not clutter the discussion here.

This is only true for flat netCDF files, once you introduce groups in a netCDF AND accept CF conventions the DataGroup approach can map 100% of the files, while the DataTree approach fails on a (admittedly small) class of them.

@alexamici can you expand on the role of the CF conventions in this statement? Are you talking about CF conventions allowing one variable in one group to refer to dimension present in another group, or something else?

I am not sure I completely understand option 2, but option 1 seems a better fit to what we are doing at ArviZ (so far we are managing quite well with the InferenceData mentioned above which is a collection of independent xarray datasets). In our case, well defined selection for multiple variables at the same time (i.e. at the dataset level) is very useful.
I was also wondering what changes (if any) would each option imply when using apply_ufunc

Often I run a function over a dataset, with each call outputing a hierarchical data structure, containing fixed dimensions in the best cases and variable length in the worst.
For this, it would make more sense to be able to have dimensions ( with optional labels and coordinates ) assigned to nodes (and these would be inherited by any descendants). Leaf nodes would hold data.
On merge, dimensions could be bubbled up as long as length (and labels) matched.
Operations with dimensions would then use the full dimension path, i.e. container.mean('A/B/time').
Datagroup and Datatree are subcases of this general structure, which could be enforced via flags/checks.
Option 1 is where the extremities of the tree are a node with dimension labels, then child/leaf nodes each containing a n-dimensional array.
Option 2 is where the extremeties of the tree are a node with dimension labels, then one single child/leaf node.
I'm sure I'm missing some big issue with the mental model I have, for instance I haven't thought of transformations at all and about coordinates. But for clarity I tried to write it down below.
The most general structure for a dataset I can think of is a directed graph.
Each node A is a n-dimensional (sparse) array, where each dimension D points optionally to a one-dimensional node B with the same length.
To get a hierarchical structure, we:

add edges of a different color, each with a label
restrict their graph to a tree T
add labels to each dimension D

We can resolve D's target by  (A) checking for a sibling in T with the same name, and then going up one level and goto (A).
Multindexes ( multi-dimensional (sparse) labels ) generalize this model, but require tuple labels in T's edges i.e. :
h/j/a[x,y,z] has a sybling h/j/(x,y)[x,y] , with z's labels being one level above, i.e. h/z[z]  ( the notation a[b] means map of index b to value a ).

Hi @LunarLanding , thanks for your ideas!

For this, it would make more sense to be able to have dimensions ( with optional labels and coordinates ) assigned to nodes (and these would be inherited by any descendants).

It sounds a bit like what you are suggesting is essentially a model in which dimensions are explicit objects, which can be referred to from other groups, like in netCDF. (NetCDF has "dimension IDs".)
This would be a bit of a departure from the model that xarray.Dataset currently uses, because right now dimensions aren't really unique entities, they are just a collective label for a shared dimension of a set of Variable objects.
Is there a specific use case which you think would require explicit dimensions to solve?

Also thanks @OriolAbril , it's useful to have an ArViz perspective.

I was also wondering what changes (if any) would each option imply when using apply_ufunc

I see apply_ufunc as a Variable-level operation - i.e. it doesn't know about the relationship between different Variables unless you explicit feed it multiple variables. So therefore whether we choose model 1 or 2 probably doesn't affect apply_ufunc much.
In either case I imagine all we might need to do is slightly extend apply_ufunc to also map over variables in a group of a tree if given one, and provide examples of using map_over_subtree or similar to map your apply_ufunc operation over multiple groups in a tree. If the user is trying to do something more complicated (like getting one variable from one level of a tree and another variable from another level, then feeding both into apply_ufunc) then I would just make the user responsible for fetching the variables in that case, and also for putting the results back into the intended place in the tree.

Hi @LunarLanding , thanks for your ideas!

For this, it would make more sense to be able to have dimensions ( with optional labels and coordinates ) assigned to nodes (and these would be inherited by any descendants).

It sounds a bit like what you are suggesting is essentially a model in which dimensions are explicit objects, which can be referred to from other groups, like in netCDF. (NetCDF has "dimension IDs".)

If I understant it correctly , in the NetCDF model all dimensions are listed at the root of the hierarchy and you can have arbirtrarily nested data variables. So NetCDF's model covers a subset of what I am describing.

Often I run a function over a dataset, with each call outputing a hierarchical data structure, containing fixed dimensions in the best cases and variable length in the worst.

By "variable" length, do you mean that the length of dimensions differs between variables in the same group, or just that you don't know the length of the dimension in advance?

I mean that I might have, for instance, a map from 2 variables to data, ie (x,y)->c, that I can write as a DataArray XY with two dimensions x and y and the values being c.
Then I have a function f so that f(c)->d[g(c)], i.e. it yields an array whose length depends on c.
I wish I could say : apply f to XY, building a variable length array as you get the output. It could be stored as sparse matrice (X,Y,G).
This is a bit out of scope for this discussion; but it is related since creating a differently named group per dimension length is often mentioned as a workaround ( which does not scale when you have a 1000x(variable length dimension) data).

Is there a specific use case which you think would require explicit dimensions to solve?

Thanks for the question; it made me really think about how I am using xarray.
The TL;DR is that explicit hierarchical dimensions allow for merging and referring to data with no collisions in a single datatree/group.
The use-case is iteratively adding values to a dataset by mapping functions over multiple variables / dimensions in arbitrary compositions; this then requires being able to deal with name clashes between function outputs that cannot be merged when they have different meaning, and the easiest solution is to stash each function output under a different tree branch.
This happens in the context of data analysis, where you start with some source data and then iteratively create analysis functions, and then want to query / display / do statistics/reductions on the set of original data + analysis.
The datastructure that is more useful for this kind of analysis is the one that is an arbitrary graph of n-dimensional arrays; forcing the graph to have a hierarchical access allows optional organization; the graph itself can exist as python objects for nodes and references for edges.
If the tree is not necessary/required everything can be placed on the first level, as it is done on a Dataset.
Example:
Notation

a:b value a has type b
t[...,n,...] : type of data array of values of type t, with axis of length n
D(n(,l)) dimension of size n with optional labels l
A(t,*(dims:tuple[D])} : type of data array of values of type t, with dimension dims
a tree node T is either:

a dict from hashables to tree nodes, dict[Hashable,T]
a dimension D
a data array A

a[*tags]:=a[tag[0]][tag[1]]...[tag[len(tag)-1]]
map(f,*args:A,dims:tuple[D]) maps f over args broadcasting over dims

Start with a 2d-dimensional DataArray:
 d0
 (
    Graph : (
        x->D(x_n,float[x_n])
        y->D(y_n)
        v->A(float,x,y)
    )
    Tree : (
        {
            'x':x,
            'y':y,
            'v':v,
        }
    )
 )

Map a function f that introduces a new dimension w with constant labels f_w_l:int[f_w_n] (through map_blocks or apply_ufunc) and add it to d0:
 f : x:float->(
    Graph:
        f_w->D(f_w_n,f_w_l)
        a->A(float,f_w)
        b->A(float)
    Tree:
        {
         'w':f_w,
         'a':a,
         'b':b,
        })

 d1=d0.copy()
 d1['f']=map(
        f,
        d0['v'],
        (d0['x'],d0['y'])
    )
 d1
  (
    Graph :
        x->D(x_n,float[x_n])
        y->D(y_n)
        v->A(float,x,y)
        f_w->D(f_w_n,f_w_l)
        f_a->A(float,x,y,f_w)
        f_b->A(float,x,y)
    Tree :
        {
            'x':x,
            'y':y,
            'v':v,
            'f':{
                'w':f_w,
                'a':f_a,
                'b':f_b,
            }
        }
 )

Map a function g, that has a dimension of the same name but different meaning and therefore possibly different length g_w_n and g_w_l:
g : x:float->(
    Graph:
        g_w->D(g_w_n,g_w_l)
        a->A(float,g_w)
        b->A(float)
    Tree:
        {
         'w':g_w,
         'a':a,
         'b':b,
        })

 d2=d1.copy()
 d2['g']=map(
        g,
        d1['v'],
        (d1['x'],d1['y'])
    )
 d2
  (
    Graph :
        x->D(x_n,float[x_n])
        y->D(y_n)
        v->A(float,x,y)
        f_w->D(f_w_n,f_w_l)
        f_a->A(float,x,y,f_w)
        f_b->A(float,x,y)
        g_w->D(g_w_n,g_w_l)
        g_a->A(float,x,y,g_w)
        g_b->A(float,x,y)

    Tree :
        {
            'x':x,
            'y':y,
            'v':v,
            'f':{
                'w':f_w,
                'a':f_a,
                'b':f_b,
            },
            'g':{
                'w':g_w,
                'a':g_a,
                'b':g_b,
            }

        }
 )

Notice that both f and g output a dimension named 'w' but that they have different lengths and possibly different meanings.
Suppose I now want to run analysis on f's and g's output, with a function that takes two a's and outputs a float
Then d3 looks like:
h : a1:float,a2:float->(
    Graph:
        r->A(float)
    Tree:
        r

 d3=d2.copy()
 d3['f_g_aa']=map(
        h,
        d2['f','a'],d2['g','a'],
        (d2['x'],d2['y'],d2['f','w'],d2['g','w'])
    )
 d3
 {
    Graph :
        x->D(x_n,float[x_n])
        y->D(y_n)
        v->A(float,x,y)
        f_w->D(f_w_n,f_w_l)
        f_a->A(float,x,y,f_w)
        f_b->A(float,x,y)
        g_w->D(g_w_n,g_w_l)
        g_a->A(float,x,y,g_w)
        g_b->A(float,x,y)
        f_g_aa->A(float,x,y,f_w,g_w)

    Tree :
        {
            'x':x,
            'y':y,
            'v':v,
            'f':{
                'w':f_w,
                'a':f_a,
                'b':f_b,
            },
            'g':{
                'w':g_w,
                'a':g_a,
                'b':g_b,
            }
            'f_g_aa': f_g_aa
        }
 }

@LunarLanding You may also be interested in awkward array.

Wanted to note issue ( https://github.com/carbonplan/ndpyramid/issues/10 ) here, which may be of interest to people here.
Also we are thinking about a Dask blogpost in this space if people have thoughts on what we should include and/or are interested in being involved. Details in issue ( https://github.com/dask/dask-blog/issues/141 ).

Over 4 years later, closed by v2024.10.0 - see the announcement discussion.
Thanks everyone - and especially to @emilbiju for your very generative original ideas here.

Huge thanks to everyone involved in making this happen! 🚀


[SCB-1007] not support CustomGeneric<Map<String,String>>

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.1%) to 86.647% when pulling ecd04e2f61b4663fc0945765b4772289625202a6 on heyile:generic_support into b6b5855910daa64be64accb69b90d5fbe868d348 on apache:master.


RC Spot Check: Molecules and Light 1.5.6-rc.2

@kathy-phet @ariel-phet @arouinfar, Molecules and Light 1.5.6-rc.1 is ready for an RC spot check. This release is for https://github.com/phetsims/special-ops/issues/175. The changes have been picked into the existing 1.5 branch of molecules-and-light, so a full RC test should not be necessary.
A modified version of the sim can be accessed with query parameter ?openSciEd, which removes the wavelength radio buttons, several molecules, and the "Light Spectrum" dialog, and adds a label to the single energy source.
QA team, can you please spot check this version, making sure that the normal version of the sim was not impacted. Then, can you please do the same with the modified version?
Please note that it was decided not to modify descriptions for these modifications. We expect references in the description to be out of sync. Also, please do not test with iOS VoiceOver for the moment. https://github.com/phetsims/scenery/issues/1094 is still an issue making iOS VoiceOver problematic. That should go out in a a separate maintenance release.
Link to sim: https://phet-dev.colorado.edu/html/molecules-and-light/1.5.6-rc.2/phet/molecules-and-light_en_phet.html
Link to Open Sci Ed version: https://phet-dev.colorado.edu/html/molecules-and-light/1.5.6-rc.2/phet/molecules-and-light_en_phet.html?openSciEd

[ ] Win 10 Chrome
[ ] Win 10 Firefox
[ ] Mac 11 Safari
[ ] iPadOS 14
[ ] ChromeOS

@kathy-phet mentioned that the label text in the openSciEd version was a little off center, I fixed it and updated the link in this issue.

@KatieWoe @ariel-phet @jessegreenberg - I moved this out of QA because I had shared this version with the client for a quick check, and they said they put it in front of teachers, and want to change the label from "Energy Source" to TBD. So might as well hold off on QA until they decide.

Closing, a new version will be built with a new issue after https://github.com/phetsims/QA/issues/584#issuecomment-747826414


[cpu] Add runtime library for CPU kernels

This will make it simpler to implement device_assert, device_print etc.
I am putting this up so we can build upon it in parallel (I am working on device_assert and @minjang on device_print).

The patch is missing the runtime itself.

oops, fixed.

ah sorry, I thought the CI tests were consistently timing out
Let me look into that

Fix: https://github.com/triton-lang/triton-cpu/pull/75


Error during  diode_insertion can't find opendb python module

Hi,
This is needed by: ./scripts/write_powered_def.py
Not sure where is this is suppose to come from, didn't see it on OpenROAD repo. please advise.
Thanks!
Ronald

I think is here now: src/odb/src/swig/python
It seems it has change its name as well....

@20Mhz also mention which version of openlane you are using?  Install latest openlane version and try it. still error repeats share source file along with steps to reproduce the issue

We can’t help you if you just nuke the issue template.

OpenLane f7fd774a762629bc56d31a94ff03138fbc2e176c
OpenROAD 527cd5d8e82308baa1a46265c1338e6c1757858f
It seems the swig module has changed name, but it also broken backwards compatibility.
Related to: https://github.com/The-OpenROAD-Project/OpenROAD/issues/1028

https://gist.github.com/20Mhz/be34125a6747a559b4da7f073043ee9f

Ronald, that's not a workaround, that's a fix! If you can open a PR, I'd be happy to approve it.

Incorporated as part of #609, thanks


Fix popout location

Changes

Modified adjustWindowLocation private method of InternalChildWindowManager to not use window.top properties.
Removed dead code from the private function that was never called.
resolves #379
Removed a contain in appui-layout-react that was unexpectedly trying to contain popped out wigets, now the widget will reappear where they were last closed by the user.
resolves #381
Using pagehide event handler instead of onbeforeunload for firefox and overall posible better results.
resolves #380
Testing

Visual/Manual testing through test-apps/standalone (web)

@jason-crow @simnorm This fixes a couple of initial issues with iFramed AppUI attempting to use popouts, if the iFrame is sandboxed, it will still require the allow-popups rule but that should allow you guys to test.
Be aware that there are still issues so this should probably not used in prod yet, see #364 for a list, but mainly #384 would probably be a show stopper.


Conflict with Doorkeeper

I've used css-zero happily until installing the Doorkeeper gem into my Rails app. Doorkeeper uses Sprockets to add Bootstrap and some of their own custom CSS to the application's assets pipeline, but, unfortunately, some of this CSS (I'm not sure if Bootstrap or their custom CSS) conflicts with css-zero's and makes the rendered pages look horrible. Any suggestions for ameliorating this issue? I'm running a Rails 8 app that uses Propshaft.

It's the behavior with Propshaft when you use <%= stylesheet_link_tag :all, "data-turbo-track": "reload" %>.
You can import the files individually instead and only import doorkeeper when it's needed:
Replace:
<%= stylesheet_link_tag :all, "data-turbo-track": "reload" %>

With:
<%= stylesheet_link_tag "_reset", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "animations", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "borders", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "colors", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "effects", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "filters", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "sizes", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "transform", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "transition", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "typography", "data-turbo-track": "reload" %>
<%= stylesheet_link_tag :app, "data-turbo-track": "reload" %>
<%= stylesheet_link_tag "zutilities", "data-turbo-track": "reload" %>

ps: Hey why are you using doorkeeper when you have [authentication-zero]https://github.com/lazaronixon/authentication-zero) 😝

Hey thanks for the quick reply! Just loading each CSS file with your suggestion in the first comment worked. Thank you.
Re: not using authentication-zero, I would if it helped with making my server into an OAuth2 and OIDC provider, which is why I'm using Doorkeeper. For this project, I used Rails 8's authentication generator as a base and then implemented an Ethereum wallet authentication flow and a passkey authentication flow on top.


PostFilteringWrapper  with filter on threadid

Is it possible to use a PostFilteringWrapper in combination with a filter on the threadid equals threadid from the exception?
I want to create an email target that will send an email when an exception is logged and it should contain the buffered log messages from all levels from the same threadid

@Xharze do you know this?

This should be possible. I think you need the Event properties because the exception doesn't have the property "threadid"
In the condition you can use both ${threadid} and ${event-properties}
But I'm not sure, aren't those always the same? An if they aren't, I think you will lose messages.

@mdissel Has you question been answered?

We will assume your question has been answered. If not, let us know.


EurekaBase potential for NullReferenceException when no instance id exists

Whilst testing the Eureka Discovery client found that applications were not being returned. Investigation showed that the Eureka app data was missing an instance id for one of the application elements causing a NullReferenceException within Applications.cs, method AddInstanceToVip on the following line (currently line number 147)
instances[info.InstanceId] = info;

@putridparrot Sorry it took so long to get around to looking at this... Do you have any idea what the circumstances were around the InstanceId being null?

After some investigation, as near as I can tell this issue should be prevented by https://github.com/SteeltoeOSS/Discovery/pull/54/files so I'm going to close this issue now. Please reopen if it persists

Sorry for the delay in getting back to you. The issue was spotted on a client's machine, I will try to recreate when I get a chance and see if I can give you better info.


Error: Can't find Python executable

Yes I did install everything in that requires.
λ npm install discord-spoiler-bot --save

<EMAIL_ADDRESS>install C:\Users\hp\Desktop\Bots\test\node_modules\canvas
node-gyp rebuild

C:\Users\hp\Desktop\Bots\test\node_modules\canvas>if not defined npm_config_node_gyp (node "C:\Users\hp\AppData\Roami
e_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "C:\Users\hp\AppData\Local\Programs\Python\Python35-32\python.EXE"
gyp ERR! stack     at PythonFinder.failNoPython (C:\Users\hp\AppData\Roaming\npm\node_modules\npm\node_modules\node-g
gyp ERR! stack     at PythonFinder. (C:\Users\hp\AppData\Roaming\npm\node_modules\npm\node_modules\node-gy
gyp ERR! stack     at C:\Users\hp\AppData\Roaming\npm\node_modules\npm\node_modules\graceful-fs\polyfills.js:284:29
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:152:21)
gyp ERR! System Windows_NT 10.0.15063
gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\hp\AppData\Roaming\npm\node_modules\npm\node
gyp ERR! cwd C:\Users\hp\Desktop\Bots\test\node_modules\canvas
gyp ERR! node -v v8.4.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of erlpack@hammerandchisel/erlpack but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>No description
npm WARN<EMAIL_ADDRESS>No repository field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\hp\AppData\Roaming\npm-cache_logs\2017-09-02T08_37_52_575Z-debug.log

Looks like your Python installation is broken, try to reinstall Python completely.
Alternatively, you could do a quick check to make sure Python is added to your PATH by running this in command line (Windows Key + R -> type in cmd -> Enter):
python -V

Assuming this was resolved, closing the issue.


Replacevectorwith buffer

What type of PR is this：

[ ] bug
[ ] feature
[X ] enhancement
[ ] others

Which issues of this PR fixes ：

Fixes #2594
Problem Summary(Required) ：

The changes I have made are going through the column files and wherever I saw std::vector, I replaced it with Buffer.

we don't have to apply change in these files:
be/src/column/datum_tuple.h
be/src/column/datum_tuple.cpp
be/src/column/schema.cpp
be/src/column/schema.h

we need format some code in these files:
--- /home/disk1/jenkins/workspace/starrocks_clang-format/starrocks/be/src/column/column.h
+++ /home/disk1/jenkins/workspace/starrocks_clang-format/starrocks/be/src/column/column.h (after clang format)
@@ -162,16 +162,12 @@
     // Like append_strings. To achieve higher performance, this function will read 16 bytes out of
     // bounds. So the caller must make sure that no invalid address access exception occurs for
     // out-of-bounds reads
-    [[nodiscard]] virtual bool append_strings_overflow(const Buffer<Slice>& strs, size_t max_length) {
-        return false;
-    }
+    [[nodiscard]] virtual bool append_strings_overflow(const Buffer<Slice>& strs, size_t max_length) { return false; }

     // Like `append_strings` but the corresponding storage of each slice is adjacent to the
     // next one's, the implementation can take advantage of this feature, e.g, copy the whole
     // memory at once.
-    [[nodiscard]] virtual bool append_continuous_strings(const Buffer<Slice>& strs) {
-        return append_strings(strs);
-    }
+    [[nodiscard]] virtual bool append_continuous_strings(const Buffer<Slice>& strs) { return append_strings(strs); }

     // Copy |length| bytes from |buff| into this column and cast them as integers.
     // The count of copied integers depends on |length| and the size of column value:

Is there an idea of how long the checks take?


End-to-End Network layer

DRAFT: there is an outstanding bug with the instantiation of PRSS that needs to be fixed.
Creates the HttpHelper that holds necessary global state for the long-running MPC instance that can communicate over HTTP. Exposes functionality for creating and executing a query

reopened against main repo: https://github.com/private-attribution/ipa/pull/288


upgrade snakeyaml due to CVE

BUG REPORT
snakeyaml 1.19 has a serious issue - https://mvnrepository.com/artifact/org.yaml/snakeyaml - an entity expansion vulnerability

@pjfanning thanks for your report,
@dongeforever we need upgrade it.

I submitted https://github.com/apache/rocketmq/pull/3815


chore: add ci.yml

Fixes #13.
It works on my end, cf. vil02/Fortran#1, vil02/Fortran#2 (please have a look to the action runs).
The CMakeLists.txt is not perfect, but I think it is good enough for now:

scans the modules, tests and example directories for *.f90 files,
all of the tests and examples are linked with all of the modules,
the tests and the examples, have pretty unique name: even if there would be two files with the same name in tests/sorts and tests/some_other_algorithms this would not cause any problem.

When this will be merged, there are few things on the to do list:

~make of all of the files in the other directory proper modules and add tests for them,~
fix the warnings (currently there are few unused variables),
add the -Werror flag,
generate coverage reports.

@SatinWukerORIG: Is there anything left to be done with this PR?


Tab for Spanish Speakers

can we make a tab to include basic clinic information for spanish speakers?

Seconded!
Sent from my iPhone

On May 19, 2015, at 11:49 AM, Liz Maidl<EMAIL_ADDRESS>wrote:
can we make a tab to include basic clinic information for spanish speakers?
—
Reply to this email directly or view it on GitHub.

Yeah, this could be done quite easily. @lizmaidl if you look at the first commit in PR #31 , you can see how I added a new tab to the "services" menu bar item. You already know how to do the rest! (Except the Spanish, which I'm no better with than you!)


Documentation / interface for hardware wallets

The problem that you wish to solve
Right now Daedalus somehow magically is able to connect HW wallets, but it's not obvious from API of cardano-wallet how is that possible. Is there any interface in cardano-wallet that can handle hardware wallets? If not it should be implemented.
Description
See above
Implementation suggestions
Implement API interface for handling hardware wallets, eg. let users export the public keys and create wallet just using that instead of secret passphrase. Let user export prepared TXs and let HW wallet sign them, then send them back to cardano-wallet for execution.
It would be great if format of output was compatible with existing CLI tools that can work with hardware wallets like cardano-hw-cli

The cardano-wallet backend application doesn't have a functionality to support HW wallets like Ledger. The HW support in Daedalus comes from somewhere else.

The HW support in Daedalus comes from somewhere else.

@Unisay that cannot be the whole story — ledgerjs-hw-app-cardano may be able to sign transactions, but it does not contain any provisions for interacting with the cardano-node. For the latter, Daedalus does rely on cardano-wallet.

Implement API interface for handling hardware wallets, eg. let users export the public keys and create wallet just using that instead of secret passphrase.

The API endpoint https://cardano-foundation.github.io/cardano-wallet/api/edge/#operation/postWallet with the shelley (from XPub) schema for the POST data allows you to do that — restore (synchronize) a wallet from a public key instead of a private key.

ok, can we at least create some wiki page that would show example flow where user extracts the public key (using tools like cardano-hw-cli), then create a wallet with that key, then creates and exports a TX using cardano-wallet, signs it using cardano-hw-cli and then sends it back to cardano-wallet for submission?
Because while in theory this is already possible, I still have no clue how to do that in practice. Some examples would make this more obvious to people.

We are planning a feature for interacting with hardware wallets for the Deposit Wallet.


Update akka-actor to 2.5.21

Updates com.typesafe.akka:akka-actor from 2.3.16 to 2.5.21.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Nope.

@travisbrown If you'd like to ignore updates of akka-actor in this repo, you could add a case here:
case ("circe/circe-spray", "com.typesafe.akka", "akka-actor") => false

This is a workaround until scala-steward supports repo-provided configuration via a .scala-steward.conf file or something similar.

Ah, thanks, @fthomas!

scala-steward now supports ignoring updates via a .scala-steward.conf file in the root directory of a repo. If you add one with
updates.ignore = [{ groupId = "com.typesafe.akka", artifactId = "akka-actor" }]

scala-steward would ignore any future updates of akka-actor. More information about this feature can be found here: https://github.com/fthomas/scala-steward/pull/301


Fixed #31897 -- Tweaked Settings docs to better explain settings.DEBUG example

Tweaked Settings docs to better explain settings.DEBUG example
The current example for how to use from django.conf import settings includes the example of using settings.DEBUG.
The example is fine, but if a user tries to replicate this in their test scripts settings.DEBUG will always return False as intended. This PR simply adds an admonition in the docs to warn users about this and offer an alternative with override_settings().

Closing per ticket.


Allow maxCookieAge to be specified

Currently fflip uses a maxAge value when setting the cookie of 900000, or 15 minutes.
This PR keeps the current default, but allows the user to override the value.
For example, we would prefer a value of one month. This PR will allow us to override the value.

Looks like the build broke due to modules being no longer supported in node 0.8.x. Would you mind if I removed 0.8 from the .travis.yml file?

Hey @suprememoocow, thanks for the PR! Sorry for the long wait, December was a busy month.
Just pushed https://github.com/FredKSchott/fflip/pull/9 to fix that travis issue, I think dropping support for v0.8 is a fine compromise for continuing development. I'll let you know when that gets merged and you can rebase onto master.

#9 was just merged, please rebase off of master to get tests passing and then it's good to merge!
Also if you have time to answer, I'd love to know how you're using this library. Personal, company, in production, etc.

Hi @FredKSchott,

Also if you have time to answer, I'd love to know how you're using this library. Personal, company, in production, etc.

We're using it for Gitter. We've even got a little UI where people can opt-in or -out of features (see http://next.gitter.im).
Since I submitted this PR, we've actually made some more changes to the way we use FFlip. See https://github.com/suprememoocow/fflip/pull/1. I haven't created a PR for it yet, but please let me know if it's something you would find useful in fflip.
Basically, we wanted to be able to have a bunch of criteria interacting with one another, for example:

Enable this feature for users a, b, c (for eg, let's call it usernameCriteria)
Enable this feature for 10% of users (percentageUsersCriteria)
Disable this feature in IE10 and lower (disabledBrowsersCriteria)

Unfortunately, the way criteria are currently specified, this wasn't possible, so I added a new feature, which I call veto voting (for want of a better name)
Each criteria can return one of three values:

true if the criteria votes for the feature toggle to being enabled
false if the criteria vetoes the feature toggle from being enabled
undefined/null if the criteria abstains from voting

A criteria will then be enabled when it obtains at least one vote (true) from the present criteria without receiving any vetoes (false).
This way, the criteria in the example above might interact as follows:

For user "a" running Chrome, the votes will be

usernameCriteria -> true (vote)
percentageUsersCriteria -> undefined (abstain)
disabledBrowsersCriteria -> undefined (abstain)
Therefore, the feature will be enabled.

For user "a" running IE10, the votes will be

usernameCriteria -> true (vote)
percentageUsersCriteria -> undefined (abstain)
disabledBrowsersCriteria -> false (veto)
Therefore the feature will be disabled

For user "x" (who matches percentageUsersCriteria) running Chrome, the votes will be

usernameCriteria -> undefined (abstain)
percentageUsersCriteria -> true (vote)
disabledBrowsersCriteria -> undefined (abstain)
Therefore the feature will be enabled

For user "y" (who does not match percentageUsersCriteria) running Chrome, the votes will be

usernameCriteria -> undefined (abstain)
percentageUsersCriteria -> undefined (abstain)
disabledBrowsersCriteria -> undefined (abstain)
Therefore the feature will be disabled

This might seem a little bit complicated, but it gives us the expressive power of having multiple interacting criteria. Since it's enabled by setting useVetoVoting:true in the config, it's also backwards compatible.
Is this something you would be interested in? Would you like me to create a PR for it?

@FredKSchott just rebased from upstream, so your changes are now in

@suprememoocow so happy to hear that Gitter is using this! We used Gitter on the request repo and always saw a lot of good questions come in through there.
I Love the veto feature! That is definitely a use case that would extend outside of just Gitter. If we added the ability to define whether on/off takes precedence in the face of conflict (it sounds like "off"/veto always takes precedence in your situation) that could make it even more flexible.
If you're able I'd appreciate any PR you can get together to add that feature into master. I'm happy to work on the documentation as well if that's something standing in the way.
Thanks again for the contribution, merging now :clap: :clap: :clap:

Seems like something wrong with my npm config, I'll get this pushed up to npm tomorrow


Prevent context menu on canvas

If clicking with right mouse button on canvas, the browser context menu appears. This should be prevented.

Solved with #38


Make repo URLs include repo names

Clear and concise description of the problem
Repos have URLs such as /repos/1 which doesn't really say anything about which repo it is. So when I type our woodpecker URL into the address bar I'm forced to go to the root and click myself to the right repo.
Suggested solution
If the URL was instead /repos/1-myrepositoryname (and possibly the non-URLified name was part of the title, e.g. WoodPecker: MyRepositoryName), I could just type pecker MyRepositoryName into the URL bar and it would pop up as a suggestion.
Alternative
No response
Additional context
No response
Validations

[X] Checked that the feature isn't part of the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

This was changed on purpose in #1691 to support features like gitlabs nested plugins and as you could have multiple repos with the same name when using multiple forges #138


Fix #3962 - Profiler uses DatabaseSchema

Describe your changes :

Fix #3962
Also fixes:

invalid options casting in engine creation
looker config in tests
sample data airlfow

Type of change :

[x] Bug fix
[x] Improvement
[x] New feature
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] Documentation

Frontend Preview (Screenshots) :
For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] I have added tests that prove my fix is effective or that my feature works.
[x] All new and existing tests passed.

Reviewers

Fixing sample data ingestion


Added trusted proxy to init middleware

Coverage increased (+0.4%) to 55.664% when pulling 519712089db73b89a7765f4345fef1aedcc61b2f on added_trusted_proxy_to_init_middleware into 3871ff534b5b1564a987cc7faaf8cd186f74154d on master.


LLVM14 linux-aarch64 blocker

This issue is to document a blocker for using LLVM14 on linux-aarch64.
LLVM14 on Linux-aarch64 seems to have a bug (in probably MCJIT+RuntimeDyLd) that affect fails to load exporting symbols and/or setting up GOT info.
Symptom:
The following tests fails:
numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_nested_list
numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_inferred_list_jit
numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_inferred_list
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_typed_dict_njit
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_typed_dict_from_cpython
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_inferred_dict_njit
numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest.test_unbalanced_example
numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest.test_chunksize_with
numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest.test_ex_initial_value_dict_compile_time_consts
numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest.test_ex_jitclass_type_hints

They all segfault when executing JITed callback functions that are invoked by externally compiled functions. Most of the cases can be workaround by making the callback functions have internal linkage. However, ChunksizeExamplesTest failures cannot use this workaround due to requirement of how ufunc works in Numba.
A minimal reproducer for the ChunksizeExamplesTest problems is provided in this gist. Observations in the reproducer:

the bug can somehow be avoided by doing O0 on the problematic JIT function.
saving the problematic LLVM IR to file and running it in a fresh process works without problem.

I don't know what to make off the first observation. The second observation seems to indicate the problem only occurs when multiple LLVM modules are loaded into MCJIT.
The above tests all works in LLVM11.
Related issues:

https://github.com/numba/numba/issues/8567 but this time is in linux only; OSX is fine.
https://github.com/JuliaLang/julia/issues/42295

Future fixes:

https://github.com/JuliaLang/julia/pull/43664#issuecomment-1025304316 answers if OrcJIT+JITLink does not support aarch64-elf as of early 2022.
GSoC 2022 has work that add support for aarch64-elf to jitlink: https://blog.llvm.org/posts/2022-12-21-compiler-research-internships/

I'nm going to spend a little time looking at this - I was able to reproduce this on a Jetson AGX Xavier with test_ex_nested_list so far.

A simpler reproducer:
from numba import njit
from numba.typed import List

mylist = List()
l = List()
l.append(0)
mylist.append(l)

Backtrace:
#0  0x0028000852800028 in ?? ()
#1  0x0000ffffe0e518b0 in list_incref_item (lp=0xaaaaaba7b900, item=0xaaaaabb94ac0 "") at numba/cext/listobject.c:101
#2  0x0000ffffe0e51d24 in numba_list_append (lp=0xaaaaaba7b900, item=0xffffffffe5f0 "") at numba/cext/listobject.c:345
#3  0x0000ffffe7f5004c in numba::typed::typedlist::_append[abi:v7][abi:c8tJTIcFHzwl2ILiXkcBV0KBSgP9CGZpAgA_3d](ListType_5bListType_5bint64_5d_5d, ListType_5bint64_5d) ()
#4  0x0000ffffe7f502b4 in cpython::numba::typed::typedlist::_append[abi:v7][abi:c8tJTIcFHzwl2ILiXkcBV0KBSgP9CGZpAgA_3d](ListType_5bListType_5bint64_5d_5d, ListType_5bint64_5d) ()
#5  0x0000ffffe0df3324 in call_cfunc (self=0xffffe0665080, cfunc=0xffffdfc46b10, args=0xffffe088e540, kws=0x0, locals=0x0) at numba/_dispatcher.cpp:638
#6  0x0000ffffe0df3404 in compile_and_invoke (self=0xffffe0665080, args=0xffffe088e540, kws=0x0, locals=0x0) at numba/_dispatcher.cpp:662
#7  0x0000ffffe0df3dc4 in Dispatcher_call (self=0xffffe0665080, args=0xffffe088e540, kws=0x0) at numba/_dispatcher.cpp:934

The errant address, 0x0028000852800028, is stable across runs (at the moment! :slightly_smiling_face: )

The second time we call list_incref_item, just prior to the segfault, the item_incref function has the invalid address:
(gdb) p *lp
$2 = {size = 1, item_size = 16, allocated = 4, is_mutable = 1, methods = {item_incref = 0xf828000852800028, item_decref = 0xd65f03c091000500},
  items = 0xaaaaabbd0e40 "Pi\246\253\252\252"}

This has been set using values from typedobjectutils._get_incref_decref called from listobject._list_set_method_table, called from listobject.impl_new_list.
This then causes the segfault:
│   93          /* Increment a reference to an item in a list.                                                                                │
│   94           *                                                                                │
│   95           * lp: a list                                                                                │
│   96           * item: the item to increment the reference for                                                                                │
│   97           */                                                                                │
│   98          static void                                                                                │
│   99          list_incref_item(NB_List *lp, const char *item){                                                                                │
│B+>100             if (lp->methods.item_incref) {                                                                                │
│   101                 lp->methods.item_incref(item);                                                                                │
│   102             }                                                                                │
│   103         }

Assembly of the _get_incref_decref-generated code and the eventual use of its output dumped with NUMBA_DUMP_ASSEMBLY=1 and taken from impl_new_list.<locals>.imp:
.LBB0_4:
        ldr     x21, [sp, #24]
        adrp    x8, :got:.numba__ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v6B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx.list_incref
        ldr     x8, [x8, :got_lo12:.numba__ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v6B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx.list_incref]
        adrp    x9, :got:.numba__ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v6B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx.list_decref
        ldr     x9, [x9, :got_lo12:.numba__ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v6B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx.list_decref]
        stp     x8, x9, [sp, #8]
        add     x1, sp, #8
        mov     x0, x21
        bl      numba_list_set_method_table

There is also another variant of the call to numba_list_set_method_table, but this just sets the methods to null pointers (I think this is for the list containing ints rather than the list of lists):
.LBB0_4:
        ldr     x21, [sp, #24]
        stp     xzr, xzr, [sp, #8]
        add     x1, sp, #8
        mov     x0, x21
        bl      numba_list_set_method_table

Looks like :got_lo12:.numba__ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v6B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx.list_incref (and decref) have somehow resolved to 0xfffff7faa000 and 0xfffff7faa008:
ldr     x21, [sp, #24]
adrp    x8, 0xfffff7faa000 <nrt_atomic_add>
ldr     x8, [x8]
adrp    x9, 0xfffff7faa000 <nrt_atomic_add>
ldr     x9, [x9, #8]
stp     x8, x9, [sp, #8]
add     x1, sp, #0x8
mov     x0, x21
bl      0xffffe7f5438c ; numba_list_set_method_table

The wrong value 0xf828000852800028 (for the incref function is therefore getting loaded from those addresses, so it's just loading bytes of the nrt_atomic_add function:
(gdb) x/4x nrt_atomic_add
0xfffff7faa000 <nrt_atomic_add>:        0x52800028      0xf8280008      0x91000500      0xd65f03c0

A small observation: nrt_atomic_add is the very first function that was added - output from the Numba assembly dump:
;================================================================================
;----------------------------------ASSEMBLY nrt----------------------------------
        .text
        .file   "<string>"
        .globl  nrt_atomic_add
        .p2align        2
        .type   nrt_atomic_add,@function
nrt_atomic_add:
        mov     w8, #1
        ldadd   x8, x8, [x0]
        add     x0, x8, #1
        ret
.Lfunc_end0:
        .size   nrt_atomic_add, .Lfunc_end0-nrt_atomic_add

...

The relocations for incref and decref seem to be treated as if they are local relocations - they appear in RuntimeDyld::Relocations. At:
│   147         void RuntimeDyldImpl::resolveLocalRelocations() {
│   148           // Iterate over all outstanding relocations
│   149           for (const auto &Rel : Relocations) {
│   150             // The Section here (Sections[i]) refers to the section in which the
│   151             // symbol for the relocation is located.  The SectionID in the relocation
│   152             // entry provides the section to which the relocation will be applied.
│   153             unsigned Idx = Rel.first;
│   154             uint64_t Addr = getSectionLoadAddress(Idx);
│  >155             LLVM_DEBUG(dbgs() << "Resolving relocations Section #" << Idx << "\t"
│   156                               << format("%p", (uintptr_t)Addr) << "\n");
│   157             resolveRelocationList(Rel.second, Addr);

The first relocation is the address from above:
(gdb) p/x Addr
$8 = 0xfffff7faa000

Notes on GOT optimization, which may be involved here: http://maskray.me/blog/2021-08-29-all-about-global-offset-table#got-optimization

Perhaps this optimization should have been applied at the call site: https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#relocation-optimization

Perhaps this optimization should have been applied at the call site: https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#relocation-optimization

(Or there should still be a GOT entry...)

Files / notes I'm using whilst looking into this: https://github.com/gmarkall/numba-issue-8738

An llvmlite-only reproducer of the issue is at https://github.com/gmarkall/numba-issue-8738/blob/main/llvmlite_only.py
It loads the 8 object files produced by Numba into an Execution Engine then prints out the address of the function that has relocations incorrectly applied, so that it can be inspected with gdb. Though, it turns out GDB can see the symbols in the loaded objects so it's not strictly necessary - one can look at the output, e.g.:
(numballvm14) gmarkall@gm-agx:~/numbadev/issues/8738$ python llvmlite_only.py
Function is at 0xffffa9c57000

and do
(gdb) disass 0xffffa9c57000

or by name:
(gdb) disass _ZN5numba5typed10listobject13impl_new_list12_3clocals_3e3impB2v4B42c8tJTIcFHzwl2ILiXkcBV0KBSmNGHkyiCKJEEwA_3dE32typeref_5bListType_5bint64_5d_5dx

to see that the relocations are incorrect:
   0x0000ffffa9c57084 <+132>:	adrp	x8, 0xffffa9c58000 ; incorrect
   0x0000ffffa9c57088 <+136>:	ldr	x8, [x8, #64]
   0x0000ffffa9c5708c <+140>:	adrp	x9, 0xffffa9c58000 ; incorrect
   0x0000ffffa9c57090 <+144>:	ldr	x9, [x9, #72]
   0x0000ffffa9c57094 <+148>:	stp	x8, x9, [sp, #8]
   0x0000ffffa9c57098 <+152>:	add	x1, sp, #0x8
   0x0000ffffa9c5709c <+156>:	mov	x0, x21
   0x0000ffffa9c570a0 <+160>:	bl	0xffffa9c57238

Note how the bl target is correct, and points at something looking like a GOT entry:
(gdb) x/8i 0xffffa9c57238
   0xffffa9c57238:	movz	x16, #0x0, lsl #48
   0xffffa9c5723c:	movk	x16, #0x0, lsl #32
   0xffffa9c57240:	movk	x16, #0x0, lsl #16
   0xffffa9c57244:	movk	x16, #0x0
   0xffffa9c57248:	br	x16
   0xffffa9c5724c:	movz	x16, #0x0, lsl #48
   0xffffa9c57250:	movk	x16, #0x0, lsl #32
   0xffffa9c57254:	movk	x16, #0x0, lsl #16

but that relocation was to a function in another object, numba_list_set_method_table.

Here's a reproducer that only uses a little bit of C to generate the code - llvmlite is only used as a convenient way to load the resulting objects into RuntimeDyld: https://github.com/gmarkall/numba-issue-8738/tree/main/c

With this patch applied to LLVM 14.0.6:
diff --git a/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp b/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
index 3c7f4ec47eb8..205ee5273b27 100644
--- a/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
+++ b/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
@@ -2407,6 +2407,7 @@ Error RuntimeDyldELF::finalizeLoad(const ObjectFile &Obj,
   }

   GOTSectionID = 0;
+  GOTOffsetMap.clear();
   CurrentGOTIndex = 0;

   return Error::success();

with Numba and llvmlite main following the LLVM 14 merge:
(numballvm14) gmarkall@gm-agx:~/numbadev/numba$ python runtests.py numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest -v -m
Parallel: 9. Serial: 3
test_ex_literal_list (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_initial_value_list_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_literal_dict_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_inferred_list_jit (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_jitclass (numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest) ... ok
test_ex_inferred_list (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_jitclass_type_hints (numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest) ... ok
test_ex_nested_list (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_initial_value_dict_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_chunksize_manual (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok
test_chunksize_with (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok
test_unbalanced_example (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok

----------------------------------------------------------------------
Ran 12 tests in 95.065s

OK

All tests listed in the issue description pass. The runtime is huge, but I have a debug build of LLVM with assertions enabled, and LTO disabled for llvmlite (using LTO ICEd the old version of GCC on my Jetson install 😆 ).

Upstream LLVM issue: https://github.com/llvm/llvm-project/issues/61402

Upstream LLVM Differential: https://reviews.llvm.org/D146938


Is there a way to specify the base URL?

I'd like to deploy to :  mydomain.com/new  for client to review for example.

AFAIK the purpose of baseURL is different, it's explained in the main README.
I don't have any experience on cactus deploy but I'm setting up a GCS account to test it. I guess that the server directory should be defined in that context.

Have you tried adding the full URL in config.json?
{
    "site-url": "http://mydomain.com/new"
}

duplicate of #177