scverse datastructure for AIRR data

Now that scirpy is part of scverse, we could think of an improved data structure for scAIRR data. See also the discussion at https://github.com/theislab/scanpy/issues/1387.
The challenge with scAIRR data is that

1 cell can have n chains. Up to four of them are biologically meaningful but there could be more for technical reasons.
Each chain has a lot of fields. See the AIRR rearrangement standard.

The current pragmatic solution is to store all fields in adata.obs.

All columns from the airr rearrangement schema are repeated four times
Excess chains are serialized into JSON and stored in an extra column. These chains are not used by scirpy, but enable lossless conversions.
The downside is that there can easily be 100+ columns in adata.obs. Also serializing excess chains is not really elegant.
The advantage is that it works really well with scanpy, i.e. any AIRR variable can immediately be used for grouping, plotting etc.

New options are

mudata. AIRR data could be saved as a separate modality. Even if we keep the current reprepsentation of a wide data frame, it would at least not clutter the rest of adata.obs.
awkward array support. Allows storing an arbitrary number of values per row. See https://github.com/theislab/anndata/pull/647

The new representation should also aim at being a community standard for the scverse ecosystem and should build upon the AIRR rearrangement standard. Ideally, we could get additional stakeholders onboard, including conga, dandelion, tcrdist3 and possibly members of the AIRR community.

[x] what's the state of the AIRR single-cell schema? And what are its advantates over the rearrangement schema.

Hi All, I am active in the AIRR Standards group, can try and answer questions. We are looking at integrating scirpy into the iReceptor Gateway (gateway.ireceptor.org) to do analysis on AIRR Cell data - so have a vested interest... 8-)
The AIRR Cell schema was just announce at the AIRR meeting this week, with a tagged (v1.4) release expected on github in the next few weeks. This is an "Experimental" schema, meaning that we think it is pretty stable, but will probably need some minor changes before we it is finalized, probably in an AIRR Standard v2.0 release sometime in the "near" future.

The Cell schema is intended to represent a Cell as an object. It is a light weight object, and contains a cell_id that is used to cross reference information in other collections such as Rearrangement and CellExpression. So if you have a cell_id from a Cell you can find all of the Rearrangements associated in an AIRR TSV file by searching for that same cell_id. Similarly you can find all of the CellExpression information by searching a CellExpression file for that same cell_id.

Here are the current docs on the master branch, which will be tagged with a v1.4 release soon.
https://docs.airr-community.org/en/latest/datarep/cell.html

One of the outstanding things we need to resolve in the AIRR Standard is what if any "efficient" file format should be supported. Currently the default standard for all AIRR objects is a JSON representation of the standard objects. For Rearrangement there is a defined TSV file. Such a file format does not yet exist for Cell and more importantly CellExpression. We could probably use some advice from this community...
We have an about to be release version of our AIRR repository (https://github.com/sfu-ireceptor/turnkey-service-php/tree/production-v4) that can store Cell and CellExpression data, but currently what you get out is a very big, verbose JSON file. It works, but...

BTW, I think it would be interesting to decorate a cell's adata.obs with AIRR Repertoire metadata. That is, it would be quite easy (I think) to decorate the adata.obs with all sorts of subject/sample metadata for each cell such as subject.disease.disease_daignosis, subject.age_min, subject.sex, sample.tissue etc so that one can use these criteria for coloring, group, etc as well. It seems to me that these fields in adata.obs would be more valuable than the most of the Rearrangement fields. I could see wanting some Rearrangement fields, maybe VDJ calls, CDR3 length, locus for the paired chains, but it seems like most of the Rearrangement fields would not be very interesting from a slicing/coloring/visualization point of view.

Hi @bcorrie,
thanks for reaching out, I'm definitely interested in working with the AIRR community to make this as interoperable as possible.
Your description of the Cell schema is really helpful, before I wasn't sure it was to replace the Rearrangement schema, but now I understood it's built on top of it.
Actually scirpy internally uses something very very similar to the Cell schema already. Every file format read into scirpy is first represented as a list of AirrCell objects that have a cell_id, a list of Rearrangement entries and arbitrary cell-level attributes that are added as columns to adata.obs. Once the Cell schema is final, I could simply adjust my AirrCell implementation to match the cell-schema, which would make reading and writing AIRR-compliant data trivial.
The internal representation is one side of the medal, the other is representing the AIRR data as part of either an AnnData or MuData container, which allows to link the data to other modalities, allows efficient file storage in h5ad/h5mu and enables the interaction with scanpy or other scverse tools. The current way is to dump everything in .obs which is not ideal.

Such a file format does not yet exist for Cell and more importantly CellExpression. We could probably use some advice from this community...

Well, how about AnnData? If you are looking for a lower level of abstraction, I think hdf5, zarr or tiledb can handle these sparse data efficiently, but @ivirshup or @gtca are the experts here. Is it really necessary to define yet another format for single-cell expression?
Note that reading/writing gene expression formats would be out-of-scope for scirpy, this should be handled by scanpy or AnnData, or possibly by a future scverse IO package.
It's also worth noting here that the CZI is currently looking into standardizing Feature Observation Matrices (FOM) for single-cell data. Maybe it would be worth for the AIRR community and them getting in touch for talking about a AIRR-specific extension. Their public presence is still very sparse, but maybe @ivirshup can tell more about this initiative and make the contact.
Best,
Gregor

Such a file format does not yet exist for Cell and more importantly CellExpression. We could probably use some advice from this community...

Well, how about AnnData? If you are looking for a lower level of abstraction, I think hdf5, zarr or tiledb can handle these sparse data efficiently, but @ivirshup or @gtca are the experts here. Is it really necessary to define yet another format for single-cell expression?

Yes, AnnData seems logical and along the lines of what I was thinking... Definitely do not want YAF (yet another format) 8-)
Like you, this is likely out of scope for the AIRR Community - at least in the sense that the obvious thing for us to do would be to have the AIRR python library (https://github.com/airr-community/airr-standards/tree/master/lang/python) use scanpy to read and write these formats.
The CZI FOM is interesting as well...

Actually scirpy internally uses something very very similar to the Cell schema already. Every file format read into scirpy is first represented as a list of AirrCell objects that have a cell_id, a list of Rearrangement entries and arbitrary cell-level attributes that are added as columns to adata.obs. Once the Cell schema is final, I could simply adjust my AirrCell implementation to match the cell-schema, which would make reading and writing AIRR-compliant data trivial.

Very nice - one would hope that these would be closely aligned, and hopefully we can now make sure they are completely aligned 8-)

Draft proposal for an awkward-array-based data structure
I've been playing around with the draft implementation of awkward array support in AnnData (https://github.com/scverse/anndata/pull/647). This allows to store in .obsm (or in the future maybe also in .obs, see the comments there) ragged arrays of mixed types for each cell -- exactely what we need to represent the 1:n relationship of cells with chains.
An "awkward array" can be directly created from a list of AirrRearrangement dictionaries, e.g.
import awkward as ak
import scirpy as ir
airr_cells = ir.io.to_airr_cells(adata)
chains_as_awkarr = ak.Array([cell.chains for cell in airr_cells])

# an array of length 3000 (= number of cells), variable width (= number of chains) and
# AIRR Rearrangement fields on the 3rd dimension
> chains_as_awkarr
<Array [[{c_call: 'TRBC2', ... v_cigar: None}]] type='3000 * var * {"c_call": op...'>

To get, e.g. all junction_aa sequences of the first chain of each cell, this awkward array can be sliced:
> chains_as_awkarr[:, 0, "junction_aa"]
<Array ['CASSLMRLAGDTQYF', ... 'CATEEYGNKLVF'] type='3000 * string'>

It is therefore straightforward and computationally efficient to retrieve individual elements.
In summary: We simply take AIRR compliant data, put it in an efficient data structure and store it an AnnData. Not a lot new to define from our side
Caveats and possible solutions
1. Primary and Secondary chains
The list of chains does not know the concept of primary and secondary chains. This can be solved by adding an index array:
# the index arrays point to the element in the list of chains
primary_vj = [0, 0, 1, 2, 0, 1, ...]
primary_vdj = [1, 3, 5, 3, 2, ...]

The awkward array can then be sliced
> chains_as_awkarr[:, primary_vj, "junction_aa"]

to retrieve e,g. the junction AA sequences for the primary VJ chains.
2. Getting values for plotting
The current solution to put everything in obs was amongst others a pragmatic solution to make all AIRR data immediately available to scanpy plotting functions. This is not possible anymore with the proposed new implementation. Arguably, most of those columns are used very rarely for those purposes anyway.
If they are, here are some solutions to do so

implement a ir.get function to retrieve data from the awkward array (see also #184). The result can be re-assigned to adata.obs for plotting and/or grouping
re-implement all scanpy plotting functions as e.g. ir.pl.umap, where color is retrieved from the awkward array instead of .obs.

3. Where to store the data
The current draft implementation only supports awkward arrays in .obsm. IMO it would be completely fine to have an .obsm["airr"]. In the future, it may be possible to have a column .obs["airr"].
It was also previously suggested to use mudata as AIRR data can be considered a modality. Since we don't have any data for .X and .var, I don't see a clear advantage of using mudata, though.

I'm looking forward to feedback, especially from people that develop ecosystem packages for
AIRR analysis (@zktuong, ..., ?)

@bcorrie, this should make it possible that we merely define a mapping how the AIRR standard is represented as AnnData object rather than converting between a scirpy and an AIRR representation. Need to hash that out in more detail, but roughly

Cell metadata -> adata.obs
Rearrangement chains -> adata.obsm["airr"] (as awkward array)
CellExpression -> adata.X or mudata (in case of CITE-seq)

The CZI FOM is interesting as well...

@bcorrie from discussion with the SOMA team, the intent was to keep anndata and SOMA (formerly FOM) pretty consistent with each other. Ideally they are the same thing eventually.
Though I haven't heard much on SOMA development recently, so hopefully we are still on the same page here.

Where to store the data

@grst, I've recently heard people wanting to treat their re-arrangment data as a separate modality. E.g. have an AnnData with just re-arrangement info which could then be merged with other modalities. I think the main point here was to avoid having to arbitrarily associate AIRR data with another modality.
I think @bio-la and @crichgriffin could elaborate on this.

I don't remember the details, but one of the issues was with subsetting. Wanting to filter AIRR obs, but not the RNA.
I think it would work with AnnData right now, I think the issue was more about want to use scirpy with a MuData where RNA and AIRR were in separate modalities? I think this sorta worked, but there were issues with mudata operations.

In the context of multimodal data and using MuData, I like the idea of keeping the AIRR data separate from any specific modality. My current workflow is to store the scirpy anndata object as a separate modality in muon, where I also have rna and adt data, as separate modalities. But I agree that it doesn't make a lot of sense, since the scirpy anndata object is mostly empty.
I agree with @grst that storing airr data could potentially work in both AnnData and MuData by using .obsm["airr"] except (iirc) in AnnData .obsm row dimensions match .obs row dimensions, and I assume the same applies to MuData.
What would you do in the instances of cells not having AIRR data?  E.g. in PBMC data, monocytes would not have airr data, but you would want to keep them in your AnnData/MuData.

What would you do in the instances of cells not having AIRR data?

fill the corresponding rows with NaNs? This is how scirpy also handles this currently.
Although this is a good point in favor of mudata. It would also make ir.pp.merge_with_ir superfluous.
I think it should be easy to support both. Just need to think what to advertise as default in the tutorial.

all of this sounds awesome. Just have some naive question about how well/fast does it deal with situation when there's >100k+ cells? would chains_as_awkarr take a while to retrieve the entries?

That's a fair point, I need to try this out, but it should be very fast. Awkward array claims to be similarly scalable as numpy arrays.

Hi All, sorry for the lack of input - both fighting "the big C" and travelling to meetings (I don't recommend trying to both at the same time) the last several weeks.
I am not an AnnData expert - yet - but my observations thus far.
We are experimenting with Conga and it annotates each AnnData cell with heavy and light chain VDJ/CDR3 in the .obs of the object. Other tools like CellTypist also populate the .obs with other data.
This seems messy, and .obsm seems like a good option for adding specific annotations to a cell object on a per "tool" basis. So there might be a 'AIRR' obsm object, but also a 'Conga' and 'CellTypist' object. This would separate these cell annotations cleanly - would that be considered good practice. It seems to make sense to me that the community might encourage this???

2. Getting values for plotting
The current solution to put everything in obs was amongst others a pragmatic solution to make all AIRR data immediately available to scanpy plotting functions. This is not possible anymore with the proposed new implementation. Arguably, most of those columns are used very rarely for those purposes anyway.

Yes, it occurred to me that plotting might be challenging using obsm. I personally think this would be quite important.
I would anticipate using the obsm['airr'] to store not only rearrangement annotations, but also what we call repertoire annotations (by repertoire AIRR means study/subject/sample metadata). For example, it is quite easy for us to generate a pool of cell data from many subjects in a study, possibly across disease conditions (healthy, mild covid, severe covid). We would pool that data and create a single h5ad file. We would want the 'disease_state' and 'subject_id' to be in the AIRR obsm object so we could visualize cells based on these fields. Same for the AIRR 'tissue' field and a range of others potentially.
It seems like a bad idea to have this difficult.

A quick example - this is from a combined Conga/CellTypist analysis, with the .obs annotated with Conga and Cell Typist fields. This is trivial to visualize VDJ annotations per cell next to CellTypist majortiy voting - at the same time - because the columns are added to .obs. If they were in .obsm and the visualizations could not make use of this easily, this would be challenging...

@bcorrie, this should make it possible that we merely define a mapping how the AIRR standard is represented as AnnData object rather than converting between a scirpy and an AIRR representation. Need to hash that out in more detail, but roughly

Cell metadata -> adata.obs
Rearrangement chains -> adata.obsm["airr"] (as awkward array)
CellExpression -> adata.X or mudata (in case of CITE-seq)

This seems to make sense to me as a first pass, although I would probably suggest that adata.obsm['airr'] might contain more than just info about rearrangement chains (see https://github.com/scverse/scirpy/issues/327#issuecomment-1172887702) as AIRR Repertoire metadata (disease_state, tissue, age, sex, ...) on a cell basis will often be very valuable.
One could have  adata.obsm['airr_rearrangement'] and adata.obsm['airr_repertoire'] -  but maybe too clunky? Another related link that cells potentially have is to "Receptors" - which are known B/T cell receptors that have a specific antigen/epitope specificity (think of the B and T cell specificity info in IEDB). Again, maybe worth capturing.

For what it's worth, on the R side we've been using MultiAssayExperiments with a "rearrangement" experiment that includes the AIRR Rearrangement data for storing multimodal single-cell data that includes AIRR data, GEX, CITE-seq, and/or whatever people dream up.  The AIRR assay data (equivalent to .X) is stored as a BumpyMatrix, specifically a BumpyDataFrameMatrix. We have a primary key (row names) derived from the locus field to easily separate out IGH/TRB, but I don't think that's necessary. Sample/cell level metadata goes into the colData (.obs) as is typical.
If I'm understanding the awkward array correctly, and I may not be, this would be the same as using the "record" array implementation to populate .X with an awkward array in a mudata object.  A pandas DataFrame with multi-indexing seems like the most natural fit for working cellular Rearrangement data (eg, key on something like ['cell_id', 'locus', 'sequence_id']) and it looks like the conversion from awkward array to multi-indexed DataFrame is trivial. Unfortunately, my python is a bit rusty these days, so I could be misunderstanding.
PS:
"We" is not the AIRR Standards WG in this case. I don't think we should have an official opinion on implementation.

where to store the data (.obs vs. .obsm vs. .X)

The AIRR assay data (equivalent to .X) is stored as a BumpyMatrix, specifically a BumpyDataFrameMatrix. We have a primary key (row names) derived from the locus field to easily separate out IGH/TRB, but I don't think that's necessary. Sample/cell level metadata goes into the colData (.obs) as is typical.

@javh this sounds pretty much like what I would imagine! Interesting to have the BumpyMatrix in .X.
In combination with mudata this actually makes a lot of sense and would be consistent to other modalities.

I would probably suggest that adata.obsm['airr'] might contain more than just info about rearrangement chains (see https://github.com/scverse/scirpy/issues/327#issuecomment-1172887702) as AIRR Repertoire metadata (disease_state, tissue, age, sex, ...) on a cell basis will often be very valuable.

Other tools like CellTypist also populate the .obs with other data.
This seems messy, and .obsm seems like a good option for adding specific annotations to a cell object on a per "tool" basis. So there might be a 'AIRR' obsm object, but also a 'Conga' and 'CellTypist' object. This would separate these cell annotations cleanly - would that be considered good practice.

@bcorrie, I'll try to summarize here a bit the (typical) purpose of the different AnnData slots.

.obs has been designed to hold all cell-level metadata. I would stick to that and add the repertoire fields (disease state, tissue, etc.) to .obs to make them readily available for plotting and other scanpy functions. So far, AnnData does not distinguish between columns added by different tools. I like the idea of having more structure in .obs, but that's something we would need to discuss on the AnnData side.

.obsm has been designed to hold data matrices aligned to .obs, e.g. UMAP or PCA. One could argue that Rearrangement data is not a data matrix either. An alternative would be to add an "awkward array column" to adata.obs:
adata.obs["airr_rearrangement"] = ak.Array(...)

Yet another option would be to put the awkward array in .X as @javh suggests.
The thing here is that having awkward columns requires some additional effort on the AnnData side and it is unclear how long this would take (see https://github.com/scverse/anndata/pull/647). Awkward arrays in obsm already work, but the PR needs to be wrapped up. Awkward arrays in X are not implemented either, but proably easier to fix than the pandas extension.

accessing data for plotting

If they were in .obsm and the visualizations could not make use of this easily, this would be challenging...

I agree that cell-type, patient info etc. must be directly accessible for plotting (which they would be if they are in .obs).
One can argue how useful a UMAP plot with V/D/J genes really is, similar to other information from the rearrangement schema. I would envisage something like this:
# if one really wants to plot rearrangement fields, add them to adata.obs explicitly
adata.obs["my_v_gene_col"] = ir.get("v_call", receptor_arm="VJ", dual_ir="primary")
sc.pl.umap(adata, color="my_v_gene_col")

That would be easy enough IMO and makes it explicit which "chains" to choose from the possibly multiple chains that are available for each cell.

@javh in R, did you use any specific package for immune receptor analysis?

@grst, I use the Immcantation packages, because I'm biased like that. :)
Regarding cell metadata in .osb, that seems cleaner to me as well.  You're going to want to be able to slice and copy cell metadata (1) across modalities (eg, cell types from the gene expression object, clonal clusters from rearrangement, etc) and (2) from the top level .obs if you go the mudata route. Same for .osbm data from other modalities (eg, look at clones on the UMAP from genes expression, correlate gene expression with mutation frequency / selection pressure). "Look in .obs for every modality" seems simplest.

@grst, I use the Immcantation packages, because I'm biased like that. :)

I'm asking because I was wondering if we can extend this effort beyond the scverse world and have a 1:1 mapping to a MultiAssayExperiment representation.

@grst From the MuData perspective, it is more general than MultiAssayExperiment. So it should be possible to align with how the data is stored there!

After the input from @javh and the discussion in https://github.com/scverse/anndata/pull/647, it seems cleanest to me to go the mudata route with AIRR data stored in .X. Prerequisite is that anndata will get awkward array support for .X with two aligned, fixed-length axes.
.X would then look something like this:

axis 0 holds observations (=cells) (fixed-length)
axis 1 holds AIRR rearrengement variables (e.g. v_call, duplicate_count, ...) (fixed-length)
axis 2 holds 0-n chains (variable-length)

Data could be retrieved as follows:
mdata.mod["AIRR"][:, "junction_aa"].X

For what it's worth, on the R side we've been using MultiAssayExperiments with a "rearrangement" experiment that includes the AIRR Rearrangement data for storing multimodal single-cell data that includes AIRR data, GEX, CITE-seq, and/or whatever people dream up. The AIRR assay data (equivalent to .X) is stored as a BumpyMatrix, specifically a BumpyDataFrameMatrix

@javh, do you happen to have an example where this is implemented in R? Specifically the AIRR assay data + bumpy matrix part?

We are working on exporting AIRR Cell/Expression data in an anndata file, with AIRR metadata features as annotations. So not the Rearrangement fields, but rather the Repertoire metadata fields - so we know which cells have which subject/disease/etc characteristics.
I am not an expert on anndata and mudata so need to wrap my head around what we are talking about.
In the following:

axis 0 holds observations (=cells) (fixed-length)
axis 1 holds AIRR rearrengement variables (e.g. v_call, duplicate_count, ...) (fixed-length)

I think these fixed lengths are the same, correct? That is for every entry on axis 0 there is an entry on axis 1? Or that
length(mdata.mod["AIRR"].obs) == length(mdata.mod["Cell"].obs)
If want all of the AIRR Data on dimension 1 I do the following since dimension 1 has a name AIRR:
mdata.mod["AIRR"].X
And I get the original observations (cells) on dimension 0 in a similar way (and it would have a name as well)?
mdata.mod["GEX"].X
Would you suggest that these modality names are standardized for the different modalities of data, or are the arbitrary and up to the user?

In the following:

axis 0 holds observations (=cells) (fixed-length)
axis 1 holds AIRR rearrengement variables (e.g. v_call, duplicate_count, ...) (fixed-length)

I think there is some confusion here. These axes refer to the axes of the matrix holding the AIRR data, not the obs_names axes of the modalities. E.g. mdata.mod["AIRR"].X[axis_0, axis_1]

@bcorrie, from when you mentioned:

The AIRR assay data (equivalent to .X) is stored as a BumpyMatrix, specifically a BumpyDataFrameMatrix

Could you link to a vignette or some code where you are using a bumpy matrix to work with AIRR data?

axis 0 holds observations (=cells) (fixed-length)
axis 1 holds AIRR rearrengement variables (e.g. v_call, duplicate_count, ...) (fixed-length)

I think these fixed lengths are the same, correct? That is for every entry on axis 0 there is an entry on axis 1? Or that
length(mdata.mod["AIRR"].obs) == length(mdata.mod["Cell"].obs)

What I meant with axes  here are the dimensions of one single anndata object, in this case the one holding AIRR data.

axis 0 = cells (e.g. n=20,000)
axis 1 = variables (v_call, d_call, junction_aa, etc)

When talking about a mudata object with multiple modalities e.g. mdata.mod["AIRR"], mdata.mod["GEX"], the number of observations is not necessarily the same. For instance, GEX could contain all sorts of immune cells, while AIRR data is only available for B/T cells. However, cells that are shared between the modalities would have the same identifier in .obs_names.

Would you suggest that these modality names are standardized for the different modalities of data, or are the arbitrary and up to the user?

From the scirpy side, I would not require the modality to have a specific name, but AIRR sounds like a reasonable convention. If this is becoming part of some AIRR standard, then it's anyway up to you, but I guess standardized modality names would make sense there.

@grst, yep, here's an example using one of 10x's older public PBMC data sets (attached):
library(SummarizedExperiment)
library(BumpyMatrix)
library(airr)

# Load data into a BumpyMatrix
df <- read_rearrangement("pbmc3_ig_db-pass.tsv.gz")
f <- factor(df$locus, levels=unique(df$locus))
bumpy <- splitAsBumpyMatrix(DataFrame(df), row=f, column=df$cell_id, sparse=TRUE)

# Make SummarizedExperiment
se <- SummarizedExperiment(list(airr=bumpy))
colData(se)$Cell <- colnames(bumpy)
colData(se)$Sample <- "A"

# All records for a locus
assay(se, "airr")["IGH", ]

# All records for a cell
assay(se, "airr")[, "AAACCTGAGTCGATAA-1"]

# All records for a cell and locus
assay(se, "airr")["IGH", "AAACCTGAGTCGATAA-1"]

# Back to DataFrame
x <- unlist(assay(se, "airr"))

pbmc3_ig_db-pass.tsv.gz

@grst, since it took me a sec, here's a quick way to install:
mamba create -n r-airr bioconductor-summarizedexperiment bioconductor-bumpymatrix r-tidyverse
conda activate r-airr

r # start an r session
install.packages("airr")

In something like data shape, the dimensions are: Regular(len(locus)) * Regular(len(cell)) * var * Record([sequence_id, sequence,  rev_comp, ...]).

Thanks @javh! This is again slightly different to what I had in mind. You introduce a separate dimension for locus, whereas I would have gone for putting all chains into a single dimension, and store separate indices that allow to retrieve VJ (i.e. TRA/TRG/IGK/IGL) or VDJ (i.e. TRB/TRD/IGH) loci. Both is possible and maybe worth another discussion.
What I didn't understand yet: Is this format something you use in-house only or is it part of the AIRR stndard, some other community standard or any R package?

After discussion with @ivirshup, I'm still a bit on the fence if the awkward array with AIRR data should rather go into adata.X or into adata.obsm["airr"].
adata.X
Store as a ragged array, rely on adata.var for the field names.
Advantages

When working with mudata and other modalities, it seems most "symmetric" to have the main data of a modality in mudata["airr"].X
Easily get an overview of the available variables through adata.var_names (without having to touch an awkward array)
Allows to store variable-metadata in adata.var (like a description, but it is unclear if there would be anything other than that)

Disadvantages

will require a bit more effort on the anndata side (though nothing that couldn't be resolved)

This is what the array would look like:
[
    # cell0: 2 chains
    # locus        , junction_aa
    [["TRA", "TRB"], ["CADASGT...", "CTFDD..."]],
    # cell1: 1 chain
    [["IGH"], ["CDGFFA..."]],
    # cell2: 0 chains
    [[], []]
]

adata.obsm
Store as an array of records
Advantages:

more similar to the AIRR rearrangement tsv data
the order of the dimensions feels more natural ("cell" > "chain" > "field")

This is what the array would look like:
[
    # cell0: 2 chains
    [{"locus": "TRA", "junction_aa": "CADASGT..."}, {"locus": "TRB", "junction_aa": "CTFDD..."}],
    # cell1: 1 chain
    [{"locus": "IGH", "junction_aa": "CDGFFA..."}],
    # cell2: 0 chains
    [],
]

Consequences
Implementation-wise and for the end-user the difference between the two should be limited:

In both cases, there would be an array in obsm that allows to retrieve the VJ/VDJ chains:
[
   "VJ_1": [0, None, None],
   "VDJ_1": [1, 0, None],
]

Having an additional locus or receptor_arm dimension as proposed by @javh would be an alternative to this, and could be adapted to both version.

With the new datastructure, there will be a new module to easily retrieve the data roughly like this:
ir.get.airr(adata, "junction_aa", "VJ_1")

See also the discussion at #184.

To broaden the scope even further, looping more people into the discussion:
@ncborcherding is maintaining scRepertoire, the scirpy-equivalent in the Seurat world. Nick, maybe you could give us your perspective on how you store AIRR data in Seurat-objects?
@kmayerb and @phbradley, with TCRdist3 and conga you maintain other immune-cell receptor analysis packages in the python universe that are reasonably popular and complementary to scirpy. In an ideal world we would be using the same data structure for better interoperability. Maybe you are interested in joining this discussion.
I'm aware that this is quite a thread to read through, but the top-level comment describes what this is all about, and in the comment just before are the latest proposals of how this could be implemented.

@bcorrie, from when you mentioned:

The AIRR assay data (equivalent to .X) is stored as a BumpyMatrix, specifically a BumpyDataFrameMatrix

Could you link to a vignette or some code where you are using a bumpy matrix to work with AIRR data?

I think that was @javh that was talking about "bumpy matrix"

@kmayerb and @phbradley, with TCRdist3 and conga you maintain other immune-cell receptor analysis packages in the python universe that are reasonably popular and complementary to scirpy. In an ideal world we would be using the same data structure for better interoperability. Maybe you are interested in joining this discussion.

One of the tools we are integrating with (and therefore exporting to) is Conga - that is exactly our use case 8-)

You introduce a separate dimension for locus, whereas I would have gone for putting all chains into a single dimension, and store separate indices that allow to retrieve VJ (i.e. TRA/TRG/IGK/IGL) or VDJ (i.e. TRB/TRD/IGH) loci. Both is possible and maybe worth another discussion.

If I had my druthers, I would also have that index be the chain type instead of the locus, because that more naturally fits with how you're likely to work with the data (avoiding the awkwardness of always indexing on c("IGK", "IGL")). Which just means doing this instead:
df <- read_rearrangement("~/Downloads/pbmc3_ig_db-pass.tsv.gz") %>%
    mutate(chain=alakazam::getChain(locus))
chain <- factor(df$chain, levels=c("VH", "VL"))
bumpy <- splitAsBumpyMatrix(DataFrame(df), row=chain, column=df$cell_id, sparse=TRUE)

However, this has nomenclature problems and is very B cell centric. There's no comparable terminology to VH and VL for T cells, which instead uses alpha-beta and gamma-delta. Hence, there's no field in the AIRR Rearrangement standard that defines standard nomenclature for chain type.  That's trivial to overcome. But, that does mean whatever your solution, it won't necessarily be consistent across tools. Eg, I have minor reservations about VDJ vs VJ as the chain type. We decided to use VH and VL in alakazam, which is also fine as a key, but still incorrect terminology in the case of TCRs.
I'm also not certain in makes sense to combine analyses for TRA and TRG.
Hence, locus ended up being the index. Not ideal, because downstream analysis often starts with adding the chain field. But, it doesn't suffer from any wrongness upon import either.

What I didn't understand yet: Is this format something you use in-house only or is it part of the AIRR stndard, some other community standard or any R package?

In-house only. It doesn't appear in a public package (that I'm aware of). And it is not part of the AIRR Standards efforts or the AIRR Data Commons stuff @bcorrie is talking about. So there's no need to conform to this data structure if you have something better in mind.

Very interesting discussion and thanks for looping me in!
As of right now, scRepertoire is appending a portion of the filtered TCR/BCR alignments to the meta data of a single cell object (either Seurat or SingleCellExperiment).  It is not necessarily ideal as it does not preserve AIRR format or other information that users have been requesting, like cdr1/2 sequences.
In terms of implementing a change for consistency - from the R side of things, SingleCellExperiment functions as a SummarizeExperiment and is similar to the python equivalent. However, I don't think Seurat data format would be compatible with the array you are proposing. But I can do some investigating

I decided to go with the adata.obsm variant described in https://github.com/scverse/scirpy/issues/327#issuecomment-1238067096, storing all chains in a single dimension rather than making an additional dimension for loci.
This makes IO agnostic of loci, which aren't standardized or even a mandatory field in rearrangement data.
Calling "primary" and "secondary" chains would then happen in a separate step, which would also allow to implement different strategies for chain ranking in the future.
If anyone has reservations against this approach, now would be a good time to speak up, otherwise it might be too late.

I'm finally making some progress on the new datastructure (#356). Here is how it is currently working out:

IO functions read in data into an awkward array in adata.obsm["airr"]. At this point no filtering happens, all information from an AIRR rearrangement file is preserved.
# adata.obsm["airr"]
[
    # cell0: 2 chains
    [{"locus": "TRA", "junction_aa": "CADASGT..."}, {"locus": "TRB", "junction_aa": "CTFDD..."}],
    # cell1: 1 chain
    [{"locus": "IGH", "junction_aa": "CDGFFA..."}],
    # cell2: 0 chains
    [],
]

The new function scirpy.pp.index_chains is called. It adds another awkward array to adata.obsm["chain_indices"]. This function applies what is described as the scirpy receptor model, i.e. it flags multichain cells, and calls primary and secondary VJ and VDJ cells.
# adata.obsm["chain_indices"]
[
    {"VJ": [0], "VDJ": [1], "multichain": False}, # single pair
    {"VJ": [0, 2], "VDJ": [1,3], "multichain": False}, # dual IR
]

The beauty of this is that this can easily be adapted to other receptor models. For instance, the index_chains function can be called with productive=False to keep non-productive chains. It would also be thinkable to develop a completely different receptor model with an arbitrary number of chains per obs. This could be relevant for spatial or bulk TCR data. This would require new analysis functions in scirpy (or another library), but it all fits into the same data structure.

Get AIRR data by slicing the adata.obsm["airr"] with the indices in adata.obsm["chain_indices"]. For convenience, there is now a helper function that allows to request one, or multiple columns from the AIRR data, e.g.
>>> scirpy.get.airr(adata, "locus", "VJ_1")
AAACCTGAGAGTGAGA-1     TRA
AAACCTGAGGCATTGG-1     TRA
AAACCTGCACCTCGTT-1    None
...

Work with MuData. MuData will be the new recommended way of combining gene expression and AIRR data.
mdata = md.MuData({"airr": adata_airr, 'gex': adata_gex})
ir.tl.chain_qc(mdata["airr"])

I still need to iron out a few details, and most importantly, update the documentation and tutorial to reflect all these changes. There will be an automatic check and a conversion function to transfer data from the previous format into the new data structure.

The new datastructure is now available in the v0.13.0rc1 pre-release :tada:!
I'm happy about everyone who tries it out and gives feedback!
You can install it using
pip install --upgrade --pre scirpy

Please also check out

the release notes
the documentation, in particular the page about the data structure and the updated tutorials.

I can see in the tutorials how I can access Rearrangement fields which is excellent. What about Subject or Sample fields from AIRR such as the age of donor/patient etc; metadata that would be the same for a subset or all of the cells under obs

I agree - great to see this moving forward - thanks @grst and others.
@gszep we talked about Repertoire metadata earlier (https://github.com/scverse/scirpy/issues/327#issuecomment-1172887702) but I think currently scverese libraries only load AIRR Rearrangements. Would love to hear that I am wrong, but I believe that to be the case.

There is an example in the docs (https://scverse.org/scirpy/tags/v0.13.0rc1/tutorials/tutorial_io.html#Combining-multiple-samples) where multiple samples are combined, and the equivalent of the AIRR sample_id is set for multiple samples in the obs data. In this case the obs_name is used to assign a sample obs.
So the trick would be to load in an AIRR Repertoire file and process it to do something similar based on the repertoire_id in the Rearrangement data. One day soon I hope to get to doing something like this for the Single Cell downloads from the iReceptor Gateway.

Thank you for your reply.
I am still learning the best practices for anndata. Say we have a usecase of 20 repertoires each with 100,000 cells or so.
In this case, would it make sense to have 20 Anndata objects and in each anndata object one would have the 100,000 'Rearrangement' rows stored under the airr path and the AIRR Repertoire stored in uns or similar?

I am unfortunately not an anndata/scirpy expert either, I come from the AIRR side. My simple answer would be it depends on if you want to compare the repertoires. If yes, then having them in a single object is probably best (as in the example - you have to worry about batch effects). That way you can slice and dice across any of the metadata fields from the Repertoir.
I don't yet have any experience with how well scirpy and related tools will scale when you are throwing together data sets of this size. Any scirpy experts want to comment.

Yes, I would guess keeping them separate until you want to compare them, then concatenate them as required for specific comparisons. Still an interesting scalability question as to how many samples can you practically compare in a single scirpy object.

i can help answer some of these questions (disclaimer - i still need time to familiarise myself with the new scirpy data structure but i'm familiar with the other main bits):

@gszep we talked about Repertoire metadata earlier (https://github.com/scverse/scirpy/issues/327#issuecomment-1172887702) but I think currently scverse libraries only load AIRR Rearrangements. Would love to hear that I am wrong, but I believe that to be the case.

The main thing is that the anndata.obs slot index is cell_id, whereas the airr data's index is sequence_id. So scirpy stores the airr data separately in the .uns and uses the awkward arrays to both store and retrieve the information. Although i believe the awkward arrays can theoretically hold any information in the AIRR row for each contig. So you could store those information if you have it, just that it won't be retrieved automatically.

I am still learning the best practices for anndata. Say we have a usecase of 20 repertoires each with 100,000 cells or so.
In this case, would it make sense to have 20 Anndata objects and in each anndata object there would be the 100,000 'Rearrangement' rows stored under the airr path and the AIRR Repertoire stored in uns or similar?

Yes. the thing to look out for in the airr table is that your sequence_id are unique. cell_id can be repeated in the airr table but i think internally the individual contigs will still be ranked by umi count to sort them from largest to smallest, so that the highest expressing pairs of heavy/light or long/short contigs would be the main bcr/tcr for that cell.

Yes, I would guess keeping them separate until you want to compare them, then concatenate them as required for specific comparisons. Still an interesting scalability question as to how many samples can you practically compare in a single scirpy object.

The io step and retrieval steps should be fast i believe?

 If these advantages are exposed in anndata.AnnData class then we shouldn't have to worry about running out of memory and  can concatenate and compare as much as we want. However as far as I can tell:

Lazy concatentation of different files is still an experimental feature available via anndata.experimental.AnnCollection and there are issues being tracked for this https://github.com/scverse/anndata/issues/793
Lazy partial loading within a single file is supported for .X (maybe also .layers?) using the backed="r" mode but it appears that .obs and .var are still all loaded into memory. I've create an issue to suggest supporting lazy dataframes https://github.com/scverse/anndata/issues/981
After the above two issues are solved, the lazy versions of plotting and analysis functions must be written. Hopefully some of them will work out of the box as dask closesly matches the pandas api.

Thanks for all your comments!

What about Subject or Sample fields from AIRR such as the age of donor/patient or disease_diagnosis etc; metadata that would be the same for a subset or all of the cells under obs

These are usually just stored as additional columns in .obs. Represented as categoricals, this is quite memory-efficient.

Ideally one could request an h5ad download of multiple samples and get a single h5ad file with all the GEX, Cell, Rearrangement, and Repertoire metadata embedded in that single h5ad file. Currently when you do a download you get 4 separate files, one for each type of data.

I'm definitely open to add more reader functions to load other AIRR schemas than Rearrangement. Could you maybe point me to an example dataset that has all four data types? Then I'll take a look how to best represent them as AnnData.
I'll separately comment on scalability later.

Regarding scalability:
Different steps in the scirpy pipeline are subject to different limitations. My goal is to enable improve scalability of scirpy such that analysis of (few) millions of cells is conveniently possible on a single workstation (e.g. >200GB RAM, >30 cores). This is tracked here: https://github.com/scverse/scirpy/issues/370.
For anything >10M cells, I believe we need to move to out-of-memory and out-of-core approaches. Solutions for this are still being figured out on the AnnData side as @gszep has pointed out. AnnData's current "backed mode" does not support layers and obsm. I believe random access of serialized awkward arrays requires some additional effort, but will be possible.

To get a better idea of current limitations of scalability, I'll share here my experiments with omniscope's longitudonal COVID19 dataset with 8M TCR-beta chains (and no gene expression data):

reading the rearrangement files: ~15min (can be further optimized & parallelized)
the index_chains function: 2:30h (I think I can speed this up significantly using numba, #386).
retrieving a data frame of all "junction_aa" sequences: 13sir.get.airr(adata, "junction_aa")

subsetting anndata and retrieving 10k "junction_aa": 34msir.get.airr(adata[200000:210000], "junction_aa")

size of AnnData object on-disk (uncompressed): 23GB

The real bottlenecks of the scirpy workflow are further downstream:

levenshtein or alignment distance take ~24h on 44 cpus
clonotype calling takes ~24h on 1 cpu, parallelization as currently implemented is not effective.

I'm definitely open to add more reader functions to load other AIRR schemas than Rearrangement. Could you maybe point me to an example dataset that has all four data types? Then I'll take a look how to best represent them as AnnData.

Any chance you have an iReceptor Gateway account 8-)
I just did a download of the data from one subject from a cancer study from Yost et al (http://doi.org/DOI:10.1038/s41591-019-0522-3). The download contains three Repertoires from a single subject at different time points, two pre-treatment and one post-treatment. The data will have a rearrangement file, a cell file (AIRR JSON format), and a GEX file (AIRR JSON format) (as well as some other files). Each will contain all of the data of one type from all three Repertoires. That is the rearrangement file will contain rearrangements from all three repertoires (actually six repertoires in the rearrangement case, as they are split into TRA and TRB for each time point). It is a 10X study with ~4500 Cells across all three time points.
The ZIP file is 180MB so probably want to share it outside of github.

Happy to announce that the new datastructure is now rolled out as part of scirpy v0.13.
Please check out the release notes.

Excellent!


Enforce log reader stop after failure of one of the stream readers

If my understanding is correct new generation can prolong a lifetime of stream readers and continuous calls to CDCLogReader.stop_at could even prevent them from stopping.

@piodul


Map index problem

Hello, I'm using point_ During the rend experiment, my map changes are very jittery and often reach the peak in the middle of training, but the highest value only appears a few times, and then begins to decline and rise. Is this normal

Maybe your model has started to overfit. Try to tune the hyperparameters. Also, you can try to decay the learning rate when mAP stops increasing.


Tunnel mole stuck

I am using tunnel mole to forward my server running at port 4000 but the forwarding links are not showing. Please refer to the image below. It gets stuck here.

Same here

I can't get it to work either

Not sure what happened there but I restarted the service, now the issue is fixed

Please provide the command, not finding the service as well.

its back to working, usual commands should work.

Thanks.Its working perfectly.

Workin here!

Hi! The same issue is happening again. Tried from my own computer and also a remote server.

I really appreciate this project. But I have to report the same issue.
@robbie-cahill


feat: Store large and binary file with DVC

Fixes #616

The primary motivation for exploring DVC was to store binary assets like images. We are now storing images on S3 so using DVC is no longer needed.


[Android] Optimizing layout hierarchy of XWalkView

Decouple XWalkContent from FrameLayout, this can reduce the XWalkView's 4 level view hierarchy to
to 3.
BUG=XWALK-4760

@lincsoon  @fujunwei  @halton

@halton  @lincsoon

LGTM @axinging please rebase

LGTM

@halton @lincsoon , interesting! Rebase works!

Thanks, will merge after trybot green.


feat(ui): show activity type icon in activity list

Currently, it's not immediately obvious which transactions are on-chain vs off-chain when scanning the activity list.
Add an icon that shows the activity type in the activity list to help users clearly distinguish between on-chain and off-chain transactions.
Before:

After:

Have updated to add tooltips. eg:

utACK 3d2e4dd 🎊


Measurement is very slow

The measurement used to determine paddle position is very slow. It gives a laggy feeling.

Fixed, but not sure how it happened. It seems to be related to the order of drawing sprites


create view error

hello,  I have a statistic  need  to join 5 more tables , I want to create view as  new big table, and use the table to query,I use query as create view  result. it's error.
there is sample  create sql,
CREATE VIEW test  AS
SELECT B.COM_ID  FROM ITEM_DPT_MONTH A
LEFT join DC_CUST B ON A.COM_ID=B.COM_ID

error info
DB::Exception: Unknown identifier: B.COM_ID, context: query: 'SELECT B.COM_ID FROM ITEM_DPT_MONTH  LEFT JOIN DC_CUST ON A.COM_ID = B.COM_ID' required_names: 'A.COM_ID''B.COM_ID' source_tables: 'DC_ITEM_DPT_MONTH' 'DC_CUST' table_aliases: complex_aliases: masked_columns: array_join_columns: source_columns
can you give some suggest? tks!

Does it work like that?
SELECT B.COM_ID  FROM ITEM_DPT_MONTH
LEFT join DC_CUST USING (COM_ID)

SELECT B.COM_ID  FROM ITEM_DPT_MONTH
LEFT join DC_CUST USING (COM_ID)
this is ok,but I want to use more than 4 table join it's not work

Can you please update ClickHouse version to 19.11 and retry?

The error message is straightforward. There's an USING keyword into your multiple JOIN that is not supported. You are able to rewrite it with ON.

Cannot reproduce on current master build.
CREATE VIEW test AS SELECT A.dummy FROM system.one A LEFT join system.one B ON A.dummy=B.dummy

CREATE VIEW test AS
SELECT A.dummy
FROM system.one AS A
LEFT JOIN system.one AS B ON A.dummy = B.dummy

Ok.

I updated ck, it's only execute select sql  is ok ,but   create view it's not work.


add webhook cert configs

The cert-manager approver-policy webhook server TLS certs (ref link) are configured with cert-manager dynamic authority.
Currently the generated CA is valid for 1 day and leaf cert for 1 hr (which is rotated 1/3 of the time before expiry i.e. every 40 mins).
This PR makes the webhook server dynamic_source CA duration and leaf certificate duration as configurable. It also makes the default CA Duration as 1 year and default leaf certificate duration as 7 days.
Testing
Tested in local KinD cluster,
✗ k get secrets -n cert-manager
NAME                                 TYPE                 DATA   AGE
cert-manager-approver-policy-tls     Opaque               3      21h
...

Details: Cert-Manager-approver-policy-tls secret and CA Certificate Default Duration

✗ k get secret -n cert-manager cert-manager-approver-policy-tls -oyaml
apiVersion: v1
data:
  ca.crt: LS0tLS1C<...redacted...>UtLS0tLQo=
  tls.crt: LS0tL<..redacted..>NBVEUtLS0tLQo=
  tls.key: LS0tLS<..redacted ...>S0VZLS0tLS0K
kind: Secret
metadata:
  annotations:
    cert-manager.io/allow-direct-injection: "true"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{"cert-manager.io/allow-direct-injection":"true"},"labels":{"app":"cert-manager-approver-policy","app.kubernetes.io/instance":"cert-manager-approver-policy","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"cert-manager-approver-policy","app.kubernetes.io/version":"0.0.77","helm.sh/chart":"cert-manager-approver-policy-v0.0.0"},"name":"cert-manager-approver-policy-tls","namespace":"cert-manager"}}
  creationTimestamp: "2024-06-28T21:46:08Z"
  labels:
    app: cert-manager-approver-policy
    app.kubernetes.io/instance: cert-manager-approver-policy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cert-manager-approver-policy
    app.kubernetes.io/version: 0.0.77
    helm.sh/chart: cert-manager-approver-policy-v0.0.0
  name: cert-manager-approver-policy-tls
  namespace: cert-manager
  resourceVersion: "4501010"
  uid: 94791c01-779f-44a8-92e0-e83832c1dac1
type: Opaque

# Default Duration of the CA Certificate is 1 year
✗ echo "LS0tLS1CRUdJTiBDR<...redacted...>NBVEUtLS0tLQo=" | base64 -d | openssl x509 -text - | grep 'Not Before\|Not After\|Subject'
            Not Before: Jun 28 21:46:09 2024 GMT
            Not After : Jun 28 21:46:09 2025 GMT
        Subject: CN=cert-manager-webhook-ca
        Subject Public Key Info:
            X509v3 Subject Key Identifier:

Details: Describe Approver-Policy Pod in local kind cluster

✗ k describe pod cert-manager-approver-policy-7bc58774d6-mwg8p -n cert-manager
Name:         cert-manager-approver-policy-7bc58774d6-mwg8p
Namespace:    cert-manager
Priority:     0
Node:         rgodha-local-worker2/<IP_ADDRESS>
Start Time:   Fri, 28 Jun 2024 14:46:08 -0700
Labels:       app=cert-manager-approver-policy
              app.kubernetes.io/instance=cert-manager-approver-policy
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cert-manager-approver-policy
              app.kubernetes.io/version=0.0.77
              helm.sh/chart=cert-manager-approver-policy-v0.0.0
              pod-template-hash=7bc58774d6
Annotations:  <none>
Status:       Running
IP:           <IP_ADDRESS>
IPs:
  IP:           <IP_ADDRESS>
Controlled By:  ReplicaSet/cert-manager-approver-policy-7bc58774d6
Containers:
  li-cert-manager-approver-policy:
    Container ID:  containerd://277704d557ec4437729c1a7207c6d07bab7a01bb30625d7425f92fdaf6fd03a5
    Image:         cert-manager/cert-manager-approver:0.0.77
    Image ID:      docker.io/library/import-2024-06-28@sha256:26a62b26a4c7d00baca0179e2b4ee9ff905fbc4ff52a32dc6f715fd1eb36b1fd
    Ports:         10250/TCP, 9402/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --log-level=3
      --metrics-bind-address=:9402
      --readiness-probe-bind-address=:6060
      --webhook-host=<IP_ADDRESS>
      --webhook-port=10250
      --webhook-service-name=cert-manager-approver-policy
      --webhook-ca-secret-namespace=cert-manager
    State:          Running
      Started:      Fri, 28 Jun 2024 14:46:09 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4096M
    Requests:
      cpu:        500m
      memory:     1024M
    Readiness:    http-get http://:6060/readyz delay=3s timeout=1s period=7s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from temp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ppm55 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  temp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-ppm55:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

/cc @erikgb

/ok-to-test

Thanks for this @rgodha! The new default values seem to be aligned with the cert-manager defaults.
/lgtm
/cc @inteon


HW Accelerated JPEG encoder for X86 and DS 6.3

According to the changelog, DS 6.3 introduced an HW-accelerated JPEG encoder.

CHECK:

HW JPEGDEC (JETSON) CHECK
ENC IDR FRAMES

NVidia changed docker container set - https://catalog.ngc.nvidia.com/orgs/nvidia/collections/deepstream_sdk. We have to double-check our builds. It might be better to build a completely custom docker image based on GStreamer and DS SDK, including only what we need.

The bug with jpeg decoding via nvv4l2decoder is valid for DS 6.3

The current Savant code tries to use nvenc first. If the user specified output_frame: jpeg, Savant will try to build the encoder using nvjpegenc and will only use jpegenc if nvjpegenc is not available (https://github.com/insight-platform/Savant/pull/44). Thus, there is nothing to do.


Memory leak in example project

You might want to review this part of your example project:
var Commits = require('../../collections/commits');

exports.index = function(req, res, next) {
  var commits = new Commits(null, {
    owner: 'artsy',
    repo: 'flare'
  });
  commits.fetch({
    success: function() {
      res.locals.sd.COMMITS = commits.toJSON();
      res.render('index', { commits: commits.models });
    },
    error: function(m, err) { next(err.text); }
  });
};

We followed your example and stumbled across some serious problems concerning memory consumption for high traffic apps. Creating Backbone collections for every request and using them to fetch (and rendering) data seems to be problematic, because the GC won't clean them up and the Node process will allocate more and more memory.

I haven't yet fully understood if this is an Backbone issue or maybe caused by your backbone-super-sync module.

So I believe I traced down the main cause of the memory leak in backbone-super-sync
. Turns out Backbone.sync is, unsurprisingly, an easy bottle neck for leaky memory. After a lot of trial and error I came to the conclusion that a combination of forcing timeouts on requests and refactoring out closures solved the issue—at least for various applications I tested on. Here's the code for details https://github.com/artsy/backbone-super-sync/pull/33
Right now one of our Ezel applications has a flat memory graph.

Another has very spikey memory use but it appears to at least be freeing large chunks of memory. (My hunch for the cause of this is we have some routes where we load a large amount of data into Backbone collections (e.g. iterates though an API endpoint that returns paginated array data and we "fetch until end" loading thousands of entities in memory).

I tested this on the example project in Ezel and pinged it at ~4k rpm on a 512mb heroku instance and while performance would degrade (probably b/c of heavy GCing) it did hover just under the max RAM.
So :pray: the leak is fixed here, but there's still probably more wins to be had in terms of memory hungriness. Any thoughts and comments of course are much appreciated! I'll call this one closed for now.

Actually not convinced yet... keeping this open.

Ok I'm fairly convinced this is solved for now, see https://github.com/artsy/backbone-super-sync/pull/35


fix(curriculum): update array iteration with for loop challenge (Basic Javascript)

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

Closes #41741
Fixes curriculum's Iterate Through an Array with a For Loop lesson.
Corrects a typo in the array iteration example and adds an explanation on how the termination of the loop is triggered.

Hey @RaviOnline
This PR seems to make similar changes as the existing PR #41742. As such, we are going to close this as duplicate.
If you feel you have additional changes to expand upon this PR, please feel free to push your commits and request this PR be reopened.
Thanks again! 😊

If you have any questions, feel free to ask questions on the 'Contributors' category on our forum or the contributors chat room.

Hi @gikf
Thank you for your guidance. I'm new to collaborating and your posts are very informative; appreciate it much.
I have pushed a new commit to further improve this PR by adding:

an illustration of 0-based array indexing when iterating in reverse
a point on JavaScript's undefined behavior with an out of bounds index

I'm unable to see my new commit in this PR (probably because it's closed) but the updates can be seen over here.
I request you to review this PR again and reopen if it meets the expectations.
Regards,
Ravi


Installation error

I cloned the repo to my machine but I'm running on php 8 which installed automatically by running the command on my Linux terminal,I don't know if this project can be run by php 8.If yes how can I do that.And if it's php 7,how can I install php7 on my linux machine.Thanks,waiting,this project is so cool and I'd like to run it today

Please check other tutorials to install PHP 7.


Move the filter pills below the search and filter row in the inventory seedlings tables

Current dependencies on/for this PR:

main

PR #1919   👈

This stack of pull requests is managed by Graphite.


Add sensible error messages when assistant fails.

Looks good to me

Thanks @Sallaa, yes it should be ready to be merged.


Deny unused crates

This adds #![deny(unused_crate_dependencies)] to lib.rs and then does the cleanups required. Lets us drop a couple of dependencies altogether, and move a couple others to optional.

Thanks!


Fix interrupt handler for endpoints > 2

The IRQ handler calculated the endpoint number wrong when handling IN and OUT tokens.  The errant code worked by accident for endpoints 1 and 2 but was incorrect for other addresses.  For endpoints 5 and above it resulted in stray pointer read/writes that could crash the device.

Can you please sign : https://developer.mbed.org/contributor_agreement/ ?

Hi - I already signed that a couple of years ago.
Regards,Mike
Date: Mon, 4 Jan 2016 01:34:11 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [mbed] Fix interrupt handler for endpoints > 2 (#1481)
Can you please sign : https://developer.mbed.org/contributor_agreement/ ?
—
Reply to this email directly or view it on GitHub.

@mjrgh I could not locate username mjrgh in mbed, please share it to verify

Hmm... strange.  When I go to the contributor agreement page, it doesn't offer an option to sign, it just says "You accepted the Contributor Agreement on Tue 05 Aug 2014".

please share it to verify
Not sure what you mean - could you be more specific?
Regards,Mike
Date: Mon, 11 Jan 2016 02:32:35 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [mbed] Fix interrupt handler for endpoints > 2 (#1481)

@mjrgh I could not locate username mjrgh in mbed, please share it to verify
—
Reply to this email directly or view it on GitHub."

Oh, by the way, I think my previous changes to the same code were pushed under my mbed user ID, "mjr" - the new ones are on my github account, which might be why they don't appear to be connected.
Regards,Mike
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: RE: [mbed] Fix interrupt handler for endpoints > 2 (#1481)
Date: Mon, 11 Jan 2016 20:46:55 +0000
Hmm... strange.  When I go to the contributor agreement page, it doesn't offer an option to sign, it just says "You accepted the Contributor Agreement on Tue 05 Aug 2014".

please share it to verify
Not sure what you mean - could you be more specific?
Regards,Mike
Date: Mon, 11 Jan 2016 02:32:35 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [mbed] Fix interrupt handler for endpoints > 2 (#1481)

@mjrgh I could not locate username mjrgh in mbed, please share it to verify
—
Reply to this email directly or view it on GitHub."

Thanks, for sharing your mbed username. And for these fixes, I noticed you got some other fixes on mbed in your fork :+1:

Great - thanks!
Regards,Mike
Date: Thu, 14 Jan 2016 23:44:14 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [mbed] Fix interrupt handler for endpoints > 2 (#1481)
Thanks, for sharing your mbed username. And for these fixes, I noticed you got some other fixes on mbed in your fork
—
Reply to this email directly or view it on GitHub.

By the way, I've done a more extensive overhaul of the KL25Z USB HAL implementation, which I haven't entered a pull request for, because the changes are just too big.  I'm working on a project that involves multiple endpoints (that's how I ran into the bugs that these past couple of pull requests were about), and that exposed a bunch of race conditions in the KL25Z HAL code.  The race conditions mostly manifest as random USB disconnects.  Most of the race condition problems are apparent from a careful reading of the code, and they'll affect all existing users of the code, but the probability seems low enough for most simple applications that I guess no one has noticed them until now.
The race conditions strike randomly, of course, and they seem to happen quite rarely in simple code with one non-control endpoint.  The previous version of my project used a simple USB joystick interface with one non-control endpoint, and the connection was mostly stable.  I did find it necessary to put in 1ms waits around all of my USB read/write operations, which I previously chalked up to hardware bugs, but I now realize the race conditions were the real culprit.  The 1ms waits helped simply because they slowed things down and thereby reduced the probability of a USB interrupt coming in while the read/write call was in progress.
I think I've made everything deterministic and properly serialized in the overhaul.  I added critical sections around all (hopefully!) of the code that touches shared statics and that can run from application context.  I also changed the way endpoint un-stall is handled - those didn't work reliably in the official version, so a stall would generally kill a connection even though it shouldn't have.
My test case seems rock-solid with the changes.  I was able to trigger disconnects within a matter of minutes with the original code, but the new version runs for days at a time without every disconnecting.
If you're interested in taking a look, I published my fork here:
http://mbed.org/users/mjr/code/USBDevice/
I'm sure the changes are too extensive to just drop into the official library, but I thought I'd share it in case you or other of the platform developers want to take a look at it and consider adapting it.  And in any case, the fork is there in case anyone has similar problems and wants to try this version to see if it helps.
Regards,Mike
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: RE: [mbed] Fix interrupt handler for endpoints > 2 (#1481)
Date: Fri, 15 Jan 2016 18:44:14 +0000
Great - thanks!
Regards,Mike
Date: Thu, 14 Jan 2016 23:44:14 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [mbed] Fix interrupt handler for endpoints > 2 (#1481)
Thanks, for sharing your mbed username. And for these fixes, I noticed you got some other fixes on mbed in your fork
—
Reply to this email directly or view it on GitHub.

@mjrgh Users will benefit from those bugfixes in the upstream. If you can pack those changes as patches with bug fixes, as this one, would be nice. Thanks for sharing all this devel
cc @sg-


Clarify Makefile default tasks

Addresses https://github.com/tangrams/tangram-es/issues/447
make clean will currently target all platforms, which includes the Gradle steps for Android, leading to confusion for new users when Gradle starts installing and running.
This changes make clean to be a no-op that simply lists the likely tasks that the user actually wants to run. Similarly, make now lists all of the platform targets instead of going straight into the android build.

I am fine with this, because we never build/clean all target at once.

Closed in favor of https://github.com/tangrams/tangram-es/pull/452


Invalid import (BuildConfig) in BufferooApplication.kt

There is invalid import in BufferooApplication.kt
import android.support.v4.BuildConfig

Oh, sorry. Already requested


Fixes errors when not in diversity mode

Good catch! I'm going to make a few changes so it will fall in line with the oled version as well.

Ah yeah my oled screen is being shipped, did not try that version yet. Any plans to include the pcb files to the repo as well?


Feature - Pass an array to disable multiple controls

Add the possibility to pass an array to disable multiple controls

Useful 👍


tf.GradientTape.gradients() does not support graph control flow operations like tf.cond or tf.while at this time. Use tf.gradients() instead. If you need this feature, please file a feature request at https://github.com/tensorflow/tensorflow/issues/new

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with the
documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go
here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script
provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
happens on a mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Can you fill the ISSUE template with all the info required? Thanks

Duplicate issue : #36581 #31504
Could you please fill the issue template.Also Please provide the simple standalone code/ colab link to reproduce the issue at our end.Thanks


update the vr condition

this commit updates the Volume Replication(VR) conditions to
include descriptive message for every
operations
Example of the VR status:
Status:
  Conditions:
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:              failed to demote
    Observed Generation:   3
    Reason:                FailedToDemote
    Status:                False
    Type:                  Completed
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:          Volume is degraded
    Observed Generation:   3
    Reason:                Error
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2024-10-15T04:19:17Z
    Message:            failed to resync
    Observed Generation:   3
    Reason:                NotResyncing
    Status:                False
    Type:                  Resyncing
  Last Completion Time:    2024-10-15T04:19:18Z
  Message:                 volume 0001-0009-rook-ceph-0000000000000002-07d77e2c-5689-4493-b583-cf06c92e1db1 not found: Failed as image not found (internal RBD image not found)
  Observed Generation:     3
  State:                   Primary
Events:                    <none>

It would be good to have an example in the PR description. Ideally do not abbreviate things like "vr" in commit subjects.

@ShyamsundarR I have updated the pr to include specific errors for failed conditions, do take a look

The following comments have been addressed in the above PR:

Error message from ceph csi is included in conditions of VR so that it can be picked up by VRG
Error messages have been make const

The VR after this change looks like below:
Status:
  Conditions:
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:              failed to demote
    Observed Generation:   3
    Reason:                FailedToDemote
    Status:                False
    Type:                  Completed
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:          Volume is degraded
    Observed Generation:   3
    Reason:                volume 0001-0009-rook-ceph-0000000000000002-07d77e2c-5689-4493-b583-cf06c92e1db1 not found: Failed as image not found (internal RBD image not found)
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2024-10-15T04:19:17Z
    Message:           volume is not resyncing
    Observed Generation:   3
    Reason:                NotResyncing
    Status:                False
    Type:                  Resyncing
  Last Completion Time:    2024-10-15T04:19:18Z
  Message:                 volume 0001-0009-rook-ceph-0000000000000002-07d77e2c-5689-4493-b583-cf06c92e1db1 not found: Failed as image not found (internal RBD image not found)
  Observed Generation:     3
  State:                   Primary
Events:                    <none>

cc @nirs @Madhu-1 @nixpanic @ShyamsundarR  please add your reviews to the changes

this commit updates the Volume Replication(VR) conditions to include descriptive message for every operations
Example of the VR status:
Status:
  Conditions:
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:              failed to demote
    Observed Generation:   3
    Reason:                FailedToDemote
    Status:                False
    Type:                  Completed
    Last Transition Time:  2024-10-15T04:21:29Z
    Message:          Volume is degraded
    Observed Generation:   3
    Reason:                Error
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2024-10-15T04:19:17Z
    Message:            failed to resync
    Observed Generation:   3
    Reason:                NotResyncing
    Status:                False
    Type:                  Resyncing
  Last Completion Time:    2024-10-15T04:19:18Z
  Message:                 volume 0001-0009-rook-ceph-0000000000000002-07d77e2c-5689-4493-b583-cf06c92e1db1 not found: Failed as image not found (internal RBD image not found)
  Observed Generation:     3
  State:                   Primary
Events:                    <none>

This example does not reflect the actual change.

Looks like we create useful message now, but it is not clear to me when and why messageFromDriver apply to which condition.
Can you share example VR status in all the error cases? It can be simulated by faking failures, or reproducing failure conditions on a real system (e.g. trying to replicate a cloned pvc with the wrong dr policy).
If you provide an image, I can test this in ramen testing env.

sure, let me create an image and share with you

I install the image provided by @yati1998 using this:
https://github.com/RamenDR/ramen/pull/1639
It passed ramen e2e job:
https://github.com/RamenDR/ramen/actions/runs/11683242163/job/32532029150
I'm testing the message for validation issue now.

Changes to reflect message in condition to include errors from the RPCs is good, moves it closer to the condition that faced the error.
This still does not catch certain errors as @yati1998 pointed out, e.g MirrorEnable returns the -22 and no description, so we miss the context further that the error was actually something like this:
$ rbd mirror image enable ocs-storagecluster-cephblockpool/csi-vol-7c561e23-151b-4f8d-a219-41abd252d713 snapshot
2024-10-25T13:56:25.349+0000 7f3006be3c00 -1 librbd::api::Mirror: image_enable: image has a parent, snapshot based mirroring is not supported

Which is being addressed in https://github.com/ceph/ceph-csi/pull/4941 we may need other corner cases as it may arise to report better failures/errors. Otherwise the move to report known errors and failures in messages with this PR looks good.
(not adding a +1 as I am not looking at specific of the change overall)

Changes to reflect message in condition to include errors from the RPCs is good, moves it closer to the condition that faced the error.
This still does not catch certain errors as @yati1998 pointed out, e.g MirrorEnable returns the -22 and no description, so we miss the context further that the error was actually something like this:
$ rbd mirror image enable ocs-storagecluster-cephblockpool/csi-vol-7c561e23-151b-4f8d-a219-41abd252d713 snapshot
2024-10-25T13:56:25.349+0000 7f3006be3c00 -1 librbd::api::Mirror: image_enable: image has a parent, snapshot based mirroring is not supported

Which is being addressed in ceph/ceph-csi#4941 we may need other corner cases as it may arise to report better failures/errors. Otherwise the move to report known errors and failures in messages with this PR looks good.
(not adding a +1 as I am not looking at specific of the change overall)

Hey @ShyamsundarR  this error was updated further, I think you are using older version of cephcsi, due to which it seems just the error code is returned. https://github.com/ceph/ceph-csi/pull/4678/files#diff-d3713dcbb075d877c69230700478ec60c5fe029d742dae0f72499d9f6833617f

Hey @ShyamsundarR this error was updated further, I think you are using older version of cephcsi, due to which it seems just the error code is returned. https://github.com/ceph/ceph-csi/pull/4678/files#diff-d3713dcbb075d877c69230700478ec60c5fe029d742dae0f72499d9f6833617f

Quite possible, the builds in use were downstream and possibly ones in release-4.16. Overall, the new PR to improve the error message from Ceph-CSI is helpful and relates to this PR, hence cross posting it here.

I does not work. I deployed a deployment using pvc restored from snapshot, and tried to protect is. This is expected to fail since the dr policy is not enabled for flattening. https://github.com/nirs/ocm-ramen-samples/tree/flatten/workloads/flatten/regional-dr
I get empty messages for all conditions instead of the expected error message.

The test was wrong, we used the manifests form 0.10.0. Testing again with the manifests from this branch and a new version of the image:
https://github.com/RamenDR/ramen/compare/c6135d3dcbf198f302c1bc1f98ebd6a4bb2e8e2e..d65c6eb9ed478c3affd8b18adb061094c078798b

Works now with fixed ramen change:
https://github.com/RamenDR/ramen/pull/1639/commits/c496fe945f5d33a42753622d2f4f6bfa5de3c10b
% kubectl get vr -A --context dr1 -o yaml
apiVersion: v1
items:
- apiVersion: replication.storage.openshift.io/v1alpha1
  kind: VolumeReplication
  metadata:
    creationTimestamp: "2024-11-05T13:42:41Z"
    finalizers:
    - replication.storage.openshift.io
    generation: 1
    labels:
      ramendr.openshift.io/owner-name: flatten-drpc
      ramendr.openshift.io/owner-namespace-name: ramen-ops
    name: restored-pvc
    namespace: flatten
    resourceVersion: "40696"
    uid: 9bf65b35-b3da-42dc-a9c0-fbb6a3927b1b
  spec:
    autoResync: false
    dataSource:
      apiGroup: ""
      kind: PersistentVolumeClaim
      name: restored-pvc
    replicationHandle: ""
    replicationState: primary
    volumeReplicationClass: vrc-sample
  status:
    conditions:
    - lastTransitionTime: "2024-11-05T13:42:41Z"
      message: failed to promote volume
      observedGeneration: 1
      reason: FailedToPromote
      status: "False"
      type: Completed
    - lastTransitionTime: "2024-11-05T13:42:41Z"
      message: failed to enable volume replication
      observedGeneration: 1
      reason: Error
      status: "True"
      type: Degraded
    - lastTransitionTime: "2024-11-05T13:42:41Z"
      message: volume is not resyncing
      observedGeneration: 1
      reason: NotResyncing
      status: "False"
      type: Resyncing
    - lastTransitionTime: "2024-11-05T13:42:41Z"
      message: 'failed to meet prerequisite: rpc error: code = FailedPrecondition
        desc = system is not in a state required for the operation''s execution: failed
        to enable mirroring on image "replicapool/csi-vol-f4737b6e-eeff-4137-8248-301cf37a3368":
        parent image "replicapool/csi-snap-e7c91292-a272-4278-9ee9-6be7a4c8bfe0" is
        not enabled for mirroring'
      observedGeneration: 1
      reason: PrerequisiteNotMet
      status: "False"
      type: Validated
    message: 'system is not in a state required for the operation''s execution: failed
      to enable mirroring on image "replicapool/csi-vol-f4737b6e-eeff-4137-8248-301cf37a3368":
      parent image "replicapool/csi-snap-e7c91292-a272-4278-9ee9-6be7a4c8bfe0" is
      not enabled for mirroring'
    observedGeneration: 1
    state: Unknown
kind: List
metadata:
  resourceVersion: ""

The duplicate .Status.message can be eliminating, but maybe someone is depending on this?

The duplicate .Status.message can be eliminating, but maybe someone is depending on this?

Yes we can't eliminate that.

since @nirs has also tested the change is statisfied with the current modifications, I would request a final review on this.
I have addressed all the comments above .
cc @nixpanic @nirs @ShyamsundarR @Madhu-1

@nirs @nixpanic @ShyamsundarR can you please give your final review here?


Support ulysses flash attn

Deepspeed Ulysses was proposed by Microsoft in DeepSpeed Ulysses , which implements a set of sequence parallelism using all-to-all communication, a method more efficient than p2p communication. Based on the Ulysses attention implemented in long-context-attention, I have added 'ulysses_attn' to the Easy Context framework on top of the three other parallel modes and have completed its implementation. According to experiments, the training results for long contexts are identical to those of ring attention, but the training speed is faster. When using it, please also clone the necessary folders from long-context-attention, which is consistent with the way you use ring attention.

This is awesome!


[BUG] Handle error rending (default) array of strings

Is there an existing issue for this?

[X] I have searched the existing issues

Does this issue exist in the latest version?

[X] I'm using the latest release

Describe the bug?

RENDERS

To Reproduce
See tj-actions/remark@06565c7 (#38)
What OS are you seeing the problem on?
all
Expected behavior?
Uses a list of items that are properly escaped
Relevant log output
No response
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Replace "\n" with ""


Clarify information pertaining to volunteering perks

Called out no travel+accommodation support.
Invite everyone to volunteer even if they already hold a ticket

@thepetk @Vaibhavs10 @hypha please review. :)


is there a client side release ?

Hi @danigb
cross-posting from this issue  : https://github.com/danigb/tonal.chord/issues/1
as I'm a bit confused, tonal-chord was usable client side, but it seems that tonal is nodejs based and supposed to run server-side am I right ? is there a client-side distribution ?

Yes, you can use the one from 'dist' folder
El jue., 19 ene. 2017 18:36, b2renger<EMAIL_ADDRESS>escribió:

Hi @danigb https://github.com/danigb
cross-posting from this issue : danigb/tonal.chord#1
https://github.com/danigb/tonal.chord/issues/1
as I'm a bit confused, tonal-chord was usable client side, but it seems
that tonal is nodejs based and supposed to run server-side am I right ? is
there a client-side distribution ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/danigb/tonal/issues/16, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAdIdJzvv1VTApw6I08GLrN77Ax4CjDks5rT58IgaJpZM4LoZCz
.

I saw them but I cannot seem to get them working should I call an init function ?
I should be able to do something like :
chord('B7');

right away ? but I'm getting a "Uncaught ReferenceError: chord is not defined"

Is exported as Tonal:
Tonal.chord('B7')

I hope it helps. Probably I should update the README.

Great ! a capital 't' it is :)  and now I see where it is in the code : thanks !
There is a lot more functionalities here, it's quite impressive 1st_place_medal:

Thanks! Any suggestion, issue or PR are more than welcome.


Debug integration test failure

Retries were added in 5cf50df3be4ea65e01da16e9e6d6e0f674d019d4, this was not enough apparently.
  169 passing (1m)
  1 pending
  1 failing
  1) Debug start debugging:
     AssertionError [ERR_ASSERTION] [ERR_ASSERTION]: S {
  _configuration: {
    __pendingTargetId: '8889',
    __workspaceCachePath: '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/t8732wpx0QdV4rB56/d/data/User/workspaceStorage/1337f4a9/ms-vscode.js-debug-nightly',
    __workspaceFolder: '',
    address: 'localhost',
    disableNetworkCache: true,
    name: '[8889]',
    outFiles: [
      '/Users/runner/runners/2.165.0/work/1/s/extensions/vscode-api-tests/testWorkspace/**/*.js',
      '!**/node_modules/**'
    ],
    outputCapture: 'console',
    pathMa... == undefined
      at Context.<anonymous> (extensions/vscode-api-tests/src/singlefolder-tests/debug.test.ts:42:10)
      at processImmediate (internal/timers.js:439:21)

https://monacotools.visualstudio.com/DefaultCollection/Monaco/_build/results?buildId=69274&view=logs&j=a5e52b91-c83f-5429-4a68-c246fc63a4f7&t=087c9a82-9b9b-5b5f-17f2-a070cb4808e3

I really think this test should leverage mock debug and not require a process to start.

The problem was that the re-retry did not clear the previous state.
Now we should be good. Fixed via c91e2b62d7
If it keeps on failing I will look into using mock debug


Boot homeassistant dependency issue

It just appened twice, never happened before in one year with the same setup.
I have an hassio installation with:

pi-hole
mosquitto

Addons installed.
I had to reboot the pi and when it went up again I saw in the web panel that a lot of my sensors, connections failed.
I restarted home assistant and everything was up and running.
It was the first time it happened. I tryed to reboot the pi again for seeing if the same happened again and yes. It happened again.
I removed the pi-hole add on for testing, reboot and it happened again with all the mqtt things.
It looks like hass container starts before, or get ready before, the others container with the server side third parties I use.
It means that I was lucky in the past having the same software starting correctly?
There is some way for verifying that the containers starts in a predefined order and change it pushing mqtt before hass?
I mean... if it is a random startup there is a high possibility to hit a wrong order.
What if there is a power failure, an unexpected reboot, and then the system goes up again?
There is a possibility that hass / homeassistant must be restarted by hand because of a wrong startup order?
I don't know if this is the right place for writing this.
I see two point of failure, one is this. Hassio could be able to prioritize by hand, or by code that one container is up before starting another one?
Or is the home assistant "startup" that needs to see all the single parts connected? If the network provider needs more time than expected for any reason and home assistant finish powering up it display the error message and there is no way for trying another connection / try again to initialize the module. The only possible thing to do is restart the container.
If it power up in the right order, everything works as expected.
But if something is out of order a restart of the single container is needed for being fully operational again.

Hi @pvizeli
thank you for your reply.
It is, anyhow, someway obscure to me. Can you please describe where I have to act?
From the hassio panel on home assistant web interface?

@andyinno Manually, via the Hassio API is currently the only way.
Since this issue is stale for a while now, is this still a problem for you?

Hi @frenck
Telling the truth? I was not able to fix the issue, I "resolved" by installing pi-hole on another pi I had half used. After an host update mosquitto started without errors and now after a reboot it starts correctly.
For me the issue is closed


Add quota/resource limit information to the "Restrictions" section

Hello!
While working with the Disk Encryption Sets, it was identified that the default per-subscription limit of Disk Encryption Sets is 50. If you try to deploy more than 50 sets, it will give you this error:

I have submitted feedback on the quota page (that is a separate GitHub issue), but thought it was strange that was omitted from this doc as well.
Could this page (and the Windows version as well) have something added about the default resource limit of 50?
I feel like this would fit under the section talking about Restrictions.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 143bdfce-bcca-f901-c996-e22642a37561
Version Independent ID: 384e5861-97da-a89a-39b2-48e1f992ab61
Content: Server-side encryption of Azure Managed Disks - Azure CLI - Azure Linux Virtual Machines
Content Source: articles/virtual-machines/linux/disk-encryption.md
Service: virtual-machines-linux
Sub-service: disks
GitHub Login: @roygara
Microsoft Alias: rogarana

#assign: roygara

@nickmcmahan Thanks for bringing this to our attention, I checked with the product team and they say it's actually 50 per region per subscription. Accordingly, we've updated both articles to include this information. The changes will be live later today. :)
As always, we appreciate your help improving our docs! :)
#please-close


Would be really great if moment.js had a precise setInterval function

According to this SO page (not the best source, I know), setInterval and a naive setTimeout recursion can drift. I see on your home page you have a clock showing the current time (lines 82-85 here).

I put this together over the last few days https://github.com/SeverinDK/moment-timer
Think that's what you guys were talking about.


getMessage().getSendTime(); not working?

time = e.getMessage().getSentTime();
Can't seem to get the time throught this method, instead it just produces a long value (in my case,<PHONE_NUMBER>234).
Anybody else having the same problem or am I doing something stupidly wrong?

If I m not wrong this long is the time. In millisecond. Like a lot of person use it in computer science

Hello, you can convert millisecond to date.  See this topic


Added Brand and Title for the SearchItems endpoint

This is just a simple addition.
I think it might make sense to refactor the query parameter and have all the search fields (brand, title, keywords; but also the currently not implemented fields) in an extra object and flatten the whole thing for sending the query to amazon. Let me know if you want me to start doing that.

We only send values to amazon that are not NULL.
this._jsonSerializerSettingsRequest = new JsonSerializerSettings
{
    NullValueHandling = NullValueHandling.Ignore
};

I was more thinking about API usability. Having general settings in one place, and having the actual search fields in another. But I don’t mind either way.


i18n(ko): Translation of getting-started and scaffolding stuff

What kind of changes does this PR include?

Translated content

Description

What does this PR change? Give us a brief description.

translated src/pages/ko/getting-started.mdx and add src/i18n/ko directory

Thanks for this @imjlk, and welcome to Team Docs! 🙌


Get asset data address

Check for existing issues

[X] Completed

Describe the feature
We want to add an endpoint that takes an assetId and returns its related data address
If applicable, add text / mockups / screenshots to help present your vision of the feature
No response

@OlfaBensoussia, this would be awesome! Is there already an endpoint for it, or is something will come up next in the Connector?

Not needed anymore because on /v3/assets endpoint the DataAddress is part of the Asset


merge Function Literals and Lambda sections

These sections duplicated much of the grammar and text, implying Function Literals are somehow different from Lambdas, when in fact the only difference is slightly shorter syntax. The PR merges them into one grammar.

I also added handy links to function_literals from other files that referenced lambdas.


Rare Ice being skipped. incorrectly being classified as client mod

Describe the problem

Skipping rare-ice-0.6.0.jar since it is a client mod

when using https://www.curseforge.com/minecraft/modpacks/faehaven/files/5687725
Rare Ice mod is skipped and incorrectly marked as clientside only.
https://www.curseforge.com/minecraft/mc-mods/rare-ice
Container definition
name: Faehaven
services:
  Faehaven:
    container_name: Faehaven
    hostname: Faehaven
    image: itzg/minecraft-server:latest
    environment:
      MOD_PLATFORM: AUTO_CURSEFORGE
      CF_PAGE_URL: https://www.curseforge.com/minecraft/modpacks/faehaven
      MEMORY: 8G

Container logs
Faehaven  | [mc-image-helper] 02:53:49.106 DEBUG : FILE FETCH: uri=https://edge.forgecdn.net/files/5374/432/CreateNumismatics-1.0.6%2bforge-mc1.20.1.jar headers=[user-agent: itzg/mc-image-helper/1.39.11 (cmd=install-curseforge), x-fetch-session: 00d3717c-63ef-49ab-8824-1bf4293864aa, x-api-key: [redacted], host: edge.forgecdn.net, accept: */*]
Faehaven  | [mc-image-helper] 02:53:49.107 DEBUG : JSON FETCH: uri=https://api.curseforge.com/v1/mods/938667 headers=[user-agent: itzg/mc-image-helper/1.39.11 (cmd=install-curseforge), x-fetch-session: 00d3717c-63ef-49ab-8824-1bf4293864aa, x-api-key: [redacted], accept: application/json, host: api.curseforge.com]
Faehaven  | [mc-image-helper] 02:53:49.107 DEBUG : Skipping rare-ice-0.6.0.jar since it is a client mod
Faehaven  | [mc-image-helper] 02:53:49.107 DEBUG : JSON FETCH: uri=https://api.curseforge.com/v1/mods/551520 headers=[user-agent: itzg/mc-image-helper/1.39.11 (cmd=install-curseforge), x-fetch-session: 00d3717c-63ef-49ab-8824-1bf4293864aa, x-api-key: [redacted], accept: application/json, host: api.curseforge.com]

Issue with how mod files are marked on curseforge. client environment only visible in legacy view on CF.