Adding a SubmodelElementCollection to multiple Shells

Hi! I'd like to ask your input on how to handle a set of repetitive data when creating a series of shells.
We created the submodel element collection "contact information", which will stay the same for all our products. Since we can't add one SMC to multiple AASs we presumably need to add it as reference, which we were unable to do so far.
    contact_information = ContactInformation.create(
        street=os.getenv("AAS_STREET"),
        zip=os.getenv("AAS_ZIP"),
        city_town=os.getenv("AAS_CITY"),
        national_code=os.getenv("AAS_NATIONAL_CODE")
    )
    nameplate = DigitalNameplate.create(
            str(identifier),
            uri_of_the_product=str(uri),
            manufacturer_name=str(element.xpath(xpaths['ManufacturerName'])[0]),
            contact_information=contact_information,
            manufacturer_product_designation=str(manuproddesi),
            year_of_construction=str(element.xpath(xpaths['YearOfConstruction'])[0]),
            manufacturer_article_number=str("4242")
        )

        class DigitalNameplate:
            @classmethod
            def create(cls, identifier: str, uri_of_the_product: str, manufacturer_name: str,
                       contact_information: SubmodelElementCollection, manufacturer_product_designation: str,
                       year_of_construction: str, manufacturer_article_number: str) -> Submodel:

               #....

        submodel.submodel_element.add(contact_information)

        return submodel

    aas = model.AssetAdministrationShell(
            asset_information=asset_information,
            id_=str(aas_identifier),
            id_short=aasx.NameFriendlyfier().get_friendly_name(myAAS.get_submodel_property_value(nameplate, "ManufacturerProductDesignation")),
            submodel={model.ModelReference.from_referable(nameplate)},
        )

        object_store.add(aas)
        object_store.add(nameplate)

Is the ContactInformation SubmodelElementCollection intended to be freshly instantiated and copied in each of the Submodels?

As far as I know, there is at the moment no way to globally reference a SubmodelElementCollection (as in across multiple Submodels) within the model itself.
I therefore see multiple options:

Put the contact information in its own globally Identifiable Submodel with nothing else and reference that Conctact InformationSubmodel in multiple different AASs
If you are using a Server, using the official AAS API, you may be able to use a ReferenceElement that contains the Submodel Identifier and the idShort-Path to the SubmodelElementCollection (I know very little about that, so I can't say anything about how that would work, or if this is expected)

However, if you want to use and follow a standardized submodel template, I'm afraid instantiating the SubmodelElementCollection each time anew is the only way to be compliant to the template. Programmatically you may want to have the contact information in one point only (single source of truth), but when you offer the AASs to the outside, you must give a copy each time.
I hope that helps.


strconv: ParseFloat seems not working correctly with a large exponent

What version of Go are you using (go version)?

$ go version
go version go1.15.4 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/hiroshi/.cache/go-build"
GOENV="/home/hiroshi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/hiroshi/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/hiroshi/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/hiroshi/Downloads/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/hiroshi/Downloads/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build943543493=/tmp/go-build -gno-record-gcc-switches"

What did you do?
Run the below program (Go Playground: https://play.golang.org/p/BKUGd_uDl9_Q)
fmt.Println(strconv.ParseFloat("0."+strings.Repeat("0", 100000)+"9e100000", 64)) // 0 <nil>
fmt.Println(strconv.ParseFloat("9"+strings.Repeat("0", 100000)+"e-100000", 64))  // 0 <nil>

What did you expect to see?
I assume the first line should print 0.9 <nil> and the second should print 9 <nil>.
What did you see instead?
I see the below output.
0 <nil>
0 <nil>

probably to do with https://golang.org/src/strconv/atof.go#L289 and the comment above:

// optional exponent moves decimal point.
// if we read a very large, very long number,
// just be sure to move the decimal point by
// a lot (say, 100000).  it doesn't matter if it's
// not the exact number.


Optimized readVizgen()

Hi
as discussed in #3 I did a major update for readVizgen() to handle older and the newest Vizgen/Merscope output data, as well as .filter_polygons()

New optional args, they are to FALSE as default, not sure what would be the best?

filter_counts = FALSE # if to keep cells with counts > 0, this will subset polys as well.
add_molecules = FALSE # if to add molecule/transcript coords to rowGeometries
use_bboxes = FALSE # if no segmentation output present, use cell_metadata to make bounding boxes instead

I tested it on few old-new datasets we have here, all went fine on my side.
Only plotting including the image_id has a small issue with -> Voyager:::geom_spi_rgb(., max_col_value = 255), I will open a separate issue for Voyager on this.
I use tidyverse and pipe operator from magrittr a lot when implementing, hope that's ok.
I will work further on converter from/to SFE <-> Seurat, is utils.R the proper place to add it?
This is the run example:
# install this PR
devtools::install_github(repo = "alikhuseynov/SpatialFeatureExperiment", ref = "seurat_v4")

# load libs ----
suppressPackageStartupMessages({
  library(ggplot2)
  library(Seurat)
  library(dplyr)
  library(magrittr)
  library(BiocParallel)
  library(scuttle)
  library(SpatialFeatureExperiment)
  library(Voyager)
  library(sf)
})

# set directory to read from
dir_use <- "./merfish_seurat_debug/" # or something like "./region_0/"

message("Reading data from:" , "\n", dir_use)
gc() %>% invisible()

start.time <- Sys.time()
sfe <-
  readVizgen(data_dir = dir_use,
             z = 3L,
             sample_id = "test.merfish",
             min_area = 15,
             image = "DAPI",
             flip = "geometry",
             max_flip = "50 MB",
             filter_counts = TRUE, # keep cells with counts > 0, this will subset cell segmentation as well
             add_molecules = TRUE, # add molecule/transcript coords to `rowGeometries`
             use_bboxes = TRUE, # if no segmentation output is present, use `cell_metadata` to make bounding boxes
             BPPARAM = BiocParallel::MulticoreParam(14,
                                                    tasks = 80L,
                                                    force.GC = FALSE,
                                                    progressbar = TRUE)
  )
end.time <- Sys.time()
message("Time taken to read data = ",
        round(end.time - start.time, digits = 2), " ",
        attr(c(end.time - start.time), "units"))
sfe

Thank you for the PR! I can change the magrittr pipe to the base pipe so I don't introduce another dependency. The PR will go into the Bioc 3.18 version of Voyager, and R 4.3 is required anyway so the base pipe should be available. Please put the Seurat conversion function in the coerce.R file where the SCE and SPE conversion functions are. After reviewing the PR, I might add you as an author of the Voyager paper before (re)submitting it.

Sounds great, thanks! 👍
I did it on R 4.3.1, can change the magrittr pipes in this PR to base pipe operator, no problem. If you have few Vizgen datasets for independent testing on your side, would be great.

It seems that base pipe |> does not like curly braces.
function '{' not supported in RHS call of a pipe
Also, on R pipes comparison youtu.be video, the execution speed of base pipe is slower than with magrittr especially when using it with dplyr or tidyverse.
I think adding magrittr would be the easiest solution, what do you think? Else one would need to change all the code to base-like style. It just makes it easier to follow the code with pipe operator, especially when it's linear.
And, in base pipe, I could not find magrittr equivalent backward assign pipe %<>% which is quite handy.
Let me know what changes I would need to do here.
Thanks! 😊

While I use Tidyverse in my analyses, I usually stick to base R as far as I can in packages to avoid additional dependencies whenever possible. Unless using Tidyverse has significant performance improvement and that's important to users, I would prefer to use base R. To not to make you spend too much time on this, I can translate the code to base R.

Since dplyr and magrittr are already commonly used, it's most likely fine, though it doesn't seem too difficult to translate your code to base R, so I'll still try translating.

Ok, sounds good. Thanks for translating to base R. tidyverse and magrittr are really just personal preference.
Last commit fixed few bugs related to masked functions from different libs. It also works to load the example dataset -> system.file("extdata/vizgen", package = "SpatialFeatureExperiment") both for .parquet and .hdf5 files. Hopefully it will now pass the unit testing.
thanks

I'm reviewing your pull request and I have translated your code into base R. That said, I might end up using tidyverse or data.table to speed up the part for transcript coordinates when I test it on that 30 GB transcript spot file. I noticed that you would search recursively for files for cell metadata and transcript spots if the file is not found at top level. I based my original function on the directory structure of the example datasets on the Vizgen website, like this:

Are there other standard directory structures in which the cell metadata and transcript coordinates are inside another directory?
At this point I'm also not sure if it would be more efficient to load the transcript coordinates into memory and convert into MULTIPOINT or to convert it to raster with terra with a separate function, write it to disk, and let terra handle the on disk operations without loading the whole thing into memory. I'm not sure what use there is to load the transcript coordinates into memory. Maybe one use is to plot them with a very zoomed in image of cells. I talked to the Giotto team at the Bioconductor conference in Boston; they said converting to raster makes it faster to match transcript spots to cells.

Can you give me a schematic for the directory structure standards for the older and newest Vizgen output you mentioned? This should also include file name conventions. This is so that I can make toy test data and write unit tests for your function.

I'm reviewing your pull request and I have translated your code into base R. That said, I might end up using tidyverse or data.table to speed up the part for transcript coordinates when I test it on that 30 GB transcript spot file. I noticed that you would search recursively for files for cell metadata and transcript spots if the file is not found at top level. I based my original function on the directory structure of the example datasets on the Vizgen website, like this:
Are there other standard directory structures in which the cell metadata and transcript coordinates are inside another directory?
At this point I'm also not sure if it would be more efficient to load the transcript coordinates into memory and convert into MULTIPOINT or to convert it to raster with terra with a separate function, write it to disk, and let terra handle the on disk operations without loading the whole thing into memory. I'm not sure what use there is to load the transcript coordinates into memory. Maybe one use is to plot them with a very zoomed in image of cells. I talked to the Giotto team at the Bioconductor conference in Boston; they said converting to raster makes it faster to match transcript spots to cells.

Thanks a lot. tidyverse and data.table (then one doesn't need to specify anything in vroom call) would be the most efficient ways to load the data, that what I used for Seurat PR.
As for the transcript coordinates, this is optional and up to user to include them. If one can handle those with terra and term them into raster, that would be awesome.
Most of the times those molecule coordinates as used for plots (zoomed-in or cropped/bbox-like area). For the whole section, I would suggest to subsample them prior to plotting, using an arg like n_mols = 1000 and plot together with polygon geometries (centroids or cell boundaries).

Can you give me a schematic for the directory structure standards for the older and newest Vizgen output you mentioned? This should also include file name conventions. If the column names in the cell metadata and detected transcripts are different between versions, please provide those as well. This is so that I can make toy test data and write unit tests for your function.

Sure, yeah. Since users might have older data and new data. Here are examples of the output dirs:
# for hdf5 files
region_0/
|-- cell_boundaries
|-- cell_by_gene.csv
|-- cell_metadata.csv
|-- detected_transcripts.csv
|-- images
`-- summary.png

# this one has Cellpose dir
region_0/
|-- Cellpose
|   |-- cellpose_cell_by_gene.csv
|   |-- cellpose_cell_metadata.csv
|   |-- cellpose_micron_space.parquet
|   `-- cellpose_mosaic_space.parquet
|-- detected_transcripts.csv
|-- images
`-- summary.png

# same as above but Cellpose dir has another name
region_0/
|-- Cellpose_polyT_CB3
|   |-- cellpose_cell_by_gene.csv
|   |-- cellpose_cell_metadata.csv
|   |-- cellpose_micron_space.parquet
|   `-- cellpose_mosaic_space.parquet
|-- detected_transcripts.csv
|-- images
`-- summary.png

# the most recent version
|-- cell_boundaries.parquet
|-- cell_by_gene.csv
|-- cell_metadata.csv
|-- detected_transcripts.csv
`-- images

# some users had no segmentations
|-- cell_by_gene.csv
|-- cell_metadata.csv
|-- detected_transcripts.csv
`-- images

For cell_metadata:

transcript_count col was added,
cell ids were previously stored in ...1, new version has cell col

For detected_transcripts:

cell_id col was added to match molecule coordinates to cell ids, -1 means no assigned to a cell.

As for upcoming PRs, I can do more base R style such that you don't need to do much of additional translation from pipe magrittr tidyverse style of coding? It's just easier to follow/understand the code that way, especially where there are a number of operations. Generally I try to avoid nested coding style for that reason.

use_cellpose is no longer needed

I put it back for this reason: when running code in a separate R session, such as rendering the package website with pkgdown or running unit tests on CI, sometimes the arrow package can't be loaded, I don't know why because it would work when run interactively. I always run unit tests locally before pushing. I need to force it to use HDF5 for the remote processes to work.

I can try dusting off my C++ to stream the ginormous transcript spots file line by line without loading the whole thing into memory to construct the raster or a parquet file. In principle it can be parallelized. This way it's possible to never load the whole thing into memory. If all we need is plotting a very zoomed in region, then it shouldn't be necessary to load the whole thing. If making a heatmap of transcript density for one gene, it shouldn't be necessary to load the other genes.

Probably you already heard MoleculeExperiment
using BumpyMatrix to store molecule coords.
They make emphasis on analysis value of molecules that are not assigned to a segmented cell.
Probably BumpyMatrix could be used instead to store rowGeometries
I think, one could make it up to a user, if to load all molecules and used them for analysis, or if only for visualization, then your approach of not loading all molecule coords into memory would be enough?

Probably you already heard MoleculeExperiment, using BumpyMatrix to store molecule coords. They make emphasis on analysis value of molecules that are not assigned to a segmented cell. Probably BumpyDataFrameMatrix could be used instead to store rowGeometries? Generally, one could make it up to a user, if to load all molecules and used them for analysis, or if only for visualization, then your approach of not loading all molecule coords into memory would be enough?

I still want something that supports geometric operations, so I would prefer either sf or geoarrow or terra. One could preform point process analyses with spatstat, but that requires a spatstat's own class.

I'm still working on this pull request and I really tend to overthink. I noted that you would filter the transcript spot coordinates by z-plane, but these coordinates are actually 3D. There're also images at each z-plane. It's just that at least for the example datasets from Vizgen, the cell segmentation is the same in all z-planes. Is that always the case? Since you work with MERFISH data a lot, I wonder if you find it useful to visualize transcript spots from all z-planes or only one z-plane. I think it could be useful to have the option to load data from all z-planes.
Also, how useful do you think the cell IDs for transcript spots are? I think already having the transcript assignment to cells can help with analyses on transcript localization within each cell, but when the cell segmentation is only in one z-plane, these analyses can't get very far. If cell IDs are useful, then maybe the transcript spots should be loaded into colGeometries. Maybe I can write a function just to read the transcript spots, and give options on whether to use cell IDs and read into colGeometries or read all spots regardless of whether they're in cells into rowGeometries.
I'm not prioritizing ways to not load everything into memory for now since I have other features to work on for Bioc 3.18.

Probably you already heard MoleculeExperiment, using BumpyMatrix to store molecule coords. They make emphasis on analysis value of molecules that are not assigned to a segmented cell. Probably BumpyDataFrameMatrix could be used instead to store rowGeometries? Generally, one could make it up to a user, if to load all molecules and used them for analysis, or if only for visualization, then your approach of not loading all molecule coords into memory would be enough?

I still want something that supports geometric operations, so I would prefer either sf or geoarrow or terra. One could preform point process analyses with spatstat, but that requires a spatstat's own class.

makes sense of course. If users then want to do analyses on molecule coords, then those can be easily extracted from SFE object and taken further for downstream analyses (point process, knn clustering using kd-tree etc..).
Another thing for Binning part, I thought it would be good to have a function to do spatial binning of molecules and construct SFE object based on bins-gene count matrix, adding bins polygons as colGeometries, and centroids of bins, and keeping molecules as an option.

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:

Currently testing readVizgen from your last commit, found some small issues, also in df2sf.R.
..will try to fix them. Once it works, I will test this function on several (new-old) Vizgen datasets we have here.

What are the issues?

What are the issues?

When running readVizgen basically in https://github.com/alikhuseynov/SpatialFeatureExperiment/blob/seurat_v4/R/df2sf.R#L27:
Error in `df[!df[[id_col]] %in% ids_rm, ]`:
! Can't subset rows with `!df[[id_col]] %in% ids_rm`.
✖ Logical subscript `!df[[id_col]] %in% ids_rm` must be size 1 or 1441835, not 0.

Others I fixed, will commit later..

Also, since you are testing, can you convert the tests into formal unit tests so it's easier to rerun later and automatically as in Bioc's daily check? There already is a small toy Vizgen dataset in inst/extdata/vizgen from one FOV in the mouse liver dataset from the Vizgen website. I can add images from other z-planes later today. Maybe you can move the files around to make it look like other versions of Vizgen datasets.

The datasets I'm using are not public unfortunately, but I can try out the toy data to mimic other versions of Vizgen.
What is your approach to make small toy dataset from a large dataset? ..eg downsampling the large SFE object and exporting each relevant file?
Thanks!

Looks like that error is caused by whichever column indicated by id_col not existing. I see what caused it, since the id_col shouldn't be required for MULTIPOINT since it can totally be the same as group_col. The most recent push should fix this.

My general approach to making the toy spatial dataset is to use a small bounding box to subset a dataset. For MERFISH, I took the first FOV and downsampled the transcript spots so the file is small. Existing tests for functions to read data are in tests/testthat/test-read.R

Looks like that error is caused by whichever column indicated by id_col not existing. I see what caused it, since the id_col shouldn't be required for MULTIPOINT since it can totally be the same as group_col. The most recent push should fix this.

another issue with BPPARAM in .df2sf_point and likely in similar functions like it.
Error in h(simpleError(msg, call)): error in evaluating the argument 'BPPARAM' in selecting a method for function 'bplapply': object 'BPPARAM' not found
BPPARAM here https://github.com/alikhuseynov/SpatialFeatureExperiment/blob/seurat_v4/R/df2sf.R#L40  is not found for bplapply in here https://github.com/alikhuseynov/SpatialFeatureExperiment/blob/seurat_v4/R/df2sf.R#L52

I'm fixing more issues since the new formatTxSpots function had not been tested before. Sorry for the confusion. I'm writing the new unit tests for readVizgen and formatTxSpots.

This error is caused by the gene count matrix being empty, which can be because there're no polygons in the z-plane specified. I think I need to re-make the MERFISH toy dataset in the package to include at least 2 z-planes to test functionalities to read multiple z-planes; the current version only has z = 0.

Cellbound is just raster cell segmentation masks right? Also, when you said Cellbound is often followed by a number like Cellbound3, what does the number mean?

Cellbound is just raster cell segmentation masks right? Also, when you said Cellbound is often followed by a number like Cellbound3, what does the number mean?

That's a cell boundary staining image (each stain has a number), eg, mosaic_Cellbound1_z3.tif, there can be several such stains per z-plane.
One test dataset we have, also contained mosaic_QC1_z3.tif, I don't recall exactly what QC exactly meant there.
I will try to make a toy dataset from the test dataset we have, and upload to inst/extdata/vizgen_2 or something like that.

This error is caused by the gene count matrix being empty, which can be because there're no polygons in the z-plane specified. I think I need to re-make the MERFISH toy dataset in the package to include at least 2 z-planes to test functionalities to read multiple z-planes; the current version only has z = 0.

Yes, try out. I was just testing it today, works well with single z plane.
Do you think, it is possible to store transcripts spots from all z-planes if z="all", then segmentations and centroids would always be from one mid plane, ie 3L (since segmentations are all the same in all planes from default output)
Also, it might be useful when plotting rowGeometries, eg with potential plotMolecules(), where  z arg can be used to plot molecules from "all" (if present) or only from selected z plane.

There should be a more informative error message.

That would help a lot 👍

Do you think, it is possible to store transcripts spots from all z-planes if z="all", then segmentations and centroids would always be from one mid plane

Yes, 3D MULTIPOINT geometries can be constructed when all z-planes are used. However, the underlying GEOS library does not support geometric operations on 3D geometries, so it's just storing data. This can be used when plotting. Right, since the segmentation is only available for 1 z-plane, any z-plane can be used to get the polygons so maybe we can keep one fixed z-plane for polygons regardless of the z argument.

That's a cell boundary staining image (each stain has a number), eg, mosaic_Cellbound1_z3.tif, there can be several such stains per z-plane. It typically helps the Cellpose model to better segment cells.

That's really cool. When we have this we can apply the really cool biophysical model from Adrien Hallou on how membrane tension led to different tissue compartments in development.

After testing with a large dataset (the one with 30GB detected_transcripts.csv), I'll switch to data.table to deal with the transcript spots, which I find faster and more memory efficient than tidyverse, which is then faster than base R.

After testing with a large dataset (the one with 30GB detected_transcripts.csv), I'll switch to data.table to deal with the transcript spots, which I find faster and more memory efficient than tidyverse, which is then faster than base R.

data.table plus tidyverse is the best to deal with large files.

I did few commits and I tested readVizgen on some (small and large) datasets we have here, with single z sections z = 3L and all planes z = "all", everything works well.
I will try to prepare 2-3 small toy datasets with several z sections, images, etc.. for unit tests.
Will you optimize the reading/processing transcript part with data.table? Or, you can point me where we should have it and I will add

Will you optimize the reading/processing transcript part with data.table?

Yes, I'm working on it. Also, is there a reason why use mid z-plane while the cell segmentation in all z-planes are the same?

Will you optimize the reading/processing transcript part with data.table?

Yes, I'm working on it.

👍

Also, is there a reason why use mid z-plane while the cell segmentation in all z-planes are the same?

..just to be consistent with the default z = 3L, can be any z-plane of course

some issue with row names in .sc2cg
>>> filtering geometries to match 24897 cells with counts > 0
Warning message:
“non-unique values when setting 'row.names': ‘3.92760610694571e-301’, ‘3.92760610702854e-301’, ‘3.92760610702855e-301’, ‘3.92760610702856e-301’, ‘3.92760610702857e-301’, ‘3.92760610702859e-301’, ...

Error in `.rowNamesDF<-`(x, value = value): duplicate 'row.names' are not allowed
Traceback:
1. readVizgen(data_dir = dir_use, z = "3L", sample_id = "mb299",
 .     min_area = 15, image = c("DAPI", "PolyT", "Cellbound"), flip = "geometry",
 .     max_flip = "50 MB", filter_counts = TRUE, add_molecules = TRUE,
 .     use_bboxes = TRUE, BPPARAM = BiocParallel::MulticoreParam(14,
 .         tasks = 80L, force.GC = FALSE, progressbar = TRUE))
2. SpatialFeatureExperiment(assays = list(counts = mat), colData = metadata,
 .     sample_id = sample_id, spatialCoordsNames = c("center_x",
 .         "center_y"), unit = "micron", BPPARAM = BPPARAM)
3. .spe_to_sfe(spe, colGeometries, rowGeometries, annotGeometries,
 .     spatialCoordsNames, annotGeometryType, spatialGraphs, spotDiameter,
 .     unit, BPPARAM)
4. .sc2cg(spatialCoords(spe), spotDiameter, BPPARAM)
5. `rownames<-`(`*tmp*`, value = rownames(coords_use))
6. `row.names<-`(`*tmp*`, value = value)
7. `row.names<-.data.frame`(`*tmp*`, value = value)
8. `.rowNamesDF<-`(x, value = value)
9. stop("duplicate 'row.names' are not allowed")

btw, for documentation isn't it more convenient to have :: (It actually helps debugging), like data.table::fread and completely avoiding imports like @importFrom and @import?

Heads up: I tried to write the 3D MULTIPOINT geometries of transcript spots from all 7 z-planes of a MERFISH liver dataset with nearly 400,000 cells and over 400 million transcript spots into a parquet file, and got an error like
CapacityError: array cannot contain more than<PHONE_NUMBER> bytes, have<PHONE_NUMBER>

Turns out that the wkb (spatial binary) is too large for arrow to handle, and it's a problem with arrow itself, nothing to do with R. Maybe I'll split by z-plane to write a separate file for each z-plane so geometric operations on the entire SFE object would still work.
I can try using sedona to write the huge file to rdd so next time the whole thing won't be loaded into memory. Maybe there're more implications downstream than I expect. That's beyond the scope of this pull request so you don't have to deal with it. On disk spatial operations is on my wishlist so I should play around with it, in a separate branch, since I'm not sure if I have enough time to deal with the downstream implications like I need to make sure code works for both regular sf and sedona.
Anyway, what's the point of all this? Using the MULTIPOINT geometry in 3D doesn't help much because the underlying GEOS library doesn't support 3D geometric operations, while the advantage of using sf over a data frame of coordinates is the geometric operations. Plotly supports 3D sf geometries, though I'm not sure how often people would plot the transcripts from a few cells in 3D, so that favors splitting by z-planes. But it only takes some easy data wrangling to put 2D coordinates from multiple z-planes together for plotly. Well, if sedona geometric operations work for 3D (I don't know if it does), then the advantage remains. A limitation of splitting by z-plane is that it doesn't work for Xenium where z is continuous. Also, while most of 3D spatial transcriptomics right now uses serial sections so we can still split by z, that won't work for thick sections.

some issue with row names in .sc2cg

Looks like it's caused by the cell ID column read as numeric rather than character.

some issue with row names in .sc2cg

Looks like it's caused by the cell ID column read as numeric rather than character.

Seems to work now from the last commit, thanks

The debate on whether to construct 3D geometries or split by z-planes: R packages that seem relevant to 3D geometric operations: cgalMeshes and Rcvg, outside the geospatial tradition. Looks like 3D geometric operations are no small deal so I won't blame GEOS. But I feel like those packages are overkill since I've never seen 3D mesh cell segmentations. Confocal microscopes image z-plane by z-plane anyway, and most 3D data come from serial sections. Then a hacky way around GEOS's limitation by doing the geometric operation z-plane by z-plane or temporarily dropping z and then adding it back should suffice for now. Maybe whether to split by z-plane or construct a 3D geometry can be a new argument in the relevant functions. Then it would be nice to have metadata on row or colGeometries to group the geometries in SFE v2 (not this release). Because thick slices aren't common yet, I would not worry about meshes for now, maybe for a few years.

Heads up: I tried to write the 3D MULTIPOINT geometries of transcript spots from all 7 z-planes of a MERFISH liver dataset with nearly 400,000 cells and over 400 million transcript spots into a parquet file, and got an error like
CapacityError: array cannot contain more than<PHONE_NUMBER> bytes, have<PHONE_NUMBER>

Turns out that the wkb (spatial binary) is too large for arrow to handle, and it's a problem with arrow itself, nothing to do with R. Maybe I'll split by z-plane to write a separate file for each z-plane so geometric operations on the entire SFE object would still work.
I can try using sedona to write the huge file to rdd so next time the whole thing won't be loaded into memory. Maybe there're more implications downstream than I expect. That's beyond the scope of this pull request so you don't have to deal with it. On disk spatial operations is on my wishlist so I should play around with it, in a separate branch, since I'm not sure if I have enough time to deal with the downstream implications like I need to make sure code works for both regular sf and sedona.
Anyway, what's the point of all this? Using the MULTIPOINT geometry in 3D doesn't help much because the underlying GEOS library doesn't support 3D geometric operations, while the advantage of using sf over a data frame of coordinates is the geometric operations. Plotly supports 3D sf geometries, though I'm not sure how often people would plot the transcripts from a few cells in 3D, so that favors splitting by z-planes. But it only takes some easy data wrangling to put 2D coordinates from multiple z-planes together for plotly. Well, if sedona geometric operations work for 3D (I don't know if it does), then the advantage remains. A limitation of splitting by z-plane is that it doesn't work for Xenium where z is continuous. Also, while most of 3D spatial transcriptomics right now uses serial sections so we can still split by z, that won't work for thick sections.

I see the point, those things are indeed beyond this PR yes. Also readXenium in on my list (was playing around) and converter from/to Seurat-SFE is under way.

The debate on whether to construct 3D geometries or split by z-planes: R packages that seem relevant to 3D geometric operations: cgalMeshes and Rcvg, outside the geospatial tradition. Looks like 3D geometric operations are no small deal so I won't blame GEOS. But I feel like those packages are overkill since I've never seen 3D mesh cell segmentations. Confocal microscopes image z-plane by z-plane anyway, and most 3D data come from serial sections. Then a hacky way around GEOS's limitation by doing the geometric operation z-plane by z-plane or temporarily dropping z and then adding it back should suffice for now. Maybe whether to split by z-plane or construct a 3D geometry can be a new argument in the relevant functions. Then it would be nice to have metadata on row or colGeometries to group the geometries in SFE v2 (not this release). Because thick slices aren't common yet, I would not worry about meshes for now, maybe for a few years.

I was using Rvcg and related packages mesheR, rgl a lot when working with 3D (CT images or volumetric) data, works great. For cell segmentation in 3D would be challenging, since each cell would represent a mesh3d object, there will be a lot of them, then plotting those could be a pain since meshes are basically vectors.
Going beyond again, one could reconstruct and generate 3D mesh from 2D cell boundary outline as well, I did that for brain organoid images (paper Fig2 and repo it's an old one though)

You can replace the existing Vizgen toy dataset in inst/extdata with yours that has multiple z-planes. But please put the code you used to make the dataset in inst/script.

For cell segmentation in 3D would be challenging, since each cell would represent a mesh3d object in the end, and there will be a lot of them, then plotting those could be a pain since meshes are basically vectors (except, if only like 20-30 cells are plotted at the same time).

I think another challenge when visualizing many cells in 3D is that they're like 3D blocks of sand and you can't see through them to get the spatial structure. Giotto can plot sections through the 3D block. It would be nice to be able to plot sections with arbitrary planes though. Like static plots with ggplot2 that goes nicely into publications. ImageJ is better for the interactive part.

For cell segmentation in 3D would be challenging, since each cell would represent a mesh3d object in the end, and there will be a lot of them, then plotting those could be a pain since meshes are basically vectors (except, if only like 20-30 cells are plotted at the same time).

I think another challenge when visualizing many cells in 3D is that they're like 3D blocks of sand and you can't see through them to get the spatial structure. Giotto can plot sections through the 3D block. It would be nice to be able to plot sections with arbitrary planes though. Like static plots with ggplot2 that goes nicely into publications. ImageJ is better for the interactive part.

Yes, they would be quite packed, one can make them slightly transparent but yeah, wouldn’t be easy to see.. unless one applies some small erosion to get space between cells.

It would be nice to be able to plot sections with arbitrary planes though. Like static plots with ggplot2 that goes nicely into publications.

you mean like 2D static plots for selected z-planes separately?
So far I’m quite happy with plotly for interactive part, I completely avoid Fiji/ImageJ since years.

You can replace the existing Vizgen toy dataset in inst/extdata with yours that has multiple z-planes. But please put the code you used to make the dataset in inst/script.

I thought adding few folders with downsampled datasets, for testing purposes, like only with hdf5 files, then another with parquet, etc.. then one can prepare unit tests.
Yes, I will definitely add the scripts to show how I made them, however the original large datasets input wouldn’t be available.

..tested readVizgen with z = 3L and z = "all" from most recent commit, all works!
btw, I can only see readCosMX in like SpatialFeatureExperiment:::readCosMX

I can only see readCosMX in like SpatialFeatureExperiment:::readCosMX

That's because I forgot to update the documentation. I'm working on unit tests for readCosMX. After you push your unit tests, I'll merge the pull request.

It seems that the rowGeometry(sfe) are no longer XYZ MULTIPOINT, it returns 2D XY even with z = "all".
Probably changes in your last commit broke that?

FYI it's OK if you can't finish before next Monday when the 3.18 branch is frozen for release. I'm not going to announce the release anyway because I haven't been able to implement many features I planned. I may quietly add your functions to the release branch afterwards. Then I'll do a major announcement for the 3.19 release next April.

FYI it's OK if you can't finish before next Monday when the 3.18 branch is frozen for release. I'm not going to announce the release anyway because I haven't been able to implement many features I planned. I may quietly add your functions to the release branch afterwards. Then I'll do a major announcement for the 3.19 release next April.

Yeah, won't manage by Monday. Adding things afterwards and officially for 3.19 would be more realistic. Thanks

Added Vizgen toy dataset to ./seurat_v4/inst, basically vizgen_cellbound and vizgen_cellbound_Cellpose (same as vizgen_cellbound but has Cellpose dir).
Script to make that dataset from a larger dataset -> vizgen_cellbound.R using magrittr pipe and dplyr, hope that's ok.
I will make 1 more dataset that contains hdf5 files. Could you write/update the relevant unit test scripts for those toy datasets?
I'm trying prioritize readXenium and Seurat <-> SFE converter

Yes, I can update the unit tests. Actually one toy dataset with everything is sufficient. I would copy the directory elsewhere and change it to test different scenarios.

Yes, I can update the unit tests. Actually one toy dataset with everything is sufficient. I would copy the directory elsewhere and change it to test different scenarios.

sounds good.
so, the another one with hdf5 is not need then?

The hdf5 files can be generated from the same toy example you already made. Doesn't have to be a completely separate directory.

another thing I noticed when readVizgen with z = "all" & z_option = "3d", it still reads the transcript coords and writes them even if the .parquet or relevant dir with files is already present/written. Something to do with formatTxSpots I think.

Looks like a bug in that function.

The hdf5 files can be generated from the same toy example you already made. Doesn't have to be a completely separate directory.

ok, I haven't yet figured how to make hdf5 files from parquet or export them from colGeometry of SFE. I have separate large directory with hdf5 files that I used to test functionality locally, since if parquet is present then hdf5 files won't be loaded.

Looks like a bug in that function.

I can take a look later as well.

ok, I haven't yet figured how to make hdf5 files from parquet or export them from colGeometry of SFE. I have separate large directory with hdf5 files that I used to test functionality locally, since if parquet is present then hdf5 files won't be loaded.

I would copy the directory with the test data elsewhere and delete the parquet file there when testing hdf5. My code to make a small hdf5 file out of an original hdf5 file for testing purposes, though not from colGeometry: https://github.com/pachterlab/SpatialFeatureExperiment/blob/bf421daebb46e5b5534b07195450cb50ca5e37ac/inst/script/vizgen.R#L41
OK, what's the size of each hdf5 file in your large directory? You can use one of them if it isn't too large. The test data is included in the installed package, and there's the 5 MB limit in Bioc policy. SFE has already exceeded that limit but it's fine it it doesn't exceed too much (now like 6 MB). If the test data is too large, I suppose I'll host them on Box or Zenodo and download them when testing.

I would copy the directory with the test data elsewhere and delete the parquet file there when testing hdf5. My code to make a small hdf5 file out of an original hdf5 file for testing purposes, though not from colGeometry:
https://github.com/pachterlab/SpatialFeatureExperiment/blob/bf421daebb46e5b5534b07195450cb50ca5e37ac/inst/script/vizgen.R#L41

I will check that as well.

OK, what's the size of each hdf5 file in your large directory? You can use one of them if it isn't too large. The test data is included in the installed package, and there's the 5 MB limit in Bioc policy. SFE has already exceeded that limit but it's fine it it doesn't exceed too much (now like 6 MB). If the test data is too large, I suppose I'll host them on Box or Zenodo and download them when testing.

single hdf5 is like 7 MB, I have another one where it might be ~3-4 MB. BTW, the toy dataset I uploaded should be ~10MB in total.
For unit tests, all of them are using that toy vizgen data that you have within the package, or do you test additionally on other datasets locally?

For unit tests, all of them are using that toy vizgen data that you have within the package, or do you test additionally on other datasets locally?

Yes, I always use the toy vizgen data within the package, so the relevant code can be tested in Bioconductor's daily check in case changes in a dependency breaks the code. But some existing Visium unit tests use data downloaded from 10X's website (< 20 MB), so I can also download the toy Vizgen data if you can't make them small enough to be contained inside the package.

For unit tests, all of them are using that toy vizgen data that you have within the package, or do you test additionally on other datasets locally?

Yes, I always use the toy vizgen data within the package, so the relevant code can be tested in Bioconductor's daily check in case changes in a dependency breaks the code and in the automated GitHub Action that runs unit tests when I push to the devel branch. But some existing Visium unit tests use data downloaded from 10X's website (< 20 MB) or datasets from the SFEData package, downloading on the fly when the tests are run, so I can also download the toy Vizgen data if you can't make them small enough to be contained inside the package.

now the dataset dir is ~2.6MB, I will try to make hdf5 files from rowGeometires later and put them in the same toy dataset dir

it fails to install this branch after your last commits, something to do with false formatted parts for 'Depends', 'Suggests', 'Imports' or 'Enhances'. DropletUtils seems to be an issue.
Does it work for you?

When making toy datasets, I noticed that readVizgen would return SFE with cellSeg(sfe) that are POLYGON.
If I crop the object using some bbox like this:
sfe_sub <-
SpatialFeatureExperiment::crop(sfe, colGeometryName = "cellSeg", sample_id = "vizgen_test", y = bbox_use)

Then the cropped object's cellSeg(sfe_sub) are stored as MULTIPOLYGON. Is this intended to be like that, or is a minor bug in crop?

Added readXenium, accessible via SpatialFeatureExperiment:::readXenium.

Are you already working on a test dataset for readXenium? If not then I'll work on it. I think I might upload all the test data for Vizgen, CosMX, and Xenium to the Bioconductor Data Lake, with the SFEData package, so users don't have to download the test data unless used in examples.

currently SFE Xenium count matrix is is a class 'DelayedMatrix', however Vizgen is a class 'dgCMatrix'.
assay(sfe_xen, "counts") %>% str
Formal class 'DelayedMatrix' [package "DelayedArray"] with 1 slot
...
assay(sfe_vz, "counts") %>% str
Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
...

should we keep them the same, either 'DelayedMatrix' or 'dgCMatrix'?

Yes, please keep them the same. Having DelayedMatrix is helpful for larger datasets so you don't have to load everything into memory but it makes computation slower.

always keep cell ids in $ID in cellSeg (and nucSeg)? which is removed in readVizgen (readCosMX has $ID in polys as well)

Yes, please keep them the same. Having DelayedMatrix is helpful for larger datasets so you don't have to load everything into memory but it makes computation slower.

sounds good, also readCosMx has 'dgCMatrix', should be fixed as well then

Oh, hold on, I don't mean making them all DelayedMatrix. What I mean is to keep whatever class originally read in. In the case of Xenium, DropletUtils::read10xCounts reads the h5 file as DelayedMatrix. dgCMatrix is fine for other technologies when csv's are read in.

Oh, hold on, I don't mean making them all DelayedMatrix. What I mean is to keep whatever class originally read in. In the case of Xenium, DropletUtils::read10xCounts reads the h5 file as DelayedMatrix. dgCMatrix is fine for other technologies when csv's are read in.
ok, then we keep as it is read in

always keep cell ids in $ID in cellSeg (and nucSeg)? which is removed in readVizgen (readCosMX has $ID in polys as well)

I think keeping $ID in polys is better for convenience, will add it to readVizgen


Stuck in Waiting for Operator to register custom resource definitions

Deploying Prometheus Operator
serviceaccount/prometheus-operator unchanged
clusterrole.rbac.authorization.k8s.io/prometheus-operator configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator configured
service/prometheus-operator unchanged
deployment.apps/prometheus-operator unchanged
Waiting for Operator to register custom resource definitions..........................................................
I found that I have waited for a long time without action. Is there any part of the error?

I'm getting the same here.
GCP - Kubernetes Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.6"

This is why: error: the server doesn't have a resource type "prometheusrules"
seems the resource is no longer available
kubectl -n monitoring get prometheusrule.monitoring.coreos.com


Web Chat: Mock Bot as separate repository and hosted

[x] Separate Mock Bot into its own repository

[x] Bring existing Mock Bot features over (without the test part)
[x] Re-architect it using SDK v4, instead of mocking Direct Line service

We will lose the ability to hosting it locally
We can use Emulator Core to host it
We should consider mocking HTTP server or Restify, so we can host it in non-Node.js environment, e.g. Service Worker

[x] Host it thru CI/CD

Done.


UI stretching thumbnails

becomes:

The original picture isn't loading. I'm not seeing stretching on that record - it really is a long  skinny one.

Still the case. It's long and skinny, but I think something with the CSS is causing the browser to upsample the thumbnail. Image is 52 x 150px but its getting stretched to 121x350.
Will show up with this query: http://plum.princeton.edu/catalog?utf8=✓&q=ars

This looks good to all of us. Can you check again, @jpstroop, and then close?


Update deprecated Node.js 12 actions in devfile repos

Which area/kind this issue is related to?
/area releng
Issue Description

Node.js v12 has been removed from the Github action runner but a number of our repos still rely on this deprecated version.  We should update to the latest v16 version of the gitub actions and ensure actions are pinned to their commit hash: https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions
The following repos should be inspected for deprecated actions:

[x] api https://github.com/devfile/api/issues/1224
[x] library https://github.com/devfile/api/issues/1225
[x] registry-support https://github.com/devfile/api/issues/1226
[x] registry https://github.com/devfile/api/issues/1227
[x] devfile-web https://github.com/devfile/api/issues/1228
[x] integration-tests https://github.com/devfile/api/issues/1229

@kim-tsao should we treat #1230 as an independent item, as it is handled externally, and close EPIC 992?

@kim-tsao I've removed the last item https://github.com/devfile/api/issues/1230 as it wasn't handled by the devfiles team. As all other items have been closed I'll close this epic too.

@kim-tsao should we treat #1230 as an independent item, as it is handled externally, and close EPIC 992?

Yes, let it be an independent item


AOT fails with type errors even when no type errors in the project

I have absolutely no type errors in my project, npm run build.prod builds without any issues. But npm run build.prod.exp fails with type errors
All the errors seem to be originating from *.ngfactory.ts files
Showing partial output
dist/tmp/app/app.component.ngfactory.ts(350,40): error TS2346: Supplied parameters do not match any signature of call target.
dist/tmp/app/app.module.ngfactory.ts(437,23): error TS2339: Property 'status' does not exist on type 'typeof "/Users/sreddy/space/angularjs/regression-status/dist/tmp/app/app.module"'.
dist/tmp/app/app.module.ngfactory.ts(438,25): error TS2339: Property 'platform' does not exist on type 'typeof "/Users/sreddy/space/angularjs/regression-status/dist/tmp/app/app.module"'.
dist/tmp/app/configure/configure.component.ngfactory.ts(1876,103): error TS2339: Property 'branches' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(1931,104): error TS2339: Property 'areas' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(1986,105): error TS2339: Property 'sandbox' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(2041,105): error TS2339: Property 'performance' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(2096,105): error TS2339: Property 'tags' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(2862,132): error TS2339: Property 'broadcast' does not exist on type 'Object'.
dist/tmp/app/configure/configure.component.ngfactory.ts(2862,196): error TS2339: Property 'broadcast' does not exist on type 'Object'.
dist/tmp/app/login/login.component.ngfactory.ts(398,40): error TS2341: Property 'loginForm' is private and only accessible within class 'LoginComponent'.
dist/tmp/app/login/login.component.ngfactory.ts(523,43): error TS2341: Property 'loginForm' is private and only accessible within class 'LoginComponent'.
dist/tmp/app/platform/compare.component.ngfactory.ts(154,67): error TS2341: Property 'queryParams' is private and only accessible within class 'CompareComponent'.
...
...
...
lass 'StatusComponent'.
dist/tmp/app/status/status.component.ngfactory.ts(1392,43): error TS2341: Property 'diffDeltaSelectionOptions' is private and only accessible within class 'StatusComponent'.
dist/tmp/app/status/status.component.ngfactory.ts(1452,29): error TS2346: Supplied parameters do not match any signature of call target.
dist/tmp/app/status/status.component.ngfactory.ts(1500,72): error TS2346: Supplied parameters do not match any signature of call target.

Yes, this is because tsc will run type checking in ngfactories which is something which doesn't happen runtime because the JiT compiler generates JavaScript instead of TypeScript.
This is not an issue angular-seed but a problem that you have introduced. Make sure all your bindings have the correct types.

@mgechev where is the correct location to place ones own typings?

tools/manual_typings/project but this is not caused by lack of typings, I believe. More like semantic errors caused by accessing non existing properties of objects of specific type within the templates.


drivers: can: provide default callback to can_send() if NULL

Provide a default, internal callback to CAN controller drivers implementation of can_send() if no callback was provided by the caller.
This allows for simplifying the CAN driver implementations of can_send() as these no longer need special handling for callback/no callback operation.

The changes are included in already merged  #50547.


I'm getting an error while following the build guide. #1404

Ok so I had an issue lime build windows and it gave me the error: Error: Source path "C:/HaxeToolkit/haxe/lib/extension-webm/git/ndll/Windows64/extension-webm.ndll" does not exist so I used ctrl + f to search the build guide for extension-webm. So I first ran haxelib git extension-webm https://github.com/KadeDev/extension-webm it updated what I already have. Then I tried lime rebuild extension-webm windows but it gave the error
Error: ExternalInterface.cpp
C:\HaxeToolkit\haxe\lib\hxcpp\4,2,1\include\hx\OS.h(59): fatal error C1083: Cannot open include file: 'stddef.h': No such file or directory
Please can I have some help?

i hate vscode dependencies
and yes that's the cause of the issue

Can you please elaborate. And help me fix it @prokube?

@KadeDev do you know what to do?

This is dead so I'm reasking.

I don't know if you are still looking for an answer but I found that the problem can be solved by downloading Windows 10 SDK from here https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/ Hope this helps

@littlestalker omg thanks!

@littlestalker I downloaded the win 10 SDK, now what? What do I do now?


Error: spawn EINVAL running autorest legacy on windows

A recent Node JS update includes a breaking change to patch a security vulnerability: https://nodejs.org/en/blog/vulnerability/april-2024-security-releases-2
When running autorest csharp v2 on Windows, the "spanw EINVAL" error occurs and stops execution.
Tested with both Node v21.7.3 and v18.20.2 (the patch was applied across all major Node versions)
Command:
autorest --input-file=swagger.json --csharp --legacy --add-credentials --namespace=MyApp.Model.Data --output-folder=MyApp.Model\Data

Console output:
AutoRest code generation utility [cli version: 3.7.1; node: v21.7.3]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    | AutoRest core version selected from configuration: ~2.0.4413.
   Loading AutoRest core      'C:\Users\594106\.autorest\@microsoft.azureautorest-core<EMAIL_ADDRESS>(2.0.4429)
No configuration found at 'file:///C:/Users/594106/repos/PPIThePricingAppIntegration/PricingApp/'.
   Loading AutoRest extension<EMAIL_ADDRESS>(~2.3.79->2.3.102)
   Loading AutoRest extension<EMAIL_ADDRESS>(2.3.55->2.3.55)
Process() cancelled due to exception : spawn EINVAL
  Error: spawn EINVAL

Suggested solution from the Security Release announcement:

If the input to spawn/spawnSync is sanitized, users can now pass { shell: true } as an option to prevent the occurrence of EINVALs errors.

Should be labeled v2--haven't tested on v3


Learn K8S - Tue

There are 2 potential books

Kubernetes in Action
Kubernetes Up and Running

You can choose 1 or 2 and read that, the main goal is to understand the

What is K8S?
Why do we need K8S?
K8S Core component?

Keep in mind that we just need the basic knowledge, no need for any advanced knowledge, for example, Networking.

FInished chapter 8 of Kubernetes Up and Running


Fix Scheduler#timeline method stuck in infinite loop with a cron job using a 'first_in:' parameter

Fix for Issue #151. I am not sure if this is the right solution as if the job is created with a first_at == Time.now, the first_at would be ignored.

Many thanks!

Thanks for releasing it so quickly! :smile:


randomly stops working after a few  games

gets stuck at the main menu and sometimes in title select

logs?

it seems that it's stopped auto pressing start on the main menu
the only logs i get from loading the main menu is
Source map error: request failed with status 404
Resource URL: https://steamcommunity-a.akamaihd.net/public/javascript/saliengame/pixi/pixi.min.js?v=.tDHAybGykI1O
Source Map URL: pixi.min.js.map[Learn More]
Source map error: request failed with status 404
Resource URL: https://steamcommunity-a.akamaihd.net/public/javascript/saliengame/pixi/pixi-spine.js?v=.7OsC6yYlmcju
Source Map URL: pixi-spine.js.map[Learn More]
Source map error: request failed with status 404
Resource URL: https://steamcommunity-a.akamaihd.net/public/javascript/saliengame/pixi/pixi-filters.js?v=.0zUgeJIQqNXu
Source Map URL: pixi-filters.js.map[Learn More]
Source map error: request failed with status 404
Resource URL: https://steamcommunity-a.akamaihd.net/public/javascript/saliengame/pixi/pixi-sound.js?v=.x9Q08vlRNyLh
Source Map URL: pixi-sound.js.map[Learn More]
Error: WebGL warning: drawElements: This operation requires zeroing texture data. This is slow.
pressing start continues normal actions

I have noticed the same thing recently. Twice it's stopped on the main screen where you have to press start, and pressing start will make it work again.

I have the same bug, in chrome (67.0.3396.87 x64 \ Tampermonkey
4.6) - script not push button "Press start"
and in Opera (53.0.2907.106 \ Tamper 4.2.5291) - all fine work
if I do it manualy it work until next while game not enter again at start screen
before 1 hour ago all working fine =)

Just failed again; it went back to the tile screen this time and all of the tiles appeared to have 0 progress (new planet or something?). Kept repeating the same thing in the console screenshot. I clicked one of the empty tiles and it started working again.

i have the same issue.. it stops after a few rounds and i have to click on play again to just run the "bot" again..


Make a parser for EXPLAIN

Provided with a query, we want to be able to discover:

Which tables and indexes it needs to access
Which tables and indexes it needs to write

Can EXPLAIN output do this? Pretty sure we can make it emit JSON so parsing itself might be easy and it may be hard to walk through the plan to discover what objects it uses.

Might not be necessary with #36, it appears that pg_query has some support to access this information. Notable we can:

Check if a statement is DDL/DML or query
Check which tables it accesses

Here's an interesting check I just made:
#[cfg(test)]
mod tests {

    #[test]
    fn play_around() {
        let s = "UPDATE tab SET col = 1 FROM s WHERE tab.id = s.id ;";
        let p = pg_query::parse(s).unwrap();
        println!("{:?}", p.tables());
        println!("{:?}", p.select_tables());
    }
}

This prints the following:
["s", "tab"]
["s"]

So it seems like we're able to check which tables that we need to lock with AccessShare and which we need with a higher lock.

This is no longer useful.


Add auto_decompress to pco_python

This gets a start on adding an auto_decompress function to the Python module which does not require a pre-allocated Numpy array as an argument.
However, I'm a bit stuck on how to dynamically determine the data type of the output, as described in #150.

@rabernat Let's do a bit of coordination on this one in the discord, if that's ok by you


CCTP: No protocol registered for Solana:AutomaticCircleBridge

I'm testing out a small script that is mostly based on the cctp example:
  const wh = new Wormhole("Mainnet", [EvmPlatform, SolanaPlatform]);

    source = await getStuff(wh.getChain("Base"));
    destination = await getStuff(wh.getChain("Solana"));

  await cctpTransfer(wh, source, destination, {
    amount,
    automatic: true,
    nativeGas: undefined,
  });
}

Using version 0.3.0-beta.16 of all the dependencies:
    "@wormhole-foundation/connect-sdk": "0.3.0-beta.16",
    "@wormhole-foundation/connect-sdk-evm": "0.3.0-beta.16",
    "@wormhole-foundation/connect-sdk-evm-cctp": "0.3.0-beta.16",
    "@wormhole-foundation/connect-sdk-evm-core": "^0.2.6",
    "@wormhole-foundation/connect-sdk-solana": "0.3.0-beta.16",
    "@wormhole-foundation/connect-sdk-solana-cctp": "0.3.0-beta.16"

Stack trace
Error: No protocol registered for Solana:AutomaticCircleBridge
    at getProtocolInitializer (/cctp/node_modules/@wormhole-foundation/sdk-definitions/src/protocol.ts:78:20)
    at create (/cctp/node_modules/@wormhole-foundation/sdk-definitions/src/protocol.ts:89:16)
    at SolanaPlatform.getProtocol (/cctp/node_modules/@wormhole-foundation/sdk-definitions/src/platform.ts:106:18)
    at SolanaChain.getAutomaticCircleBridge (/cctp/node_modules/@wormhole-foundation/sdk-definitions/src/chain.ts:143:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Wormhole.circleTransfer (/cctp/node_modules/@wormhole-foundation/connect-sdk/src/wormhole.ts:99:19)
    at cctpTransfer (/cctp/src/index.ts:35:16)

No AutomaticCircleBridge exists for Solana.
In fact the native usdc transfers are only available on devnet for Solana at the moment.
If you do get this sort of error for other protocols, be sure to register the proto as in this example

In fact the native usdc transfers are only available on devnet for Solana at the moment.

Ah okay, that explains the various errors I've been getting. Thanks!


Update Chapter 4 "End-of-chapter quiz"

Description

This pull request fixes several typos in the Chapter 4 End-of-chapter quiz without altering the structure or logic in the page.
Changes

Remove repeated affirmations from explanations
Added some explanations
Clarifies some questions
Added some code marks

Purpose

Because the explanations automatically appear with "Correct!" or "Incorrect." before the explanation, explanations that begin with affirmations such as "Correct", "That's right", and "Right", have been removed to reduce repetition. This pull request adds some explanations when the previous explanation only said whether the answer was correct. Some questions have been slightly modified for clarification. Some code marks, the ` that marks code, have been added.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Version 1.74   remote    vscode media-preview startup false

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.74.1
OS Version:ubuntu

Steps to Reproduce:

In the newest version ，In my remote ubuntu server, the jpg file not showing,just black ,and I can see  vscode media-preview startup false
When I back to 1.73，It's OK    I don't know why

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.74.1. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.74.1. Please try upgrading to the latest version and checking whether this issue remains.

yes，I reinstall this afternoon,It's still doesn't work

/gifPlease

Thanks for reporting this issue! Unfortunately, it's hard for us to understand what issue you're seeing. Please help us out by providing a screen recording showing exactly what isn't working as expected. While we can work with most standard formats, .gif files are preferred as they are displayed inline on GitHub. You may find https://gifcap.dev helpful as a browser-based gif recording tool.
If the issue depends on keyboard input, you can help us by enabling screencast mode for the recording (Developer: Toggle Screencast Mode in the command palette). Lastly, please attach this file via the GitHub web interface as emailed responses will strip files out from the issue.
Happy coding!

/gifPlease

Like this，no response when click image

/gifPlease

Like this，no response

/gifPlease

Like this ,no response

i meet this question too , when will solve this bug issue ??

i meet this question too , when will solve this bug issue ??

Have you solved the problem？

Hey @mjbvz, this issue might need further attention.
@seantech2000, you can help us out by closing this issue if the problem no longer exists, or adding more information.

Duplicate of #170821


[pytket-qiskit] TketPass cannot be used in a qiskit PassManager to run multiple circuits.

When a PassManager is constructed from a TketPass, it can be used to run single circuits, but when a list of circuits is passed to the run() method it falls over with a message about not being able to pickle the pass object.
from pytket.extensions.qiskit.tket_pass import TketPass
from pytket.passes import FullPeepholeOptimise
from qiskit import QuantumRegister, QuantumCircuit
from qiskit.transpiler import PassManager

qr = QuantumRegister(1)
qc = QuantumCircuit(qr)
qc.h(qr[0])
tkpass = FullPeepholeOptimise()
qispass = TketPass(tkpass)
pm = PassManager(qispass)
pm.run(qc) # OK
pm.run([qc, qc]) # TypeError: cannot pickle 'pytket._tket.passes.BasePass' object

This makes the PassManager unusable in many qiskit algorithms.

Fixed in pytket-qiskit 0.16.0.


Repr value for nan

In Python:
>>> repr(float('nan'))
'nan'

while in RustPython:
>>>>> repr(float('nan'))
'NaN'

slight incompatibility but it is actually tested for in cmath.

(TIL NaN is lowercase in repr in python)

C also displays it as 'nan' if I'm not mistaken. I always remember seeing 'NaN' in Javascript tbh and Rust seems to follow that. Either way, 'NaN' always seemed weird to me, either capitalize em all or don't, why is a being left out like that? No respect for poor a.

@DimitrisJim I would like to work on it
would you pls guide me a bit?

Yes! Take a look at float's repr function. This calls another function to handle formatting for the different values that might be passed in. Take a look there and try and find out which case Nan/Inf fall under and what you'd need to do.

RustPython/Lib/reprlib.py
s = builtins.repr(x)
# Bugs in x.__repr__() can cause arbitrary
# exceptions -- then make up something

repr(float('nan'))
returns the builtin repr function called with float('nan')
I am stuck here...

Hey @Codemonk-adi, not sure what you mean here. Are you looking inside the implementation of repr for PyFloat? You can always fire up a debugger and get through stepping your way until the end (it takes some getting used to but its worth it).

@DimitrisJim  I am sorrry to bother you again, but I am unable to use the debugger, when I run the debug option on vscode it opens an rust python interpreter, and I do enter debugging mode in vscode but the buttons are disabled.

@Codemonk-adi hop on over to gitter if you can. This way I can help you further w/o notifying watchers.

closed with #3103


Release notes for 10.2.1535.0 version of Windows Store Remote Desktop??

Release notes for 10.2.1535.0 version of Windows Store Remote Desktop??

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 67b7b5a4-647e-fee3-6533-6ed88ae45bfa
Version Independent ID: d9102859-0147-346f-8268-6e2a75c8b0b7
Content: What's new in the Microsoft Store Client
Content Source: WindowsServerDocs/remote/remote-desktop-services/clients/windows-whatsnew.md
Product: windows-server
Technology: remote-desktop-services
GitHub Login: @Heidilohr
Microsoft Alias: helohr

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


Changes related to GO:0141189 G protein-mediated opsin signaling pathway

Hi @pgaudet
I've come across some issues when reviewing the annotations related to https://github.com/geneontology/go-ontology/issues/27415
I will just list them here. Happy to leave this until Monday meeting on the 15th.

I came across a GO:0016056 'rhodopsin mediated signaling pathway' which I think should be merged into GO:0141189 'G protein-mediated opsin signaling pathway'. Both are in the GPCR branch.
GO:0016056 'rhodopsin mediated signaling pathway' is a 'part_of' phototransduction, visible light but 'G protein-mediated opsin signaling pathway' is_a 'phototransduction'. As  suggest that there are opsins that can be activated by UV light, so the more general 'phototransduction' seems more correct.

GO:0141189 'G protein-mediated opsin signaling pathway' has a is_a relationship with GO:0007200 'phospholipase C-activating G protein-coupled receptor signaling pathway', but as we now have a transducin and phospholipase C-activating  children the parentage needs to be re-jigged, something like:

Think that we could just obsolete GO:0016060 metarhodopsin inactivation - these are all D.mel annotations, so I can move these.

I could do with a new regulation term 'negative regulation of phospholipase C-activating phototransduction signaling pathway'

Thanks @hattrill
I will obsolete the new term GO:0141189 'G protein-mediated opsin signaling pathway' and replace it by GO:0016056 'rhodopsin mediated signaling pathway'. Annotation by Flybase and ZFIN will be automatically replaced.

After discussion with @hattrill , I will rename GO:0016060 metarhodopsin inactivation  >> negative regulation of phospholipase C-activating opsin-mediated signaling pathway, since this is what this term describes.


CreateErrorResponse instead of CreateResponse

At the end of message handler you call Request.CreateResponse. Is it possible that you actually use Request.CreateErrorResponse, or make it at least configurable.
This is important because I use other message handlers made by myself and they treat your response message as actual data (instead of threating them as errors).
I know this is small change in code but probably breaking one, but making strategy about returning response configurable would be great

You can override that method and use CreateErrorResponse instead. Both CreateResponse and CreateErrorResponse have a return type of HttpResponseMessage.


Failure to upload source bundle with sentry-cli on Windows if project.build.directory contains whitespace

Maven Version
3.9.6
Version
0.0.7
Sentry SDK Version
7.6.0
Steps to Reproduce

On a Windows computer, create a new Spring Boot project in a subdirectory which has a space in it (e.g. "C:\Users\FirstName LastName\Documents\Project")
Add sentry-maven-plugin and sentry-spring-boot-starter to the project
Configure sentry-maven-plugin to upload a source bundle for the project
Build the project with maven from the command line (use '-X' flag for debug logs from maven)

Expected Result
The project should build correctly and upload a source bundle.
Actual Result
Invoking sentry-cli to upload the source bundle will fail because the plugin is not enclosing directory names in quotes.
C:\Users\FOOBAR~1\AppData\Local\Temp\.sentry-cli13348786534883263189.exe --log-level=debug [......] --output=C:\Users\Foo Bar\Documents\project\target\sentry\source-bundle [...] C:\Users\Foo Bar\project\src\main\java

Thanks for the report @btpnlsl we'll make sure to fix this.


YAML encoder that receives an io.Writer

Building a Go tool emit multiple files, I need to be able to emit JSON and YAML based on CUE values.
For JSON, I have
func writeJSON(path string, v cue.Value) error {
	f, err := os.Create(path)
	if err != nil {
		return err
	}
	defer f.Close()

	enc := json.NewEncoder(f)
	enc.SetIndent("", "    ")
	return enc.Encode(v)
}

(just using Go's encoding/json)
For YAML it's like
import "cuelang.org/go/pkg/encoding/yaml"

// notionally:
// function writeYAML(path string, v cue.Value) error {
// but I'm assembling from my code
str, err := yaml.Marshal(v)
if err != nil {
  return err
}
bs := []byte(str)
f, err := os.Create(path)
if err != nil {
	return err
}
defer f.Close()

_, err = f.Write(bs)
}

It'd be great to have a streaming YAML encoder. I think encoding a string, converting to bytes, and then writing is probably not my biggest performance problem, but it couldn't hurt.

@nyarly did you perhaps mean to reference<EMAIL_ADDRESS>above? cuelang.org/go/pkg/encoding/yaml is definition of the builtin package encoding/yaml.

I'd found the pkg implementation and was using that. The nice thing about using Encode is that it would be a []byte value already, so we can save a conversion.
Being able to stream the output remains a (perhaps only aesthetic) benefit.

Indeed. The pattern from encoding/json of being able to create encoders/decoders that effectively wrap an io.{Reader,Writer} makes sense.


Error Right at the end of npm run setup

So, I am trying to install chibisafe on my Ubuntu server right now and when I run "npm run setup", select sqlite3 as my db, and finish the setup, I get presented with the following error:
<EMAIL_ADDRESS>migrate /home/aj/chibisafe
knex migrate:latest
/home/aj/chibisafe/src/api/utils/Util.js:18
static uploadPath = path.join(__dirname, '../../../', 'uploads');
^
SyntaxError: Unexpected token =
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/aj/chibisafe/src/api/database/migrations/20210112011802_addSettingsTable.js:1:14)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>migrate: knex migrate:latest
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>migrate script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!  Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>setup script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!  I can't seem to see what's wrong. I can grab that debug log as well if requested.

Wow, turns out I had version 10... I went ahead and upgraded to v17.0.1 and NPM version 8.1.0 and rebooted my server then ran all of the NPM commands and everything went smoothly. Sorry for the hassle! I didn't even think to check the NodeJS version!


Changing logo

Can u tell how to change logo of college ??

Admin>Admin Profile>Edit Profile>Change logo


UnsupportedOperationException: eglCreateWindowSurface() Android 4.1.1 on swith camera or stop

Hi!
I tried to use your library in project and found few bugs.
One of them:
Problem device: ASUS Padfone 2 with Android 4.1.1 (API 16)
App crashes on camera switch or stop streaming.
Exception:
java.lang.UnsupportedOperationException: eglCreateWindowSurface() can only be called with an instance of SurfaceView, SurfaceHolder or SurfaceTexture at the moment, this will be fixed later.
In Camera1ApiManager.clearSurface() method
My solution change argument of this method from Surface to SurfaceHolder:
clearSurface(surfaceView.getHolder());

private void clearSurface(SurfaceHolder texture) {
...
}

I am not expert in video streaming, but I hope you wil use my solution to do your library better.
This fix works fine on all my devices:
ASUS Padfone 2 with Android 4.1.1 (API 16)
Samsung Galaxy Tab Pro with Android 4.4.2
Nexus 5 with Android 6.0.1

fixed. Thanks!!


Added clientname to session policy and environment

Upon reconnect, we needed to determine the correct running session by clientname rather than by IP number because clients may connect via service with dynamic IPs or from different clients over the same IP (NAT). Also i made available the clientname as an environment variable CLIENTNAME inside the session.
The changes consist mainly of looping through of clientname from rdp into the session.

Thanks for this, @hg-sepag
I've had a quick look through this, and it certainly like a worthwhile addition to me.
We are, however, just about to pull v0.9.18 together, and either as part of this (or shortly after) we will be merging #2011 which is going to cause some quite important changes to the SCP code. As I'm sure you're aware, this is an area of xrdp which needs quite a bit of work. #2011 is the first of a series of changes that will make SCP a lot easier to work with.
#2011 will break your code here, but it will be a simple matter to fix. At that point we can have a serious go at merging this.
A couple of other observations for you in the meantime:-

You'll need to change the value of CLIENT_INFO_CURRENT_VERSION in common/xrdp_client_info.h. There's a comment in that file here which explains that.
The comment about the Policy variable in sesman/sesman.ini.in will need updating.
The manpage template docs/man/sesman.ini.5.in will need updating.
I'm not so sure adding the CLIENTNAME to the session is a good idea, as for use-cases other than yours, the client can change between connections, or even there there at all. In the longer-term, it will be possible to query sesman to get the information directly, but we aren't there yet for the reasons I've outlined above.

Does that make sense? Happy to chat about any of the above.

Hello @matt335672 ,
thank you for your comments. Sorry to answer so late, but immediatly after this pull request i went on holiday :-)

1. You'll need to change the value of CLIENT_INFO_CURRENT_VERSION in common/xrdp_client_info.h. There's a comment in that file here which explains that.
2. The comment about the Policy variable in sesman/sesman.ini.in will need updating.
3. The manpage template docs/man/sesman.ini.5.in will need updating.

This all is quite clear, i simply did not know and will do it in the next couple of days.

4. I'm not so sure adding the CLIENTNAME to the session is a good idea, as for use-cases other than yours, the client can change between connections, or even there there at all. In the longer-term, it will be possible to query sesman to get the information directly, but we aren't there yet for the reasons I've outlined above.

This behavior is similar to M$ Windows: here %CLIENTNAME% is also set at logon via RDP. It is true that our use case is probably special, but i think the same is true for the other session allocation policies except "per user".

Hi @hg-sepag
Thanks for the reply. Definitely worth putting in a PR before having a holiday - nothing much happens for a few days anyway :-)
About my comment 1 - with more thought on my part it's probably better to just more the member declaration further down in the file to somewhere below the 'private to xrdp' comment. That way, the change will work without needing to also recompile xorgxrdp, if you're using that.
I take your point about the %CLIENTNAME% compatibility - I've just reproduced this myself on a Win 10 target. I'm happy with your justification, but we can put the point to the rest of the team when the PR is re-worked.
#2011 is now merged. As a result, all the SCP V0 code is in one place so when you re-work the PR you will need to consider what to do about the sesrun utility, as this will now fail to compile when you modify SCP. You've already got the clientname string as an optional part of the protocol, so you may also need to test for a non-null value before setting the CLIENTNAME environment variable.
Hope that makes sense - it's getting late here so it may not do. Let me know if you have any questions.

The xserver resource database is a clever idea. It does tie us in to an X server though, which looking (a fair way) ahead may not be the best way.
xrdp-sesadmin can be made as complex or easy to use for this purpose as we like - it's within our gift. We could use something like xrdp-sesadmin --query-session client_name for example. In fact we could even use xrdp-sesadmin as a wrapper around xrdb so that the user doesn't need to be concerned with the implementation detail.

I'm afraid i cannot contribute much. I just built in that clientname was
transfered into the environment but don't know anything about x sessions.
Moreover, i didn't find time to integrate it properly into the main
branch:-(
Am 09.11.2022 um 10:29 schrieb matt335672:

The xserver resource database is a clever idea. It does tie us in to
an X server though, which looking (a fair way) ahead may not be the
best way.
|xrdp-sesadmin| can be made as complex or easy to use for this purpose
as we like - it's within our gift. We could use something like
|xrdp-sesadmin --query-session client_name| for example. In fact we
could even use |xrdp-sesadmin| as a wrapper around |xrdb| so that the
user doesn't need to be concerned with the implementation detail.
—
Reply to this email directly, view it on GitHub
https://github.com/neutrinolabs/xrdp/pull/2099#issuecomment-1308464840,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AWUQM3CVWAVF2ZBNB7OEHO3WHNVIZANCNFSM5K6ASOLA.
You are receiving this because you were mentioned.Message ID:
@.***>


Refactor differencerunarray

What value is this adding? I see no performance advantage over arrayIterator (bitmap iterator is a different story)

the drive for this change was lots of heap allocation in the array iterator if I'm remembering correctly. I'm sure the code could be cleaner, I wrote it in the middle of the night last Thursday.


Fix LRO callables so that the ApiCallContext is always passed through.

This is the next step in integrating OpenCensus (#583). My plans for OpenCensus involve passing a wrapped span through ApiCallContext. Unfortunately as it stands, LRO callables don't respect ApiCallContext and drop it. This PR aims to fix the situation by always passing the ApiCallContext and removing null checks along the way.
Please note that this is a behavioral change: LRO polling attempts will now use the RPC in the RetrySettings that OperationTimedPollAlgorithm was instantiated with.

Codecov Report

Merging #600 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #600      +/-   ##
============================================
+ Coverage     75.01%   75.07%   +0.06%
  Complexity      933      933
============================================
  Files           176      176
  Lines          4074     4081       +7
  Branches        322      321       -1
============================================
+ Hits           3056     3064       +8
  Misses          865      865
+ Partials        153      152       -1

Impacted Files
Coverage Δ
Complexity Δ

...ava/com/google/api/gax/rpc/RecheckingCallable.java
90% <ø> (ø)
2 <0> (ø)
:arrow_down:

.../com/google/api/gax/rpc/OperationCallableImpl.java
100% <100%> (ø)
5 <3> (ø)
:arrow_down:

...n/java/com/google/api/gax/rpc/AttemptCallable.java
77.77% <100%> (+4.09%)
3 <0> (ø)
:arrow_down:

...om/google/api/gax/rpc/CheckingAttemptCallable.java
76.47% <100%> (+5.04%)
3 <2> (ø)
:arrow_down:

.../google/api/gax/rpc/OperationCheckingCallable.java
81.25% <100%> (+2.67%)
4 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed11321...0f78b71. Read the comment docs.

@vam-google Are you ok with the behavioral change? If not, I can add code that explicitly disables rpc timeouts for LRO polling, but I think that not having RPC timeouts for LRO polling is a bug

@igorbernstein2 please check the fix for the rpcTimeout issue you experienced.
In general, it seems we, during retries implementation, paid all the attention to the retries from the point of view of client (i.e. the actual operaiton retries and timeouts), but did not pay enough attention to the rpcTimeout value (all the bugs found are about the rpcTimeout stuff). For example, the existing tests for retries and retrying executor do not even validate the value of rpcTimeout in any way. Additional action item should be to add the tests... But it is out of scope of this PR.

@vam-google I think your commit should be done separately from this PR. The change doesn't fully address the problem. The outstanding issue now is that the initial rpcTimeout value might be skipped for the first poll. For example, take Bigtable's CreateInstance rpc, which returns an Operation to be polled:

initialCallable is invoked and return an ApiFuture<Operation>, that has not resolved yet
that future is passed to OperationCheckingCallable
which in turn is wrapped in a CheckingAttemptCallable
RecheckingCallable will then create a CallbackChainRetryingFuture which will create the first TimedAttemptSettings. Using the initialRpcTimeout.
RecheckingCallable then will call CheckingAttemptCallable.futureCall(), which will call OperationCheckingCallable.call
Since the initialFuture is incomplete,OperationCheckingCallable will not start a new rpc and will ignore the current rpcTimeout and simply return the initialFuture
CallbackChainRetryingFuture will consider this the first attempt and once it completes, it will move on to the next poll attempt using the 2nd rpcTimeout.

The first rpcTimeout will be skipped. However, if the initialFuture completes before step 5, then the initial rpcTimeout will be used.  The actual impact of this issue is almost nonexistent because almost all usecases will use a constant rpcTimeout.
Can we just compromise on this: I'll revert your commit and remove my comment and open a github issue that can be prioritized separately?

@igorbernstein2 The described behavior is by design.
 CheckingAttemptCallable does not have futureCall() method, it has only the call()  method. In general, one of the really confusing things in this whole process (it confuses me every time I see it myself) is that there are two "types" of Callables participating here: the gax-defined "callables" (like UnaryCallable, RecheckingCallable), which declare method futureCall(), and the "real" callables, which extend java.util.concurrent.Callable (like CheckingAttemptCallable).
The futureCall() is the fundamental part of GAPIC clients surface and defines the asynchronous nature of all rpc calls in the gapic libraries (i.e. if there is futureCall() called, it means that there should be an rpc call at the end of the call chain "journey").
The normal callables are implementation detail, which define how that asynchronous  behavior is shoved into JDK's concurrency model. Not each call to call() is related to an actual rpc call (they are pretty much not related).
}
The behavior you described affects not only rpcTimeout but also the retryDelay (poll delay in this context), and it was by design as well. The meaning of  OperationCheckingCallable is to encapsulate the "start & poll" behavior. To put it simple: the initial call (the one which starts the LRO) is a "first class member" of the polling process, and each "not ready" status from initial call is handled same way as "polled, poll returned "i'm not ready"). Or from another point of view: each poll, which returns "not ready" is considered to be equal to initialFuture not being completed yet, because the user makes only the initial request, and from his point of view the whole LRO thing is equal to one really long rpc.
In general, you are right, rpcTimeout for polling settings are supposed to use multiplier 1 (and I think it is actually the case for our clients). Also, in most cases the retryDelayMultiplier is also 1, which makes the whole described behavior irrelevant for practical cases.
About the bug
At the same time, the ExponentialRetryAlgorithm really has a bug, where it treats initialRpcTimeout same way as initialRetryDelay, which is simply wrong, because initialRetryDelay becomes relevant only on the first "retry" (i.e. the second "try"), while initialRpcTimeout is relevant for from the first try (and in most cases retry is not even a thing). I.e. initialRpcTimeout is used always, initialRpcTimeout is used only when there was a failure. They are fundamentally different things, which were treated in same way, which is wrong. This is a retrying logic bug and it exists for simple unary callables (no any realation to LRO).


add anvil proposal document

Part of #336
Proposal is a first draft. I will update this PR when I consider it complete.
Feel free to suggest changes.

This is now ready to merge. Once it is merged, I will submit the proposal, using the link https://github.com/Crissov/unicode-proposals/blob/master/pdf/anvil.pdf as the "public url" for the pdf.
Thanks. :)

more editing to be done in accordance with Unicode wanting more references

with https://github.com/alifeee/unicode-proposals/commit/cb8aa2d0f814494b91fe656fac6c43b5a6aafb3d this is now complete.
I have made the submission.
I linked to https://alifeee.co.uk/anvil/anvil.pdf on https://alifeee.co.uk/anvil/


Fix misspelling

I just noticed this misspelling ("fronzen").

Gotcha! I'll go ahead and merge in any spelling mistakes I spot w/o review.


added token parameter for getting channels

Max limit for getting channels is 100 then you will have to provide token parameter which is received in response of previous requests as next

Not fully implemented.....


feat: add systemd unit for scriptless

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] tested upgrade from previous version

Special notes for your reviewer:
Release note:
none

Pull Request Test Coverage Report for Build<PHONE_NUMBER>8
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.271%

Totals

Change from base Build<PHONE_NUMBER>2:
0.0%

Covered Lines:
2590

Relevant Lines:
3634

💛  - Coveralls

Successful e2e run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=104976927&view=results
Corresponding build: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=104963277&view=results


Fixing so it will compile on Linux

Added C++17 flags so will compile with gcc8+ using <filesystem>
Added #include <cstring> in files using memcpy
Added using std::memcpy
Fixed 'non-const reference cannot bind to a temporary object' compiler
error that is ignored by MSVC compiler extension but not GCC.

Do not have windows environment to confirm changes do not break on Windows.

Thanks for the PR! I can't verify it right now since I don't have a Linux environment but I'll check it out as soon as I can.

@seanfdnn thank you, man!  With gcc 8 I can't use 2.0+ because of c++20 feature executors (or something) is not available (it is availible since gcc 9) and I can't compile 1.x, because of this. :facepalm:
With your changes everything looks fine :+1:


2.12: new cats-effect failure

seen at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/1998/consoleFull
[cats-effect] [error] /home/jenkins/workspace/scala-2.12.x-integrate-community-build/target-0.9.9/project-builds/cats-effect-db3ea0a31df5a4c7495c66e0748f3b89148128e0/laws/shared/src/main/scala/cats/effect/laws/EffectLaws.scala:50:
  assignments in argument position are deprecated in favor of named arguments. Wrap the assignment in brackets, e.g., `{ cur = ... }`.
[cats-effect] [error]     val change = F.delay(cur = f(cur))
[cats-effect] [error]                              ^

root cause is https://github.com/scala/scala/pull/6089 (hello @lrytz)

@djspiewak could you make the needed changes to the cats-effect sources? (if you can do it fairly soon, awesome. otherwise, I can just disable -Xfatal-warnings for cats-effect in the context of the community build)

@SethTisue I'll take care of it this week.  Sorry for the delay.

no worries, that's plenty fast

@SethTisue This should be fixed as of 284268a.  Do you need a release?

Nope, if it’s on master, the community build will pick it up. I’ll check on it on Monday. Thanks!

Adriaan had temporarily disabled -Xfatal-warnings in 7c29e2b648ef6ff7340553567d13ee9b0370e9c7; now reverted. green in a local test


Create declarations for interop interfaces in winrt

At the moment, if you want to use the interop types in e.g. windows.ui.composition.interop.h you have to include the include file from the sdk which declares them in ::ABI and you have to consume them with a winrt::com_ptr. It would be nice to be able to use those directly in winrt or at the least have them in winrt::ABI. Otherwise, using namespace winrt; makes ABI ambiguous meaning you have to say ::ABI or winrt::ABI.

You have a few options. You can be explicit as follows:
using namespace winrt;

com_ptr<::ABI::Windows::UI::Composition::ICompositionDrawingSurfaceInterop> p;

Notice the scope resolution operator before the ABI namespace.
Another option is to use a namespace alias:
using namespace winrt;
namespace sdk_abi = ::ABI::Windows::UI::Composition;

Compositor compositor;
com_ptr<sdk_abi::ICompositorInterop> interop = compositor.as<sdk_abi::ICompositorInterop>();

You can also combine different "ABI" namespaces into one:
namespace sdk_abi
{
    using namespace ::ABI::Windows::UI::Composition;
    using namespace ::ABI::Windows::UI::Xaml;
}

Hope that helps.


Establish CI/CD

The web application should be built and deployed with each merged PR. This will allow us to establish Continuous Integration and Continuous Deployment (CI/CD) from the outset.
There will be two versions of the app online - app-latest should deploy whenever main is updated and app-stable should deploy whenever we do a release.

@chinedu117 will deploy manually to opencdms.surge.sh until the app is configurable with components (see #5).  We will then set up automatic deployment to S3 and deploy at app.opencdms.org.

Automating the deployment to S3 has been more complicated than expected. This issue will continue into the next sprint


Fix building glslang on platforms without

Certain platforms using older libc++ versions have <filesystem> support disabled, which causes the builds after 7f1d3b1cfad648be5009b67b37ee8f6ea1061d8e  to fail.
As there's already been a hack added for similar issue on macOS in upstream, this include is unused and can be removed.

Makes sense. Could you add that change in the patch file I added to document the changes we made to upstream glslang? This file will be used to reapply the changes next time we update, so if it's not updated we'll reintroduce this include by mistake.
https://github.com/godotengine/godot/blob/master/thirdparty/glslang/glslang/patches/disable-absolute-paths-for-apple-compat.patch
While at it, I notice I put in the wrong place (should be in thirdparty/glslang/patches, not thirdparty/glslang/glslang/patches. So you could move it one folder up.
I can also do it if you're not sure how to tweak the patch to add another section, let me know.

done.

Thanks!


GitHub Actionsの非推奨記法の解消。

現在こんな警告が出ている。
[lambda-deploy](https://github.com/koshikita-volleyball/turnip/actions/runs/4844958457/jobs/8633566476)
Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: aws-actions/configure-aws-credentials@v1. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
[lambda-deploy](https://github.com/koshikita-volleyball/turnip/actions/runs/4844958457/jobs/8633566476#step:5:15)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
[nextjs-build](https://github.com/koshikita-volleyball/turnip/actions/runs/4844958457/jobs/8633581760)
Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: aws-actions/configure-aws-credentials@v1. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
[nextjs-build](https://github.com/koshikita-volleyball/turnip/actions/runs/4844958457/jobs/8633581760#step:2:13)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

これを修正したい。

Closed #38