Switch t-SNE in scanpy to openTSNE

I'm not sure what t-SNE implementation is currently used in scanpy, but would it make sense to switch it to openTSNE? It's a Cython re-implementation of FIt-SNE, it's available on conda and should be very easy to depend on.
As far as I understand the scanpy architecture, it builds a kNN graph and then runs downstream analysis (like UMAP or Louvain) on this kNN graph. Is that right? I suppose this is currently not implemented for t-SNE? With openTSNE it'd be easy to use the pre-built kNN graph and run t-SNE directly on that.
Also, the default parameters of t-SNE in scanpy could IMHO be improved, see  https://www.nature.com/articles/s41467-019-13056-x. Some of these recommendations (learning rate, initialization) are now default in openTSNE.
There are some open issues at scanpy related to t-SNE such as https://github.com/theislab/scanpy/issues/1150 and https://github.com/theislab/scanpy/issues/996 but I think this suggestion would supersede them. We had a brief discussion of this at openTSNE here https://github.com/pavlin-policar/openTSNE/issues/102.
I can see four somewhat separate suggestions:

switch scanpy to using openTSNE for tSNE, using already constructed kNN graph
add tSNE support for ingest using openTSNE functionality.
change default tSNE parameters (n_iter, learning rate, initialization) following openTSNE defaults.
add some tSNE "recipes" based on https://www.nature.com/articles/s41467-019-13056-x

What of this, if any, makes sense from the scanpy point of view?

I'm not sure what t-SNE implementation is currently used in scanpy, but would it make sense to switch it to openTSNE? It's a Cython re-implementation of FIt-SNE, it's available on conda and should be very easy to depend on.

We use MulticoreTSNE if it's installed, but fall back to sklearn.

As far as I understand the scanpy architecture, it builds a kNN graph and then runs downstream analysis

Right now, we tend to use a connectivity graph built by UMAP, but are working on making this more generic. We're thinking about allowing the UMAP embedding to be generated on graphs we provide as well.

switch scanpy to using openTSNE for tSNE, using already constructed kNN graph

I think I'd like to see this. That package is much more actively maintained than our current backend, and looks interesting. I would like it if the TSNE was flexible about the graph that was used.
I'm not sure that I'll get to this, but a PR would be welcome. I'd have to see some performance/ results before thinking about changing the defaults, or whether this would go into a major or minor version change.

add tSNE support for ingest using openTSNE functionality.

@Koncopd do you have any thoughts on this?

change default tSNE parameters (n_iter, learning rate, initialization) following openTSNE defaults.

Again, I'd have to think about backwards compatibility. Maybe this could start as a sc.tl.opentsne function?

add some tSNE "recipes"

I'd be interested in this. Skimming that paper now, I really like the idea of showing regions of uncertainty for projection would be very useful. I'd be interested in how these "recipes" could be wrapped in a function.

Thanks! Can I ask for two clarifications before replying:

Right now, we tend to use a connectivity graph built by UMAP ...

UMAP uses Pynndescent to construct the kNN graph. So does it mean that you depend on Pynndescent to construct the kNN graph, and then if the user calls UMAP, it's run on this previously constructred kNN graph?
By the way, openTSNE uses Annoy instead of Pynndescent for non-sparse input data and simple metrics that are supported by Annoy (like Euclidean or cosine). It seems to work quite a bit faster. For sparse input data and/or fancy metrics, it uses Pynndescent.

... but are working on making this more generic. We're thinking about allowing the UMAP embedding to be generated on graphs we provide as well.

What are the use cases here that you thinking of?

That sounds mostly right. We use the nearest_neighbors function from umap, which uses pynndescent if it's installed
https://github.com/theislab/scanpy/blob/5bc37a2b10f40463f1d90ea1d61dc599bbea2cd0/scanpy/neighbors/__init__.py#L270-L280

By the way, openTSNE uses Annoy instead of Pynndescent for non-sparse input data and simple metrics that are supported by Annoy

I'm curious about how much the backend changes the runtime and results of nearest neighbors methods. I'm definitely for being more generic about how the neighbors graph is generated and weighted. I haven't seen anything yet which looks at the character of the inaccuracies for each method, something that's probably important when they're used for classification.

What are the use cases here that you thinking of?

Mainly cases of merged graphs, like when you have multiple datasets or modalities.

I'm curious about how much the backend changes the runtime and results of nearest neighbors methods.

You can see some quick comparisons between Pynndescent and Annoy here: https://github.com/pavlin-policar/openTSNE/issues/101#issuecomment-597178379. But I have not investigated it very thoroughly.
Anyway, returning to the main conversation:
I think switching to openTSNE makes sense even if nothing else that we are discussing is implemented. It's A LOT faster than Mutlicore t-SNE for large datasets: https://opentsne.readthedocs.io/en/latest/benchmarks.html. It is also more flexible, actively supported, conveniently packaged/distributed, etc. I don't see any possible disadvantage. You could potentially keep all the default parameters as you have now in scanpy (even though I would not recommend it, see below).
However, what I said about using pre-build kNN graph requires some thinking. T-SNE uses perplexity=30 by default and uses kNN graph with k=3*perplexity, so that's 90 by default. UMAP uses k=15 and that's what you use in scanpy by default too. I can see three options here:
i) Let openTSNE do its own thing and ignore the kNN graph built in scanpy. Advantage: that's what you do now. Disadvantage: not very consistent architecture IMHO.
ii) Use the kNN graph built in scanpy and query() it to get 90 neighbors. Disadvantage: can be a bit slow. But I think it's better than (i).
iii) Run t-SNE using 15 neighbors. Turns out, t-SNE with uniform affinities across 15 neigbours is extremely similar to t-SNE with perplexity 30. Evidence: https://twitter.com/hippopedoid/status/1232698023253303298. So you could run this version of t-SNE with uniform kernel. This will be very fast.
Regarding default parameters: learning rate = 1000 that you use by default is simply not enough for large data (sample size in millions), as shown in that Nat Comms paper in detail. If you want to keep it for compatibility reasons, that's your choice, but be aware that you are getting suboptimal tSNE embeddings. The same about initialization: UMAP smartly uses Laplacian Eigenmaps to initialize, but sklearn/multicore tSNE use random init, which is simply a bad choice (as again shown in that paper). openTSNE now uses PCA init by default which is much more sensible.

Glad to see this discussion going on. Integrating openTSNE into scanpy should be fairly straightforward but may require some thought. I think Dmitry has already pointed out the most important things such as improved defaults, which other t-SNE implementations are lagging behind in. Apart from that, we package prebuilt binaries, so adding openTSNE as a dependency would be incredibly easy
The main thing we'd have to agree on is how to deal with the KNN graphs. UMAP by default calculates 15 nearest neighbors, and from what I can tell, louvain and leiden clustering both use those 15 neighbors as well by default. t-SNE, on the other hand, calculates 90 nearest neighbors by default. This is every single t-SNE implementation, not just openTSNE. Dmitry suggested using a uniform kernel with 15 neighbors, which would fit elegantly, but then again, this isn't truly t-SNE anymore, but rather something very close.
The same goes for the ingest functionality. openTSNE does something similar to UMAP for adding new samples to existing embeddings, and then we'd again have to figure out how to calculate nearest neighbors from the new data to the reference data. I don't know how you do this currently for UMAP.
I'm not exactly sure how these neighbors are meant to be used in scanpy, since there are several different algorithms that use them. Graph-based clustering uses the KNNG, UMAP uses it, forceatlas2 uses it, PAGA probably as well? Is relying on a single k=15 from UMAP for everything really ok? For example, seurat defaults to using 30 neighbors for clustering.

Pavlin made good points. @ivirshup What are you thoughts?

UMAP by default calculates 15 nearest neighbors, and from what I can tell, louvain and leiden clustering both use those 15 neighbors as well by default.

Both of those clustering algorithms just use whatever graph is passed, so this shouldn't be an issue.

t-SNE, on the other hand, calculates 90 nearest neighbors by default.
openTSNE does something similar to UMAP for adding new samples to existing embeddings

Could there just be a separate function for computing neighbors for tsne? sc.pp.neighbors can be considered to be "compute nearest neighbors and connectivity as expected by UMAP", while a separate function could use methods and defaults appropriate to openTSNE.
@Koncopd would have more to say on how this should work w.r.t. ingest.

Is relying on a single k=15 from UMAP for everything really ok?

Ultimately, up to the user. There is an element of consistency and simplicity to using the same representation of the data for multiple parts of the analysis. I think there would have to be a good reason for using a different connectivity matrix for the 2d embedding and for the clustering.

Yep, UMAP can use a graph with any number of neighbors, so can ingest.

@ivirshup @pavlin-policar I'd like to get back to this issue. I think maybe we should postpone discussing ingest until later (as well as "recipes" based on our Nat Comms paper, and other topics raised above) and focus on switching to openTSNE first.
Scanpy's architecture is to compute kNN graph by calling sc.pp.neighbors and then run dimensionality reduction (UMAP) and clustering (Leiden) on this graph. I think this is very neat and makes everything consistent and other functions should follow this approach as much as possible. So IMHO if it's possible to run t-SNE on the kNN graph with k=15, then that's what we should do.
And luckily it is possible! I can even see two approaches. (1) Either use k=15 kNN graph with the uniform similarity kernel. As I said, and as Pavlin knows, this yields result that is very similar to using perplexity=30. (2) Or use k=15 kNN graph with UMAP weights, normalize it as t-SNE expects it to be normalized and use that. My expectation is that it would yield very similar results, but I haven't actually tried it.
Admittedly, this is not the "vanilla" t-SNE. But it's very close. And I think advantages outweigh the disadvantages. Actually this is quite a bit faster than the standard t-SNE, because it only uses k=15 instead of k=90 (3 times perplexity=30).
Moreover, we could make the standard t-SNE available by extending sc.pp.neighors with method="tsne" (there are several methods there already). What I mean is that
sc.pp.neighors()
sc.tl.tsne()

would use let's say uniform kernel on k=15 kNN neighbor graph (and maybe print a warning about it, but I am not even sure it's needed), while
sc.pp.neighbors(method="tsne", perplexity=30)
sc.tl.tsne()

would construct k=90 weighted kNN graph as standard t-SNE does and then use that.
Either way, sc.tl.tsne() runs openTSNE with the pre-defined affinity matrix.
Thoughts?

I think it would make a lot of sense to at least switch the t-SNE implementation to openTSNE at the very least. For the recipes, there' already something similar in the preprocessing module. So I'd imagine calling standard t-SNE with sc.tl.tsne and the recipes like sc.tl.tsne.recipe_multiscale.

And luckily it is possible! I can even see two approaches. (1) Either use k=15 kNN graph with the uniform similarity kernel. As I said, and as Pavlin knows, this yields result that is very similar to using perplexity=30. (2) Or use k=15 kNN graph with UMAP weights, normalize it as t-SNE expects it to be normalized and use that. My expectation is that it would yield very similar results, but I haven't actually tried it.

I very much prefer option 1. If I understand option 2 correctly, we would normalize the 15 neighbors to essentially perplexity=5. I've never once found a case where that is useful, so having this as the default behaviour in scanpy seems like a really bad idea (I foresee a lot of issues in the style "why is t-SNE not working?"). Using a uniform kernel produces results that are virtually indistinguishable from vanilla t-SNE, so that's fine IMO, and it's faster as well. It's still less than the default perplexity=30, but this seems like the best option. Whatever we agree on, the same can be applied to the ingest functionality, so adding that would also be straightforward.

Moreover, we could make the standard t-SNE available by extending sc.pp.neighors with method="tsne" (there are several methods there already).

I don't understand this, why would this belong on sc.pp.neighbors? The graph weighing should go into the sc.tl.tsne call. Are the UMAP weights assigned to the graph in sc.pp.neighbors? That seems questionable. I would expect the output to be a directed, unweighted graph, and let each method take care of the graph. If anything, I'd expect it to weight it using the Jaccard index of shared nearest neighbors, which seems to me like pretty much the standard thing to do in single-cell analysis.

I don't understand this, why would this belong on sc.pp.neighbors? The graph weighing should go into the sc.tl.tsne call. Are the UMAP weights assigned to the graph in sc.pp.neighbors?

Yes, this is my understanding of how it works in scanpy. See https://scanpy.readthedocs.io/en/stable/api/scanpy.pp.neighbors.html:
method : {‘umap’, ‘gauss’, ‘rapids’}, None (default: 'umap')

    Use ‘umap’ [McInnes18] or ‘gauss’ (Gauss kernel following [Coifman05] with adaptive width [Haghverdi16]) for computing connectivities. Use ‘rapids’ for the RAPIDS implementation of UMAP (experimental, GPU only).

If I understand option 2 correctly, we would normalize the 15 neighbors to essentially perplexity=5.

That's not what I meant. I meant taking UMAP's weights for k=15 and normalizing the matrix so that it sums to 1, as in t-SNE. That said, I would also prefer option 1 because I don't want anything that feels like it's a UMAP/t-SNE hybrid.

Yes, this is my understanding of how it works in scanpy.

Thinking about this a bit further, yes, that does make perfect sense. I suppose we'd want to add a "tsne" option there as well then.

That's not what I meant. I meant taking UMAP's weights for k=15 and normalizing the matrix so that it sums to 1, as in t-SNE. That said, I would also prefer option 1 because I don't want anything that sounds like it's a UMAP/t-SNE hybrid.

I see now and I completely agree with you: I don't really like this. This would then be some strange mesh between UMAP and t-SNE. It feels messy.


Testing Data and Random Seed

Hi,
I wanted to know that in main.py (Line 293- the comment) if we had meta-batch size as 1, then would doing 600 meta-iterations be equal to choosing 600 tasks?
Also, did you evaluate the test results with different random seed values? I was curious as I was getting significant differences in accuracies in my PyTorch implementation, and I wanted to know if this was observed by you as well.
Thanks

Hi @anishmadan23
Thanks for checking out our code and opening this issue.

I wanted to know that in main.py (Line 293- the comment) if we had meta-batch size as 1, then would doing 600 meta-iterations be equal to choosing 600 tasks?

Line 293 in main.py makes a comment about how many different tasks (600) will be available in the meta-validation set dataloader. See also line 284 which mentions that 200,000 is the number of different tasks available in the meta-training set dataloader.
I don't understand your question completely ("choosing 600 tasks"), but I'll give it a try: to have meta-learning you need meta-batch size > 1. This is because you want to meta-learn (i.e. take an outer gradient step) across multiple (>1) tasks. Technically speaking you are right: having meta-batch size = 1 and doing 600 meta-iterations is equal to choosing 600 tasks (1 at every meta-iteration).

Also, did you evaluate the test results with different random seed values? I was curious as I was getting significant differences in accuracies in my PyTorch implementation, and I wanted to know if this was observed by you as well.

I remember that for different runs (seeds) of this code we obtained similar consistent results (as in our journal paper), but didn't report on them (the different runs). Note that taking an average across many test tasks is crucial to reproduce the accuracy results, as performance can vary a lot. Other work (MAML++) has shown that MAML is quite sensitive to seed values (at least when re-implemented in PyTorch).
Hope this helps!

Thanks for the reply.
I had to apply the 600 meta-iterations to the validation set, hence did not require the outer gradient step. My code was structured in such a way that the difference between choosing tasks and using the dataloader for loading each task created some confusion. So thanks for clarifying that. I will also look at the MAML++ paper for the seed issue.


Add word: carabiner/karabiner

I would spell it 𐑒𐑨𐑮𐑩𐑚𐑰𐑯𐑼 in my dialect.
The Wikipedia page says /ˌkærəˈbiːnər/.

Doneǃ I agree with the spelling too.


GH-44950: [C++] Bump minimum CMake version to 3.25

This is currently under development. I am only exercising CI to find out what things would require to be updated
This is still being decided and a decision hasn't been finalized yet.
Rationale for this change
We want to upgrade our CMake version to 3.25 as discussed on the ML:
https://lists.apache.org/thread/h8jp16ktrj11fmjmjhlg6xvkvv9wzvjk
What changes are included in this PR?
TBD
Are these changes tested?
Yes, via CI.
Are there any user-facing changes?
Yes, the minimum CMake version to be used to build Arrow is bumped to 3.25.
This PR includes breaking changes to build systems.

GitHub Issue: #44950

I am unsure on how to fix the remaining failures for R.
For the gcc 12 job I am unsure why it is failing and for the Windows C++ RTools 40 ucrt64 it seems we install CMake from MINGW here but I am not sure if this is necessary or can be updated.
https://github.com/apache/arrow/blob/main/ci/scripts/PKGBUILD#L39

For the gcc 12 job I am unsure why it is failing

Updating the CMake lower bound will flip the default of several policies from legacy to new; sounds like you might be relying on legacy behaviour there in some way (once you figure out which policy is at fault, there's usually a migration path to keep the old behaviour)

I went through the logs of our recent checks ok cran and only one is using a version < 3.25 and that seems incidental then purposely as it's the r-odrel arm64 but the intel version has 3.26
So I don't think we should be forced to use that cmake version, additionally we have a function that fetches current cmake if an unsuited version is found but apparently there is an issue with it as seen above. IIRC there was a change to numeric version in one of the las R Versions that is causing this? I'll have a look.

It's possible the version comparison error is from this: https://github.com/apache/arrow/pull/44989/files#diff-935746c34b16289a07b0d9bf7642dbd268b18059b6187f7cdec7c464be47a3deL731-L743
cmake_version <- function(cmd = "cmake") {
  tryCatch(
    {
      raw_version <- system(paste(cmd, "--version"), intern = TRUE, ignore.stderr = TRUE)
      pat <- ".* ([0-9\\.]+).*?"
      which_line <- grep(pat, raw_version)
      package_version(sub(pat, "\\1", raw_version[which_line]))
    },
    error = function(e) {
      return(0)
    }
  )
}

The error case should probably return("0")

Two other places in the R nixlibs.R script worth updating:

https://github.com/apache/arrow/pull/44989/files#diff-935746c34b16289a07b0d9bf7642dbd268b18059b6187f7cdec7c464be47a3deR643 (bump to the latest? this pinned version was just latest at the last point we updated it)
https://github.com/apache/arrow/pull/44989/files#diff-935746c34b16289a07b0d9bf7642dbd268b18059b6187f7cdec7c464be47a3deR701 (maybe just remove the default value for the input, I think it's only called passing the value in)

The error case should probably return("0")

It would probably be more forward-looking to avoid the error entirely. Why does the function fail parsing the CMake version? Can we add cmake --version somewhere in the GH workflow?

The error case should probably return("0")

It would probably be more forward-looking to avoid the error entirely. Why does the function fail parsing the CMake version? Can we add cmake --version somewhere in the GH workflow?

I could be remembering wrong, but I believe the function is used to check for cmake of a certain version, and this is to be robust to where cmake may not be installed or not found at the path provided. It does not emit an error, it traps it.
This might not be where the error is coming from that was observed in CI, I was just browsing the source to see where you might get a numeric version error. Looking again, and reading the output it produced, I think we're hitting this: https://github.com/apache/arrow/pull/44989/files#diff-935746c34b16289a07b0d9bf7642dbd268b18059b6187f7cdec7c464be47a3deL718
      } else {
        # Keep trying
        lg("Not using cmake found at %s", path, .indent = "****")
        if (found_version > 0) {
          lg("Version >= %s required; found %s", version_required, found_version, .indent = "*****")
        } else {

should be found_version > "0". We must not have been running into this before because the "found_version" was always sufficient if found, and if it wasn't found, we were returning a numeric 0 which works in this comparison.

I've pushed the changes Neal suggested which should fix the ubuntu failure 🤞  (hope you don't mind, @raulcd !)

I did a bit of digging on the windows front, and I suspect what's going on is the the MSYS2/mingw cmake is being preferred (I'm not familiar enough with that ecosystem to know if we can override it with something on the system, but that also seems fragile itself).
https://github.com/raulcd/arrow/blob/e5d521134db4bed8507572fece3b56eb4a1b9158/ci/scripts/r_windows_build.sh#L26-L30 has some info about how to test newer dependencies, I wonder if we (conditionally) use those pacman commands to insall a newer cmake than what's in the CRAN repo (https://cloud.r-project.org/bin/windows/Rtools/4.0/ucrt64/ has the version I'm seeing installed listed, so if we override that one in our builds that might be sufficient.

I did a bit of digging on the windows front, and I suspect what's going on is the the MSYS2/mingw cmake is being preferred (I'm not familiar enough with that ecosystem to know if we can override it with something on the system, but that also seems fragile itself).
https://github.com/raulcd/arrow/blob/e5d521134db4bed8507572fece3b56eb4a1b9158/ci/scripts/r_windows_build.sh#L26-L30 has some info about how to test newer dependencies, I wonder if we (conditionally) use those pacman commands to insall a newer cmake than what's in the CRAN repo (https://cloud.r-project.org/bin/windows/Rtools/4.0/ucrt64/ has the version I'm seeing installed listed, so if we override that one in our builds that might be sufficient.

Unfortunately, that won't help us. https://github.com/r-windows/rtools-packages/ is archived; CRAN has moved away from the toolchain that Jeroen was maintaining. We should delete that comment.
For newer cmake with older rtools, maybe we can get it from https://github.com/Kitware/CMake/releases/download/v3.31.2/cmake-3.31.2-windows-x86_64.zip or something? Or maybe we can install newer rtools for cmake but use the older rtools for compilers etc.?

I don't understand why the job is installing mingw-w64-ucrt-x86_64-cmake-3.21.3-1-any.pkg.tar.x when the remote package seems to be updated to a newer version: https://packages.msys2.org/packages/mingw-w64-ucrt-x86_64-cmake
And we seem to be updating to pull the newest packages:
+ pacman --noconfirm -Syy
:: Synchronizing package databases...
downloading mingw32.db...
downloading mingw64.db...
downloading ucrt64.db...
downloading mirrors.db...

Probably naive question here, I suppose we can't update the mirrors here to point to msys2 and download the newer cmake from there before building Arrow as we would have the same issue when we try to publish to CRAN, right? @jonkeane @nealrichardson

Honestly I don't know. But it's not relevant for CRAN because (a) CRAN currently only builds on R 4.3 and 4.4, both of which have new enough cmake, and (b) we don't require cmake on CRAN anyway because we build the libarrow C++ library in our CI and download it in the CRAN build--CRAN only compiles the R bindings, which do not require cmake.
Solving the cmake issue for R < 4.3 is only for our own purposes of building C++ libraries that are compatible with older versions of R on Windows. We support more versions of R than CRAN actively checks on, though it appears that in CI, we only check on the Windows current release version, and when we test old R versions, we do it on linux.
IIUC we're building libarrow with the rtools4.0 toolchain for maximum compatibility. If we want to continue doing that, we could try installing newer cmake somewhere before https://github.com/apache/arrow/blob/main/.github/workflows/r.yml#L293 and making sure it's on the PATH. Or, to bump up, we would change the rtools version in that job from 40 to 43 to use the rtools43 toolchain with newer cmake.

We support more versions of R than CRAN actively checks on

Do we still want to do that? How is our R support policy decided?

We support more versions of R than CRAN actively checks on

Do we still want to do that? How is our R support policy decided?

We generally follow the tidyverse version support policy, which is the most recent 5 versions. Many enterprise users of R don't upgrade versions eagerly so they can be stuck on older versions longer than CRAN's testing window.
If there is an easy way to get new enough cmake into the R windows-cpp job so that we can keep building with the R 4.0 toolchain, that would be ideal. I don't see why we couldn't do that.

One more reference to 3.16 to update: https://github.com/apache/arrow/blob/main/r/DESCRIPTION#L31

After a bunch of trial and error, I seem to be stuck with the following issue if I override the CMake version to be used to either a manually 3.25.0 installed or using the latest available (3.31) on msys2:
-- Found hdfs.h at: D:/a/arrow/arrow/cpp/thirdparty/hadoop/include/hdfs.h
CMake Error at C:/rtools40/ucrt64/lib/cmake/AWSSDK/AWSSDKConfig.cmake:119 (while):
  while() given incorrect arguments:

    "NOT" "TEMP_NAME" "STREQUAL"

  Unknown arguments specified
Call Stack (most recent call first):
  cmake_modules/FindAWSSDKAlt.cmake:36 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:294 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:5389 (resolve_dependency)
  CMakeLists.txt:546 (include)

I am trying some things, like using an updated aws-sdk-cpp version, but I am unsure where to go from there

The aws-sdk-cpp it's using is coming from https://github.com/r-windows/rtools-packages/tree/master/mingw-w64-aws-sdk-cpp
If you need to update it, I think you'll have to drop it from the dependencies that get downloaded in our PKGBUILD and make it dependency source=BUNDLED. Doesn't look like we're able to make any updates happen to that repo.
cc @assignUser, in case you wanted to start pushing on the MXE direction again as an alternative 🙃

I spent some time trying to build aws-sdk-cpp under Rtools40 and got stuck building aws-c-io. I put my progress in a gist which includes my steps, a patch for aws-c-common, and my build log for aws-c-io.

Update on fixing the Rtools40 build: I've created a set of patches and the awssdk source build works now so I think the approach you were taking @raulcd will work. I'm running into an issue where the patches I record don't re-apply when I rebuild so I'm working on figuring that out now.

I figured out my issue above and now have what looks like a working set of patches so we can build the AWS SDK under Rtools40. My changes are over on https://github.com/raulcd/arrow/pull/90. Still needs some work but hopefully this helps unblock this PR.

Thanks @amoeba for the patches! I had to tweak a little but I did make some more progress. The R / AMD64 Windows C++ RTools 40 ucrt64 is now successful with bundled AWS but the related R / AMD64 Windows R release  job, which validates the built package, is currently failing when trying to install the new Arrow package. It seems to be missing some symbols on the new bundled libraries. Anyone has an idea?
g++ -shared -s -static-libgcc -o arrow.dll tmp.def RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L../windows/arrow-<IP_ADDRESS>00/lib-13.3.0/x64 -L../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt -larrow_dataset -larrow_acero -lparquet -larrow -larrow_bundled_dependencies -lutf8proc -lthrift -lsnappy -lz -lzstd -llz4 -lbz2 -lbrotlienc -lbrotlidec -lbrotlicommon -lole32 -lbcrypt -lpsapi -lcrypto -lcrypt32 -lre2 -luserenv -lversion -lws2_32 -lbcrypt -lwininet -lwinhttp -lcurl -lnormaliz -lssh2 -lgdi32 -lssl -lcrypto -lcrypt32 -lwldap32 -lz -lws2_32 -lnghttp2 -ldbghelp -LC:/rtools44/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools44/x86_64-w64-mingw32.static.posix/lib -LC:/R/bin/x64 -lR
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(file.c.obj):(.text+0x1110): undefined reference to `__imp_PathFileExistsW'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0xcb): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x694): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0xb6a): undefined reference to `DecryptMessage'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x100e): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x12bd): undefined reference to `EncryptMessage'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x1869): undefined reference to `__imp_ApplyControlToken'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x192a): undefined reference to `InitializeSecurityContextA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x1ab3): undefined reference to `__imp_DeleteSecurityContext'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x1ae2): undefined reference to `__imp_DeleteSecurityContext'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x1bc6): undefined reference to `AcquireCredentialsHandleA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x274b): undefined reference to `__imp_AcceptSecurityContext'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x27c5): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x2a0f): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x2be0): undefined reference to `InitializeSecurityContextA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x2c56): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x2e59): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x2fa2): undefined reference to `__imp_AcceptSecurityContext'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x309b): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x31e3): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x342a): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x3447): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x381c): undefined reference to `InitializeSecurityContextA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x3913): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x3a9d): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x3c9a): undefined reference to `FreeContextBuffer'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(secure_channel_tls_handler.c.obj):(.text+0x3cb7): undefined reference to `QueryContextAttributesA'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(windows_pki_utils.c.obj):(.text+0x1825): undefined reference to `NCryptOpenStorageProvider'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(windows_pki_utils.c.obj):(.text+0x1884): undefined reference to `NCryptImportKey'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(windows_pki_utils.c.obj):(.text+0x1bed): undefined reference to `NCryptFreeObject'
C:\rtools44\x86_64-w64-mingw32.static.posix\bin/ld.exe: ../windows/arrow-<IP_ADDRESS>00/lib/x64-ucrt/libarrow_bundled_dependencies.a(windows_pki_utils.c.obj):(.text+0x1bf7): undefined reference to `NCryptFreeObject'
collect2.exe: error: ld returned 1 exit status

I made one more change and CI looks better now. @raulcd

I made one more change and CI looks better now.

Awesome @amoeba ! I'll spend some time to trigger and fix extended CI (crossbow tasks) and we can move this to ready for review

@github-actions crossbow submit -g nightly-tests

@github-actions crossbow submit example-*

@github-actions crossbow submit test-alpine-linux-cpp

@github-actions crossbow submit -g cuda

@github-actions crossbow submit example-python-minimal-build-ubuntu-venv

@github-actions crossbow submit test-alpine-linux-cpp

@github-actions crossbow submit example-python-minimal-build-ubuntu-venv

@github-actions crossbow submit -g cuda

@github-actions crossbow submit -g r

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten test-ubuntu-20.04-cpp-bundled

@github-actions crossbow submit r-binary-packages

@github-actions crossbow submit r-binary-packages

Hey, @raulcd, I see the current failure on the r-binary-packages job and want to help. Do you remember why you reverted https://github.com/apache/arrow/pull/44989/commits/b772b23878fb29a8752e82a6214b7c29a3b80352 originally?

Do you remember why you reverted b772b23 originally?

The Windows job failed because those variables were already defined. I thought the current failure was due to the patch not being applied but I haven't validated.

@github-actions crossbow submit r-binary-packages


⚠ Samsung has degraded performance

In 9100f46, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2360 ms

Resolved: Samsung performance has improved in ee04ee0.


Latitude 9520 driver not found

I'm trying to import the model Latitude 9520 and it's not being found by the Driver Management. https://www.dell.com/support/home/en-pk/product-support/product/latitude-15-9520-2-in-1-laptop/drivers
What triggers new models showing up in the Driver Automation Tool?

It looks at a Dell CAB file that it pulls from their site. You can force a refresh by:

close Driver Automation Tool (DAT)
under where you have DAT installed, look for a Temp folder
delete the two DriverPackCatalog files in that Temp folder
launch DAT and select Dell
click Find Models

I did that a bit ago and the Latitude 9520 shows up.

Thank you, this worked.


Version drop down does not change selected version

Description
When looking at the docs, I tried to select a different version to compare the last release with the current release. I am able to click on the drop down and select the other version, but the page refreshes and brings me back to where I was.
If I manually edit the version string in the URL, that does switch me over to the right version of the docs. But again on the other version, if I try to use the drop down to select the other one to go back, I am now stuck on the version I entered in the URL.
Location

All docs pages, as far as I can tell.
Additional Context

Initially saw this on macOS using Brave. Also observed the behavior using Safari.

I'm seeing the same behavior:

System:

Firefox developer edition 97.0 (64 bit version)
Pop OS 21.04

Seeing the same behavior on Chrome:

Version 98.0.4758.102 (Official Build) (64-bit)

@stmcginnis @jpmcb was the PR we approved at https://github.com/vmware-tanzu/community-edition/pull/3871 in relation to this issue?
If so, can we attach it and close this one. Thanks!

Yep, that was the fix. It should be working now.


Cleanup extension samples and reset release notes

Issue describing the changes in this PR
resolves #1805
Removing WorkerExtensionBinding sample and updating the Extension sample project.
Future work: Updated best practice for output without using the output binding

@liliankasem just an FYI we were using some of these code examples in our documentation. The hint is that for most of these I added snippet tags to the source code. This caused a break in our Azure documentation build until they removed the snippets from the source .md files.
Adding @mattchenderson for awareness.

😱 oh no! I didn't realise, thanks for the info - will keep this in mind with future updates. Is there anything we can do to help update the docs?


2.7 Broken on Windows 11

Description
Just installed the new version of system bridge and the app starts but I can't access any of the settings or the API.
What Platform / OS are you running?
Windows
What version are you running?
2.7
Anything in the logs or a references that might be useful?
021-09-25 14:01:17 info: -----------------------------------------------------------------------------------------------------------------------
2021-09-25 14:01:17 info: System Bridge 2.5.0: ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge.exe","C:\\snapshot\\system-bridge\\dist\\main.js"] - undefined
2021-09-25 14:01:17 info: -----------------------------------------------------------------------------------------------------------------------
2021-09-25 14:01:17 info: Main - Open Tray: C:\Users\xxx\AppData\Local\System Bridge\system-bridge-tray.exe
2021-09-25 14:01:22 info: MQTT - Setup
2021-09-25 14:01:23 info: -----------------------------------------------------------------------------------------------------------------------
2021-09-25 14:01:23 info: System Bridge - Tray 2.5.0: ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\tray\\src\\tray.js"] - undefined
2021-09-25 14:01:23 info: -----------------------------------------------------------------------------------------------------------------------
2021-09-25 14:01:23 info: Tray - Create Tray
2021-09-25 14:01:30 info: Tray - Started
2021-09-25 14:01:29 info: Main - API started on port 9170
2021-09-25 14:01:30 info: Main - Launch on startup: true - enabled - C:\Users\dlangerm\AppData\Local\System Bridge\system-bridge.exe
2021-09-25 14:01:30 info: Main - RTC broker created on path /rtc
2021-09-25 14:01:30 info: Events - Listening
2021-09-25 14:01:30 info: WebSocket - New client registered
2021-09-25 14:01:34 info: Tray - Open URL: https://github.com/timmo001/system-bridge/releases/latest
2021-09-25 14:01:44 warn: Error: dns service error: service not running
    at objectToTXTRecord (C:\snapshot\system-bridge\node_modules\mdns\lib\advertisement.js:90:12)
    at new Advertisement (C:\snapshot\system-bridge\node_modules\mdns\lib\advertisement.js:36:17)
    at Object.create (C:\snapshot\system-bridge\node_modules\mdns\lib\advertisement.js:64:10)
    at Server.<anonymous> (C:\snapshot\system-bridge\dist\main.js)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
2021-09-25 14:01:44 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"cpu":true}
The system cannot find the path specified.
2021-09-25 14:02:06 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"gpu":true}
The system cannot find the path specified.
2021-09-25 14:02:06 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"gpu":true}
The system cannot find the path specified.
2021-09-25 14:02:08 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"cpu":true}
The system cannot find the path specified.
2021-09-25 14:02:35 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"gpu":true}
The system cannot find the path specified.
2021-09-25 14:02:35 error: Command failed with exit code 1: C:\WINDOWS\system32\WindowsSensors\SystemBridgeWindowsSensors.exe {"gpu":true}
The system cannot find the path specified.
2021-09-25 14:02:35 info: Tray - Exit application
2021-09-25 14:02:35 info: Main - Server closing.
2021-09-25 14:03:52 info: System Bridge 2.7.0: C:\Users\xxx\AppData\Local\System Bridge\system-bridge.exe - C:\Users\xxx\AppData\Local\System Bridge - ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge.exe","C:\\snapshot\\system-bridge\\dist\\index.js"] -  -  -  -
2021-09-25 14:03:53 info: Process Manager - Starting tray - ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge-tray.exe"]
2021-09-25 14:03:52 info: Process Manager - Starting api - ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge.exe","dist/components/api/index.js"]
2021-09-25 14:03:54 info: Tray - 2.7.0: C:\Users\xxx\AppData\Local\System Bridge\system-bridge-tray.exe - C:\Users\xxx\AppData\Local\System Bridge - ["C:\\Users\\xxx\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\index.js"] -  -  - C:\Users\xxx\AppData\Local\System Bridge -  -
2021-09-25 14:03:54 info: Tray - Create Tray
2021-09-25 14:03:54 info: Tray - Started
2021-09-25 14:04:35 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:04:35 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:04:45 info: Process Manager - Starting tray - ["C:\\Users\\dlangerm\\AppData\\Local\\System Bridge\\system-bridge-tray.exe"]
2021-09-25 14:04:47 info: Tray - 2.7.0: C:\Users\xxx\AppData\Local\System Bridge\system-bridge-tray.exe - C:\Users\xx\AppData\Local\System Bridge - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\index.js"] -  -  - C:\Users\xxx\AppData\Local\System Bridge -  -
2021-09-25 14:04:47 info: Tray - Create Tray
2021-09-25 14:04:47 info: Tray - Started
2021-09-25 14:04:51 info: Tray - 2.7.0: C:\Users\x\AppData\Local\System Bridge\system-bridge-tray.exe - C:\Users\x\AppData\Local\System Bridge - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\index.js"] -  -  -  -  -
2021-09-25 14:04:51 info: Tray - Create Tray
2021-09-25 14:04:51 info: Tray - Started
2021-09-25 14:05:06 info: System Bridge 2.7.0: C:\Users\x\AppData\Local\System Bridge\system-bridge.exe - C:\Users\dlangerm\AppData\Local\System Bridge - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","C:\\snapshot\\system-bridge\\dist\\index.js"] -  -  -  -
2021-09-25 14:05:06 info: Process Manager - Starting api - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","dist/components/api/index.js"]
2021-09-25 14:05:06 info: Process Manager - Starting tray - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe"]
2021-09-25 14:05:07 info: Tray - 2.7.0: C:\Users\x\AppData\Local\System Bridge\system-bridge-tray.exe - C:\Users\x\AppData\Local\System Bridge - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\index.js"] -  -  - C:\Users\x\AppData\Local\System Bridge -  -
2021-09-25 14:05:07 info: Tray - Create Tray
2021-09-25 14:05:07 info: Tray - Started
2021-09-25 14:06:34 error: getVersion Error: ENOENT: no such file or directory, open 'C:\WINDOWS\system32\package.json'
2021-09-25 14:06:34 info: Process Manager - Starting api - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","dist/components/api/index.js"]
2021-09-25 14:06:34 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:06:34 info: System Bridge 0.0.0: C:\Users\x\AppData\Local\System Bridge\system-bridge.exe - C:\WINDOWS\system32 - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","C:\\snapshot\\system-bridge\\dist\\index.js"] -  -  -  -
2021-09-25 14:06:34 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:06:34 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:06:44 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:06:44 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:06:44 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:06:54 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:06:54 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:06:54 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:02 info: System Bridge 2.7.0: C:\Users\x\AppData\Local\System Bridge\system-bridge.exe - C:\Users\x\AppData\Local\System Bridge - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","C:\\snapshot\\system-bridge\\dist\\index.js"] -  -  -  -
2021-09-25 14:07:02 info: Process Manager - Starting api - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge.exe","dist/components/api/index.js"]
2021-09-25 14:07:02 info: Process Manager - Starting tray - ["C:\\Users\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe"]
2021-09-25 14:07:04 info: Tray - 2.7.0: C:\Users\x\AppData\Local\System Bridge\system-bridge-tray.exe - C:\Users\x\AppData\Local\System Bridge - ["C:\\x\\x\\AppData\\Local\\System Bridge\\system-bridge-tray.exe","C:\\snapshot\\system-bridge\\tray\\dist\\index.js"] -  -  - C:\Users\x\AppData\Local\System Bridge -  -
2021-09-25 14:07:04 info: Tray - Create Tray
2021-09-25 14:07:04 info: Tray - Started
2021-09-25 14:07:05 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:05 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:05 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:15 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:15 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:15 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:25 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:25 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:25 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:35 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:35 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:35 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:45 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:45 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:45 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:07:55 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:07:55 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:07:55 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:05 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:05 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:05 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:15 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:15 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:15 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:25 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:25 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:25 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:35 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:35 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:35 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:45 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:45 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:45 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:08:55 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:08:55 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:08:55 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:05 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:05 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:05 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:15 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:15 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:15 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:25 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:25 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:25 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:35 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:35 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:35 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:45 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:45 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:45 info: Process Manager - tray: Restarting in 10 seconds..
2021-09-25 14:09:55 info: Process Manager - Starting tray - ["C:\\WINDOWS\\system32\\cmd.exe","/q","/d","/s","/c","\"C:\\WINDOWS\\system32\\system-bridge-tray.exe\""]
2021-09-25 14:09:55 error: Process Manager - tray: Exited with code: 1
2021-09-25 14:09:55 info: Process Manager - tray: Restarting in 10 seconds..

### Additional information

Just wondering what is going on with these files messing around with the system32 folder? That seems like a very bad idea to me if it's intentional.

Additionally, the uninstaller didn't clear any of the files out or kill any of the running system bridge processes. I had to manually kill them and delete the files myself.

This appears to be on startup. If you kill any running processes and start the app manually, what do you get?


Remove NPM credentials from CI

My Puppet-foo is pretty weak, hope I've done this correctly.
All of the packages currently published under alphagov on the NPM registry either belong to the Service Manual, Patterns & Tools team (govuk-elements-sass, govuk_frontend_toolkit, govuk_template_ejs, govuk_template_jinja, govuk_template_mustache) or the Performance Platform (performanceplatform-client.js).
They have all been migrated to Travis and are now managed by their teams as part of the Services Programme, so there is no longer any reason for Jenkins to know the npm credentials.

We'll need to move these credentials from the encrypted Puppet to the password store also.

Updated to remove the .npmrc file.

We'll need to move these credentials from the encrypted Puppet to the password store also.

@surminus is this something you or someone on infra can take care of? I don't think I have access to the right things to do it.

I'll open a PR now.


Blazor Select binding doesn't bind initial value

Describe the bug
In s/s Blazor, a bound variable which is itself initialised to null does not reflect the initial state of a select box.
This bug is #10481 and possibly #9716 but these have been closed as dups or fixed, without it actually being fixed.
I'm going to describe it again: I'm seeing it in 3.1, so it's definitely still around.
To Reproduce

Create a default server-side Blazor project in VS (ASP.Net Core 3.1, VS 16.4)
On some page of the demo app (I use the counter page) add the following markup:

<select @bind="SelectionValue">
    <option>Option A</option>
    <option>Option B</option>
</select>
<button @onclick="TestClick">Test Click</button>

and the following code
    string SelectionValue {get;set;}

    void TestClick()
    {
        Console.WriteLine($"Selection: '{SelectionValue}'");
    }

Start the application in such a way as to be able to see the console
Navigate to the page but don't touch the select box (it will be showing "Option A")
Click on the "Test Click" button
Observe that the bound variable is null (i.e. you get Selection: '' printed to the console)
The only way to actually get "Option A" is to change the selection to Option B and then change it back.

I don't need a work around (it's easy), but this behaviour doesn't feel like the 'principle of least surprise' - it's counter-intuitive that you wouldn't get the displayed value of the selection box in the bound variable.
The fact that the bug's been closed as fixed/dup previously makes me think this wasn't the original intention that it should work like this.
Further technical details

ASP.NET Core version 3.1.100
The IDE  VS16.4 on Win10

This is behaving as expected. Using @bind on an input/select writes to your model only when the "change" event occurs. It doesn't write to the model just because a @bind exists.
In this scenario, you shouldn't have the initial value of SelectionValue be something that can't be represented by the dropdown, because then you will have this inconsistent state until the user changes the dropdown value. Instead, set the initial value of SelectionValue to correspond to one of the entries in the dropdown, and then everything will be consistent.
Hope that makes sense.

Thanks Steve.


Support user and password for http backend

This would be easier if we used Requests...

Yes it would.  :crying_cat_face:  Unfortunately it breaks the single file goal for the project.
Thanks for this though!

Thank you! Do you have another release coming up?


[MRG] Sparse L1 Descent for PyTorch

Implementing the Sparse L-1 Descent Algorithm in PyTorch.

Implementing the Sparse L-1 Descent Algorithm in PyTorch, ported from the TF version.
All tests ported from TF version.

Hello @iamgroot42, thanks a lot for yout contribution and sorry for the late reply. I read your codes and it looks great! Just a few things I want to discuss with you:

Regarding sanity check, do you mind including the assertion at line 51 of cleverhans/future/torch/attacks/sparse_l1_descent.py in the asserts list, and set default sanity_checks to False?
I would suggest add grad_sparsity = torch.tensor(grad_sparsity) after line 61, since grad_sparsity may not have attribute 'shape'
Comparing to the tf version (see line 495 of cleverhans/utils_tf.py), the random initialization at line 88 of cleverhans/future/torch/attacks/sparse_l1_descent.py seems to miss the scaling step.

Thank you again!


Endpoint Provider for botocore

This pull request implements a new interface for botocore EndpointProvider. As part of the Endpoints 2.0 initiative across the AWS SDK organization, it is responsible for reading a given service's endpoint ruleset and "resolving" aka determining an endpoint to return based on a set of parameters (i.e. region, bucket name etc.).
This is done through the main API resolve_endpoint. It expects input parameters in the form of a dictionary and uses that to create a Ruleset object. This object houses the specification for each expected input parameter and rules that the provider will evaluate given those parameters. These rule objects come in three distinct types: EndpointRule, TreeRule and ErrorRule.
These three types of rules share a lot of common behavior. They all have a conditions property, which are a list of functions (can be empty but always present). These functions will evaluate to a truthy or falsy value depending on the input parameters given to the provider. All conditions in a rule must be truthy for it to be considered a match. Endpoint and error rules are terminal in that if their conditions are met, the evaluation will immediately be complete and return/raise. A tree rule is different in that it also contains a set of rules which have their own conditions that need to evaluate to a truthy value for it to be a match. However, only one of a tree rule's subordinate rules need to be a match for it itself to still be considered one as well. A tree rule is never terminal and therefore will never be returned to a provider.  If a rule's conditions are not met, the evaluation immediately terminates and proceeds to the next rule if there are any. An error is raised if no match is found.
The test_endpoint_resolution test functions run tests for all of the automatically generated test cases. The rest of the tests are handwritten to fill out any code that wasn't covered by the generated ones.

Codecov Report

Merging #2737 (fc38e96) into develop (474e7a2) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop    #2737      +/-   ##
===========================================
- Coverage    95.25%   95.22%   -0.04%
===========================================
  Files           60       61       +1
  Lines        12278    12560     +282
===========================================
+ Hits         11696    11960     +264
- Misses         582      600      +18

Impacted Files
Coverage Δ

botocore/compat.py
92.66% <100.00%> (+0.04%)
:arrow_up:

botocore/httpsession.py
91.33% <0.00%> (-0.40%)
:arrow_down:

botocore/utils.py
93.34% <0.00%> (-0.27%)
:arrow_down:

botocore/endpoints_v2.py
95.18% <0.00%> (ø)

botocore/exceptions.py
99.55% <0.00%> (+0.01%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Why are the "standard library" functions methods of BaseRule? These function don't operate on the rule object itself or use any of its properties, as can be seen at a glance because there aren't any usages of self. So they could equally well be standalone functions or staticmethods. The one exception is aws_partition which needs the partition information, but that information too doesn't need to be part of each and every rule object because it's the same for all rules.
I am mainly asking this because I am wondering how many of the standard library calls may be repeated, possibly even within a single evaluation of a single ruleset. Would we see big performance improvements by adding caching to them? And if yes, then that only works if we have a single instance of each function instead of the many we currently have.

Why are the "standard library" functions methods of BaseRule? These function don't operate on the rule object itself or use any of its properties, as can be seen at a glance because there aren't any usages of self. So they could equally well be standalone functions or staticmethods. The one exception is aws_partition which needs the partition information, but that information too doesn't need to be part of each and every rule object because it's the same for all rules.
I am mainly asking this because I am wondering how many of the standard library calls may be repeated, possibly even within a single evaluation of a single ruleset. Would we see big performance improvements by adding caching to them? And if yes, then that only works if we have a single instance of each function instead of the many we currently have.

@jonemo I mentioned this a few weeks ago, but never got any feedback on it so left them as-is. I put them there mostly for convenience since they operate on a rule's conditions, but yes they definitely can exist independently. The only issue would be turning the function name of an input string into the function object itself, but we could just add a mapping for that. I'm not sure of the performance benefit though, but happy to do some manual testing on that.


Hyperparameter tuning for glomap

At the moment, it doesn't seem like we can tune the hyperparameters for the BA process like how COLMAP used to allow.
When reconstructing a scene like KITTI with primarily forward motion, the hyperparameters for BA must be changed. Currently we must analyze the source code and edit the hyperparameters, then re-build GLOMAP to use different hyperparameters. It would be nice if we can use a config file or a set hyperparameters via CLI.

Nevermind,
Found the config option by glomap mapper -h


Possible to have two nicknames for the same user.

Checklist

[X] I'm reporting a problem with Chatterino
[X] I've verified that I'm running the most recent nightly build or stable release
[X] I've looked for my problem on the wiki
[X] I've searched the issues and pull requests for similar looking reports

Describe your issue
Similar to https://github.com/Chatterino/chatterino2/issues/2497 you can have two nicknames for the same person.
I assume this same fix that is being work on for https://github.com/Chatterino/chatterino2/issues/2497 could be applied to nicknames as well. ( https://github.com/Chatterino/chatterino2/pull/4322 )
Screenshots

OS and Chatterino Version
Chatterino Nightly 7.4.1 (commit f6c90d99) built on 2023-02-20 with Qt 5.15.2, Windows SDK, MSVC 193431942, Crashpad
Running on Windows 10 Version 2009, kernel: 10.0.22621

This is a lot harder than the command issue as we have support for regex nicknames. Checking equivalence of regular expressions is a hard task (we'd need to implement basically a regex library) and I'd like for us to avoid that at all cost.
The only way I can see this working is checking non-regex nicknames only and just ignoring the rest.


[main] Update actions

Cron -knative-prow-robot
/cc knative/operations-wg-leads
/assign knative/operations-wg-leads
Produced by: knative-sandbox/knobots/actions/update-actions
Details:

/lgtm
/appove

/lgtm
/approve

/lgtm
/approve

/test pull-knative-operator-serving-upgrade-tests


Support Amazon Linux 2023

Marking as draft while I validate with some modules that need to support Amazon 2023.

This won't work for all cases until https://github.com/puppetlabs/beaker-puppet/commit/6063d22b6c4449df795731f5853c3c75241240c4 is tagged.

This is working.  https://github.com/OSC/puppet-module-openondemand/actions/runs/7656172797/job/20863882254
The job itself is failing but that's because the RPM repos I'm testing against don't yet exist but the beaker parts to bootstrap the tests are working for Amazon 2023.


x/build: use rr-project.org for running Go builds/tests on Linux?

We have a lot of intermittent failures on the builders, which are hard to reproduce. Evidently the Firefox project had similar struggles, so they built https://rr-project.org/, which allows recording and replaying program executions on linux/x86. Perhaps we should look into using it for our builders too?
Quickly skimming build.golang.org, it's not clear to me if there are enough flaky failures on linux/x86 to warrant the ~20% slowdown (according to their website). It seems like most flakiness is on non-linux or non-x86, unfortunately, and rr doesn't currently support any other platforms.
Maybe the regabi builder could benefit from it at least. It seems to have had some flakiness a couple weeks ago.
/cc @aclements @bcmills

CC @golang/release.

I'm not sure wether this is from any help. But I've stumbled upon this issue and the description from rr-project reminded me a lot about a technique I've heard before. @mdempsky stated that most flakey tests are on non Linux. Maybe it's from interest that MS has a similar technique called Time travel debugging. I've no experience with this but maybe it's worth evaluating if TTD can help with some none Linux flakiness.


[Bug]: Email verified attribute is not set as true for External user authenticated by LDAP Identity provider

Preflight Checklist

[X] I could not find a solution in the documentation, the existing issues or discussions
[X] I have joined the ZITADEL chat

Environment
Self-hosted
Version
v2.52.0
Database
PostgreSQL
Database Version
Postgres 13
Describe the problem caused by this bug
Email verified attribute is not set as true for External user authenticated by LDAP Identity provider.
I have set action to  setEmailVerified(true)  which triggers Post-Authentication/Pre-Creation in External Authentication Flow but VerifiedEmail property is not set as true for the LDAP Authenticated user.    Instead, I am getting error in log that "message":"action run failed: function not found","logLevel":"error".
To reproduce

Deploy Self hosted Zitadel
Add LDAP Identity provider
Add Action to setEmailVerified(true) for External Authentication flow in triggers Post-Authentication/Pre-Creation
Login to Zitadel UI as External user using LDAP Identity provider

Zitadel UI lands on "E-Mail Verification"  and "code" required to input to login.
Screenshots

Expected behavior
User should be able to Navigate to User Page without the need to input the "E-Mail Verification" code after authentication.
Operating System
Kubernetes cluster
OS - Linux  5.15.0-101-generic  #111-Ubuntu SMP  x86_64  GNU/Linux
Browser - Chrome - Version 124.0.6367.208
Relevant Configuration
No response
Additional Context
No response

Hi, I have the same problem. I would therefore be interested in finding a solution. I followed the guide at the following link:https://github.com/zitadel/zitadel/discussions/4145
but unfortunately it didn't work
Thanks for your help
Francesco

Hey @singhajitk and @francescoVenuto
Can you quickly check / confirm that the name of the actions is the same as the function name side the scipt?
This is currently required for actions to be executed.

Hi @livio-a, I confirm I’ve been able to automatically set emailVerified to true using an action which name is the same as the function name.

Thanks @livio-a ,  Got it working with this change.


Add pylint and isort to requirements.txt

Since these linters are supported out of the box but not getting installed by default with installation of pylama, I think these packages worth to be listed in requirements.txt for pylama.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

base linters (pydocstyle, pycodestyle, mccabe) are installed with pylama, other could be installed as optional dependencies for example pip install pylama[pylint, eradicate


[DEV-3153] enable recipient address edit

DEV-3153 enable recipient address edit

[ ] Await merge of Packages PR and update


Date/time-based filename (when saving) for figures on Plots pane

Generate a date/time-based filename (so we don't need to rename each picture we want to save if saving multiple figures). Look at the picture below to see how Snip & Sketch generates a date/time-based file name:

Thoughts on this @jnsebgosselin ?

Thoughts on this @jnsebgosselin ?

Its a good idea to add the date.
From the screenshot above, it seems that the first 6 characters of a uuid is also added at the end to ensure uniqueness of the filename. Do you think we should do that also?

The last 6 characters represent the time in the format "hhmmss", which is great when one is saving a series of charts on the same day.

The last 6 characters represent the time in the format "hhmmss", which is great when one is saving a series of charts on the same day.

Ah I see! ahah Thanks for the clarification.
I can implement that if the feature is accepted.

I can implement that if the feature is accepted.

I like it :-) @ccordoba12 ?

Sure. it'd be nice to have in Spyder 5.

Does future means "Do it against master branch" or it can be done against the 4.x branch?

Hmmm well, if you want to work on it soon, then 4.x is fine I guess.

Side note:
We are now using zenhub to organize the workflow and in that workflow milestones are sprints, releases (a zenhub thing) groups what we used as milestones and we added this as labels as well for non zenhub users.

Hmmm well, if you want to work on it soon, then 4.x is fine I guess.

Agreed.


Update request: zig 0.10.1 → 0.11.0

Package name: zig
Latest released version: 0.11.0
Current version on the unstable channel: 0.10.1
Current version on the stable/release channel: 0.10.1
[ x] Checked the nixpkgs pull requests

Notify maintainers
@aiotter
@AndersonTorres
@andrewrk

Note for maintainers: Please tag this issue in your PR.

I'm not at home now. I'll look at it later.
Reading the release notes, I see that support for macos catalina 10.15 was dropped. It can be a bit problematic for us I suppose.


Feature Request: On library Songs page, jump to song when I start typing

OS: Windows 7
Issue Descriptions: In the past, other desktop music players have supported the ability to jump to items in the library by typing the start of the name. Ideally, it'd jump to songs when starting to type the name of the song. Stretch goal would be to jump to the item currently sorted (i.e. if sorting by artist, jump to artist)
Steps to Reproduce:

Go to Music Library
Go to Songs
Start typing the name of a song
Jump to closest song matched, keystroke by keystroke (i.e. typing "Bes" highlights "Backyard", then "Believer" then "Best Friends"

Note: I suggested this feature because multiple friends asked if Google Music could do this because they want to switch from other platforms, but wouldn't until there was a player that could handle it. Just an idea!

Unfortunately the best you're probably going to find is the search bar, which has a shortcut built in. You can see the rest of the built in GPM shortcuts here:

I know that's the best that exists now, but I imagine it'd be possible to add a key listener to that page and scroll to the item in question.

Scratch that, I just looked at how it works, and yeah, I see why that might not be possible.

Due to Google's helpful but annoying "partial rendering" technique.  At any one time only 32 songs are rendered on screen.  This means we have no idea what other songs are in the current view which makes this kind of search impossible.

Understandable, thanks!


simplify

Codecov Report
Base: 36.58% // Head: 77.30% // Increases project coverage by +40.72% :tada:

Coverage data is based on head (ed4e8de) compared to base (6d70a0e).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #1540       +/-   ##
===========================================
+ Coverage   36.58%   77.30%   +40.72%
===========================================
  Files         107       33       -74
  Lines       17962     3428    -14534
===========================================
- Hits         6571     2650     -3921
+ Misses      11053      778    -10275
+ Partials      338        0      -338

Flag
Coverage Δ

unittests
77.30% <ø> (+40.72%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

hrp/cmd/convert.go

hrp/cmd/dial.go

hrp/internal/builtin/utils.go

hrp/pkg/convert/from_curl.go

hrp/pkg/convert/from_gotest.go

hrp/pkg/convert/from_json.go

hrp/pkg/convert/from_postman.go

hrp/pkg/convert/from_yaml.go

hrp/pkg/uixt/android_driver.go

hrp/pkg/uixt/ext.go

... and 57 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add ability to force command string

As command strings are based on the shell function name, it's not possible to use dashes.
For example, using the following script:
#!/bin/sh

# @describe This is a test

# @cmd      Start the service
start_service() {
    echo do stuff
}

argc $0 "$@"

The help function is:
This is a test

USAGE: argc-test <COMMAND>

COMMANDS:
  start_service  Start the service

I would like to be able to use dashes instead of underscores as word-delimiters in command names:
This is a test

USAGE: argc-test <COMMAND>

COMMANDS:
  start-service  Start the service

I see two potential solutions:

replace all underscores (_) with dashes (-) in command names (maybe not by default, for example by adding a flag for that)
or (probably better) add a @name flag to be able to force the command name:

# @cmd      Start the service
# @name     start-service
start_service() {
    echo do stuff
}

I know it's possible to use @alias to make it work, but I would prefer to hide the service_start function name in this output:
This is a test

USAGE: argc-test <COMMAND>

COMMANDS:
  start_service  Start service [aliases: start-service]

Let me know what you think.
And thanks for this great tool ! Can't wait to see the upcoming v1.0.0 released.
EDIT: looks like v1.0.0 was released 30 minutes ago :smile:

Why not update you cmd function name?
#!/usr/bin/env bash

# @describe This is a test

# @cmd
start-service() {
    :;
}

eval "$(argc --argc-eval "$0" "$@")"

This is a test

USAGE: test <COMMAND>

COMMANDS:
  start-service  Start the service

Function names or command names are actually more free than you think
# @cmd
test:foo() {
  :;
}

# @cmd
test@foo() {
  :;
}

# @cmd
test-foo() {
  :;
}

# @cmd
test_foo() {
  :;
}

# @cmd
test.foo() {
  :;
}

COMMANDS:
  test:foo
  test@foo
  test-foo
  test_foo
  test.foo

I even choose ::  as marker for nested subcommands.
#!/usr/bin/env bash

# @cmd
cmd() {
    :;
}

# @cmd
cmd::foo() {
    :;
}

# @cmd
cmd::bar() {
    :;
}

eval "$(argc --argc-eval "$0" "$@")"

Run ./test.sh -h
USAGE: test <COMMAND>

COMMANDS:
  cmd

Run ./test.sh cmd -h
USAGE: test cmd <COMMAND>

COMMANDS:
  foo
  bar

Oh yeah you're right.
In fact I'm always writing POSIX-compatible scripts and using #!/bin/sh as shebang, so using test-func() or test::func() results in:
#~ /tmp/argc-test
/tmp/argc-test: 7: Syntax error: Bad function name

I might be too old fashion haha, but I can change my shebangs to /bin/bash. In fact the main reason why I'm always using /bin/sh is because I used to write scripts for embedded devices where I only had low footprint POSIX shell available.
Thanks for the hint.


Only allow one thread for unsafe stdlib

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

@mbed-bot: TEST
HOST_OSES=windows
BUILD_TOOLCHAINS=IAR,ARM
TARGETS=NUCLEO_F411RE,LPC1768,K64F, KL25Z,NUCLEO_L053R8,MAX32600MBED

Hello @c1728p9.
Something has been decided about the limit between the small and standard build ? Should we change the default_toolchain from uARM to ARM on standard targets too ?

Hi @svastm, it doesn't hurt to change the default to ARM.  The default toolchain field might be removed in the future though since this is duplicate information.

I don't understand the choosen "default_build" options:
e.g. an EFM32PG with 256KB/32KB (flash/ram) gets a "default_build": "standard", a lot of other MCUs several with much more flash/ram (STM32F407VG with 1024kb/192kb, ...) gets a "default_build": "small". A DISCO_F401VC gets a "default_build": "standard" and a NUCLEO_F401RE - same mcu but with twice the flash size - gets only a "default_build": "small". Why?
With this option RTOS is not available (for GCC) maybe also other things did not work.


pull request for e2e test at 01/04/2017 08:15:07

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


chore: release 1.4.4

:robot: I have created a release *beep* *boop*
1.4.4 (2021-03-19)
Bug Fixes

helm-ldap-backup: Update chart version (0472b4a)
helm: comment out example storageCLass.parameters (70a289e)
helm: update chart version (62b00dc)
images: Update tags (f4ad8eb)
jackrabbit: Set admin password for jackrabbit (6d8b0e0), closes #299
ldap: Fix serf peers parse (a0e0c1c), closes #299

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/GluuFederation/cloud-native-edition/releases/tag/v1.4.4 :sunflower:


make delete account actually delete account

as of #401

deleting account only logs them out and deletes all their notes.
we want to actually delete the account by making a supabase-functions in that repo to delete the account.
in the function, it'll delete the account from the jwt token sent from the client.

#409


Update Library to Version 6.3 of GlobalSearch - Add new API Endpoints

New Features have been added to GlobalSearch in version 6.3
Make another scraping pass at the API and add new methods/modify existing ones

Here are all the tracked changes that need to be applied

[ ] Add extendedConfig Type

[ ] Modify Field data type to include extendedConfig

[ ] Modify AdminField data type to include extendedConfig

[ ] Add Advanced Links Endpoint

[ ] Add Advanced Links Object

[ ] Modify GetEmailOptions and UpdateEmailOptions to include new GlobalEmail Options

Note: These do not need to be included on the Field data type

[ ] Add regSample Type

[ ] Modify AdminField data type to include regSample
[ ] Add regexMessage property to AdminField data type

Added 6.3 Functionality


SendToReplicas Option Issue in Cluster Mode

In cluster mode, regardless of SendToReplica and ReplicaOnly option value settings
When calling keys(), the slave node does not return data.
However, when calling scan(), the data is returned from the slave node as shown in the results below.
 My KeysTest showed the same result as the ScanTest. Could it be possibly caused by replication lag on your server?

Staled


Awesome Challenges

Post links to your favorite challenges and let us know what makes it so great.  does it have some nice solutions to study?  are the tests interesting? is the challenge just plain fun?

Pair Management - easy, but there are different solutions to it, some of them are unexpected. Good to start from!


Fixed #1016 - Static Analyze: Resource management issues

I/O resource opened but not safely closed

I need to rework for this. Close this PR.


Fix tests

okay so the tests all pass here. which I assume means i can merge into main and then the tests will also pass in version packages...


Fix CUDA Dockerfile

Include Git in the CUDA Docker image
What does this add?
This PR makes the Dockerfile install git in the final user image, as it's currently not present and is needed for the CI.
Why is it needed?
The CI operates on checking out the current commit and running the tests, as a result git was needed.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Feature add arm support and use chromium

This allows to build and run gotenberg on a raspbarry 4b.
We are using chromium as there is no arm build of chrome.
The tiny executable needs to be changed to the version that is
compatible with the correct architecture.
Also preparing for multi-architecture building of the docker images in
the publish script.
We removed the crosscompiling enviornment variables of go, as they are
preventing a compiling for arm and they will be set automatically.
fixes https://github.com/thecodingmachine/gotenberg/issues/213
Test plan
Please check if this changes are not braking anything. I have tested the code only on an Raspbarry Pi v4

This PR is a godsend.
Im using Gotenberg alongside https://github.com/jonaswinkler/paperless-ng and Apache Tika.
The latte two running easily on my Pi4B. Just Gotenberg needs to run on x86 host atm.
Looking forward to a Pi4 compatible arm release :)

Hello @toxix :) Thanks for your PR!
Alas, I don't plan to support arm64 architecture for Gotenberg 6.
That being said, I'm currently working on the next version of Gotenberg, and it will bring, among other things, arm64 support :)

The change works on my Apple M1 👍🏻


[ZEPPELIN-1061] Select default interpreter while creating note

What is this PR for?
This handles setting the default interpreter on creating a note through the zeppelin UI.
What type of PR is it?
Feature
Todos
NA
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1061
How should this be tested?

Go to the 'Create Note' dialog and provide the name with the default interpreter selected.After clicking on the Create button, the selected interpreter should be shown properly in the interpreter binding section.
If there is no interpreter selected, then the system default will be used.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

WIP?

The changes are done.

@kavinkumarks I'm looking into this.

@jongyoul any udpates on this?

@kavinkumarks Backend codes looks good to me. @corneadoug Could you please look into it?

@kavinkumarks Thanks for your contribution. Tested and it works well!
Can we reset the selected interpreter in the note creation page? Currently the dialog keeps the previously selected one.

@jongyoul & @AhyoungRyu thanks for reviewing it. @AhyoungRyu I will look into the issue.
@corneadoug any updates on this?
Thanks,
Kavin
MailTo<EMAIL_ADDRESS>

@kavinkumarks Currently there are some conflicts in here. Can you resolve them? :)

@AhyoungRyu I have resolved the conflicts and also fixed the issue "Can we reset the selected interpreter in the note creation page? Currently the dialog keeps the previously selected one."
Waiting for the travis build to be success.
Thanks,
Kavin
MailTo<EMAIL_ADDRESS>

The travis build is green after fixing the test case failure.
Thanks,
Kavin
MailTo<EMAIL_ADDRESS>

@kavinkumarks Tested and it works well as expected! Thanks again :)
Merging this into master if there are no more discussions.


Force history update on snapshot.

I'm currently working on a project where multiple developers are adding migrations into source control at the same time using EF6 CF.  Both of the developers will make a change to their migrations, but when the second one merges in the code from the first, their migrations don't contain the correct snapshot, causing the Update-Database to fail due to pending migrations (although they are covered by the merged-in migrations).
Examples:
http://stackoverflow.com/questions/19136066/why-does-add-migration-sometimes-create-duplicate-migrations
http://stackoverflow.com/questions/17921886/update-database-fails-due-to-pending-changes-but-add-migration-creates-a-duplic
I wasn't sure if there was currently a way to fix this issue other than to create an empty migration to get a valid snapshot of the state of the system.
If there is not, I propose that a command is added that allows a developer to force-update the history tracking on the latest migration so that the Update-Database command will be better able to determine if there are really pending changes or not.
Note that this would also be a problem if you were to copy/paste a migration into your migrations folder -- so it's not necessary limited to developer teams.

This is one of the pitfalls of using Code First Migrations in team environments. The Code First Migrations in Team Environments article will walk you through the steps needed to resolve issues like this. Have at least one person on your team get familiar with it. :wink:
Don't hesitate to ask if you have any specific questions on how to apply the article to your scenario.
Also, we feel your pain. This is one area that we've worked hard to improve in EF Core. We've introduced a common model snapshot file that will make merging migrations a lot easier.

Thanks Brice!
A very informative and useful answer.  I'll be sure to share the article with my team. :)


Service Message Popups

If you don't have a Service that is always the last to lounch Service messages will still popup over ly.
I will look for a solution myself but as I don't have any experience in c it could take time.

Actually this can get really tricky, we had a hard time with Oxodao trying to fix this, and the only solution we could find is described in the readme. Another working option could be to edit you kernel parameters.
Can you tell me if the first method works for you?

The method described in the readme doesn't work for me or I would have to specify each of these services.
As for the kernel parameters, I can suppress them withe the quiet option, but this also suppresses errors and I haven't found one that lets me redirect messages to a specific tty or file.
Is it possible to catch all input to the tty and redirect it to another?
I will see if I can find stuff in the agetty source as it and other programs like it have to have methods to only let errors show errors.

@RobertZenz this easy to do, but not quite a clean approach.
@L0ric0 you have to specify the last service launching on your computer that gives error messages.

No, it isn't, but it would be a quick and easy workaround. Especially when made optional.

@cylgom Yes I know but systemd works asynchronous where it can and so I have multiple services that can be the last to launch. But I have found that you can define a target in the After section of a service. So adding:
After=multi-user.target
to the service file will launch ly only when all services of multi-user.target are started circumventing the issue of knowing the last service to start.


Travis changes

merged ci/release-docker-image.sh into the .travis.yml

Sequential Stages each with multiple parallel jobs are possible with only the travis configs, too. Only, the default stages/jobs now are not explicitly named, see: https://travis-ci.com/SAP/i40-aas/builds/136344349


Bus free times are ignored

If peripheral modules request access to the bus at the same time, bus free times are ignored. After a peripheral releases their respective active line, a free time as per the I2C specification should be inserted.

Quick note: bus free time is identical to tLOW (the low time of SCL), so the easiest solution would be to wait for one half a clock cycle after the stop condition before returning to idle.


Update fabric.mod.json for newer versions

The mod works fine on 1.21 Minecraft, you just don't have it indicated as such in the fabric mod json.
Change
"minecraft": "1.20.x",
to
"minecraft": ">=1.20",

I haven't tried 1.21 yet, which is why I haven't changed it so far. I'm just believing you and changing it (in a moment).


Power to Torque for T-MATS cantera shaft block

"The unit of the power of the model of JT9D_ss_Cantera is [hp],and when entering the power of the compressor and turbine to the shafts component,there is a data should be multiplied to convert unit,and the data just equal to the unit conversion from [BTU/s] to [hp],but it has already been converted in the component of compressor and turbine."
This should be looked into.  It looks like power is being exported from the compressor and turbine blocks, however T-MATS cantera is using the default T-MATS shaft block that uses and input of torque.  Look into creating a special shaft for T-MATS cantera.

The new import for the shaft should do the trick for this.
932c72938792938081780c41935a9dd042542870


Rqlite bin

attempts with rqlite binary package, embedded.

Not going to work with rqlite. Have POC with sqlite.


Sort listing by container name.  LGTM otherwise, thanks!

signed off...

All the commits in the branch must be signed-off. Looks like your branch history contains multiple merges and multiple commits. I'd recommend you rebase it down to a single commit.

Hi,
Thanks for that.
Travis is still complaining about some formatting issue. Can you run "go fmt ./..."
(Or look for the diff on the Travis failure page)
Thanks!

Properly formatted... sorry for the amount of wasted energy.


Possible fix and unit test. Fixes #961

One possible fix for this issue.  This just ignores the failed file.  AFAIK we aren't actually doing anything with this failed file so it could just be removed.  But maybe that be done for 2.0

Should we at least log a debug message?

It already logs the original error.  I am not sure how much value another log message adds...


Updated OpenSearch appVersion to 1.2.0

Signed-off-by: Derek Diaz<EMAIL_ADDRESS>Description
Updated appVersion to 1.2.0
Issues Resolved
#151
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Hi @derek-diaz we need to pause this due to #168.
Thanks for the contribution we probably need to wait for new OpenSearch patch to be out before we can upgrade now.
Thanks.

Hi @derek-diaz can you change the version to have
OpenSearch 1.2.1 out, Dashboards stays 1.2.0.

Thanks.


Tensorflow after 1.15 - No need to install tensorflow-gpu package

This is basically an info question that was asked by mon in SO and we answered as far as we know, here.
In the official tf installation guide in documentation, it says,

For releases 1.15 and older, CPU and GPU packages are separate:

pip install tensorflow==1.15      # CPU
pip install tensorflow-gpu==1.15  # GPU
And for later version, e.g tf : > = 2.0, we simply need to do
pip install tensorflow     # CPU or GPU

But still, we have
!pip install tensorflow-gpu
....
Installing collected packages: tensorflow-gpu
Successfully installed tensorflow-gpu-2.4.1

# -------------------------------------------------------------

!pip install tensorflow-cpu
....
Installing collected packages: tensorflow-cpu
Successfully installed tensorflow-cpu-2.4.1

why tf-gpu/cpu for 2.4.1 still exits? what is the main reason here? is there any difference for tf > = 2 installing them separately or the standalone way?

@innat,

As you've already mentioned, starting with TF v2.0, the TensorFlow packages by default support both CPU and GPU.

why tf-gpu/cpu for 2.4.1 still exits? what is the main reason here?

If you explicitly require support for either CPU or GPU only, you can opt for the tf-cpu/gpu package.

is there any difference for tf > = 2 installing them separately or the standalone way?

Apart from the above mentioned point and the package size itself there isn't much difference.
Hope this answers your query. Thanks!

tensorflow and tensorflow-gpu after 2.1 are exactly the same package, just with a different name.


Cleanup configure apiserver

Includes https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/43 due to avoid merge conflict.
Addresses https://bugs.launchpad.net/charm-kubernetes-master/+bug/1841101

 kubernetes-master clean up of reconfigure_apiserver()

Tested with CK 1.15 + custom k8s-master


Added possibility to set first pick in lobby

Nice. We must be getting close to a version increment where these great changes will hit the master.


add APPLICATION_ROOT support

so that it can run behind a proxy with a root_path other than '/'

Nice addition! Did you check if dev mode is affect? (dev mode described here: https://github.com/ricklamers/gpt-code-ui/blob/main/.github/CONTRIBUTING.md#getting-started)

Nice addition! Did you check if dev mode is affect? (dev mode described here: https://github.com/ricklamers/gpt-code-ui/blob/main/.github/CONTRIBUTING.md#getting-started)

hmm, ihavnt notice that, i'll check that later


Add setField hook from feathers-authentication-hooks

This moves the new setField from feathers-authentication-hooks.
Closes #433

:shipit:

Continuing at #664 (typescript version of this PR).