grunt test fails with node 8.1, phantoms 2.1

Tests fail to run.
Symptom:
$ grunt test --stack
...
Running "qunit:normal" (qunit) task
Testing http://localhost:8000/test/test.html
PhantomJS threw an error:ERROR
>> 0 [ '' ]
Warning: PhantomJS exited unexpectedly with exit code null. Use --force to continue.

When run with --force:
$ grunt test --stack --force
...
PhantomJS threw an error:ERROR
>> 0 [ '' ]
Warning: PhantomJS exited unexpectedly with exit code null. Used --force, continuing.
Warning: 0/0 assertions ran (0ms) Used --force, continuing.
(node:37480) [DEP0013] DeprecationWarning: Calling an asynchronous function without callback is deprecated.

Environment (Mac OSx 10.12.6):
$ node --version
v8.1.0
$ phantomjs --version
2.1.1
$ grunt --version
grunt-cli v1.2.0
grunt v0.4.5

Is there a specific phantomjs or node version ceiling?  Are there some libraries we could update?
Thanks,
Thatcher

Interesting! I'm getting the same result. Travis is still running it successfully (see https://travis-ci.org/openseadragon/openseadragon). Anyway, we should definitely fix it! I'm guessing maybe just upgrading qunitjs and/or grunt-qunit-istanbul. Would you be up for giving that a try?
FWIW, here is my environment (I'm also on Mac 10.12.6):
$ node --version
v6.9.5
$ phantomjs --version
-bash: phantomjs: command not found
$ grunt --version
grunt-cli v0.1.6
grunt v0.4.5

Looks like I don't have phantom installed globally; it must just be using the local install from the package.json... I assume it's installed by grunt-qunit-instanbul or something.

Yes I will work on this one today, thanks Ian.

Wonderful, thank you!

I started looking at this because I want to be able to run some tests. The good news is that it's mostly QUnit API/syntax changes that need to take place in order to get us going. The bad news is thatgrunt-qunit-istanbul no longer works with the latest QUnit, so we'll have to find some other way of doing code coverage.

"we renamed everything" : http://qunitjs.com/upgrade-guide-2.x/

@nein09 Thank you for digging into this! Any thoughts on what we could use for code coverage? It's not a big deal if we're without for a while, but it would be good to get it working again at some point. Fixing the tests themselves is definitely higher priority.

@iangilman I'll have to experiment, but I saw a little demo of qunit outputting something that I think coveralls could use. If it's not a ton of work, I'll put that in this fix.

Sounds good! Thank you for looking into it. I know very little about this stuff...


ProgressBar update

Updated the existing progress bar sample to use the new WinUI version of the control.
Description

Moved the app to latest WinUI preview version
Changed the namespace used in progress bar sample
Marked progress bar page as "preview" until we switch to a release version of WinUI

Motivation and Context

New-to-WinUI control
How Has This Been Tested?

manually verified
Screenshots (if appropriate):

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

When this PR gets through, should we (or I) take a quick look if we can clean up some workarounds for issues that are now fixed with the prerelease of WinUI 2.3? 😅

Note that the ProgressBar test is now failing (in an expected way, I think). Will follow up with @hassanuz

@chingucoding

When this PR gets through, should we (or I) take a quick look if we can clean up some workarounds for issues that are now fixed with the prerelease of WinUI 2.3? 😅

You know we won't say "no" to help 😍

It seems like we won't be able to get rid of some bug fixes. Even the fix for untargeted TeachingTips does not work in XCG for some reason. Maybe we need to look into this or is this related to the fact that the TeachingTip is in the header?
Only thing I would do is replace the number input TextBoxes on the "ImplicitTransition" and "GridView" page with the new NumberInput control, if thats fine.

You could also remove the "Hold and Change Value" button

You could also remove the "Hold and Change Value" button

What exactly do you mean? 😅

You could also remove the "Hold and Change Value" button

What exactly do you mean? 😅

If you are using a NumberBox, that has up and down Spin Buttons that allow changing the value

If you are using a NumberBox, that has up and down Spin Buttons that allow changing the value

Oh I see. My aim was to clean up some code in XCG, but since we use the same button in the ProgressBar sample, I will change that too.


Read BTB and BHT even if processing an IF2 override

Summary
Previously, the predictors would read garbage when the PC was the target of an override. This garbage would be used for the following prediction and for the writeback to the BHT, trashing the row.
Issue Fixed
No open issue. Mentioned to Dan via Slack.
Area
bp_fe_pc_gen
Reasoning (outdated, confusing, verbose, etc.)
The BHT/BTB read happens in IF1. The read prediction data is for a prediction of the next instruction, i.e. the next one to enter IF1. There's no reason we would want to avoid the prediction due to the preceding instruction being originally mispredicted.
I suspect the original intent may have been to save power and skip the read for the invalidated IF1 PC on an override, but if I'm understanding correctly what it's actually doing is using that invalidated PC's entry for the real PC that follows since the BHT uses a bypass reg to latch the old value when it's not reading.
Additional Changes Required (if any)
None
Analysis
CoreMark before change:
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 100
Total time (secs): 100.000000
Iterations/Sec   : 0.050000
Iterations       : 5
Compiler version : GCC9.2.0
Compiler flags   :
Memory location  : STATIC
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0xf24c
Correct operation validated. See README.md for run and reporting rules.
CoreMark 1.0 : 0.050000 / GCC9.2.0  / STATIC
[CORE0 FSH] PASS
All cores finished! Terminating...
$finish called from file "/mnt/users/ssd1/homes/kaelinl/repos/black-parrot/bp_top/test/common/bp_nonsynth_host.sv", line 181.
[CORE0 STATS]
        clk   :              1880473
        instr :              1531935
        mIPC  :                  814

CoreMark after change:
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 100
Total time (secs): 100.000000
Iterations/Sec   : 0.050000
Iterations       : 5
Compiler version : GCC9.2.0
Compiler flags   :
Memory location  : STATIC
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0xf24c
Correct operation validated. See README.md for run and reporting rules.
CoreMark 1.0 : 0.050000 / GCC9.2.0  / STATIC
[CORE0 FSH] PASS
All cores finished! Terminating...
$finish called from file "/mnt/users/ssd1/homes/kaelinl/repos/black-parrot/bp_top/test/common/bp_nonsynth_host.sv", line 181.
[CORE0 STATS]
        clk   :              1874424
        instr :              1531935
        mIPC  :                  817

This is an 0.3% improvement in cycle count.
I don't have other benchmarks available to me to run, but I'd be happy to do so if desired. CoreMark is likely a rather poor indicator.
Verification
Cursory examination of the wave indicates that the reads are being forwarded back in the attaboy/redirect as expected. I haven't done any rigorous verification.
Risk of the change is low, as reading when not needed is unlikely to cause a regression. The bypass register latching the old read does introduce some possibility of intentional behavior lost here, but it certainly seems wrong to me.
These changes do not seem to affect the bp-tests loop_test program at all (with all the parts enabled at once, i.e. with some branch predictor contamination).
A short test program with dummy jumps that I'm using for other purposes performs worse after this change. I haven't had time to investigate yet. It's only a 10-cycle difference on a 50-cycle function call so it's likely to be an aberration due to poorly trained predictors.
Additional Context
None

Thanks Kaelin! We've done much more for a 0.3% on coremark...


SQL Server on Mac M1 fails to start

Fails with the message:  ERR  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
If it helps, the following command works docker run -e "ACCEPT_EULA=1" -e "MSSQL_SA_PASSWORD=MyPass@word" -e "MSSQL_PID=Developer" -e "MSSQL_USER=SA" -p 1433:1433 -d --name=sql mcr.microsoft.com/azure-sql-edge
Taken from https://medium.com/geekculture/docker-express-running-a-local-sql-server-on-your-m1-mac-8bbc22c49dc9

@morrislaptop I've tested out that image and was able to get it running on my M1 and on my Linux laptop (same image). But is this new image fully compatible with the previous one?
For reference, I've created a branch called fix/tm-mssql-arm and here's the diff.

Taken from https://medium.com/geekculture/docker-express-running-a-local-sql-server-on-your-m1-mac-8bbc22c49dc9
Damn tried to follow the steps from this post and I am unable to get the database to run

Damn tried to follow the steps from this post and I am unable to get the database to run

idk why wasn't able to get it to pull image correclty but doing this with docker itself is working:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=useA$strongPas1337' -p 1433:1433 -v mssql_data:/var/opt/mssql --name sql1 -d mcr.microsoft.com/azure-sql-edge:latest


Addons terrain for phones does not work

Tested versions
4.2.1
System information
Phone / Android
Issue description
There is a problem with the terrain on the phone. When I download an add-on to Godot, the terrain does not work. There is a problem where it is not possible, even when I touch, the ground becomes terrain, or even I want to add only terrain and terrain shader
Steps to reproduce

Minimal reproduction project (MRP)

This should be reported to the plugin developer, not here 🙁

Assuming this is the Terrain3D add-on, this is already being tracked in https://github.com/TokisanGames/Terrain3D/issues/137. Also, note that most add-ons aren't developed with touch input in mind, so you need a hardware mouse and keyboard to be able to use them efficiently.

ok thanks


Remove src/parser.c and  src/tree_sitter/ from .gitignore

Emacs function treesit-install-language-grammar requires src/parser.c to exist.
I suggest to remove these lines from .gitignore:
src/parser.c
src/tree_sitter/

Example repos where src/parser.c and src/tree_sitter/ are present
Official grammars

https://github.com/tree-sitter/tree-sitter-c/tree/master/src
https://github.com/tree-sitter/tree-sitter-php/tree/master/php/src
https://github.com/tree-sitter/tree-sitter-go/tree/master/src
https://github.com/tree-sitter/tree-sitter-python/tree/master/src

Third-party grammars

https://github.com/tree-sitter-grammars/tree-sitter-markdown/tree/split_parser/tree-sitter-markdown/src
https://github.com/Wilfred/tree-sitter-elisp/tree/main/src

No, this is deliberate; parsers are too big to track under version control; the linked parsers will also stop including them soon.
These will be part of release artefacts, so if Emacs can't handle including a tree-sitter generate step in its install procedure, you should wait until the necessary infrastructure is set up. (Hopefully before Christmas.)

@clason: Thank you for your detailed answer!

the linked parsers will also stop including them soon

Could you share some sources for this information? I would like to ensure that these plans are not limited to this repository. It would be helpful to have links to them so that the corresponding changes can be made in Emacs.

https://github.com/tree-sitter/tree-sitter/issues/930 (under "Mergeable Git Repos"). The plans are definitely not limited to this repository; all parsers under https://github.com/tree-sitter-grammars (that don't already) will do this.


Docs : Add Coming Soon

Codecov Report

Merging #250 (1094ca5) into master (d860dd4) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #250   +/-   ##
=========================================
  Coverage     38.42%   38.42%
  Complexity      556      556
=========================================
  Files           178      178
  Lines          3284     3284
  Branches        389      389
=========================================
  Hits           1262     1262
  Misses         1884     1884
  Partials        138      138

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d860dd4...1094ca5. Read the comment docs.


Configure credentials for the container

Need an ability to securely pass credentials to the container not otherwise available on the host.
Should be able to use k8s secret to inject env variables and mount files.

In the .runtainer.yaml - should be able to specify:
secrets:
  env:
    - foo
  volumes:
    - bar

Also from command line:
runtainer --secret-env foo --secret-volume bar alpine sh

Resulting pod will be generated as follows:
apiVersion: v1
kind: Pod
metadata:
  name: runtainer
spec:
  volumes:
    - name: secret-bar
      secret:
        secretName: bar
        optional: true
  containers:
    - name: runtainer
      image: alpine
      envFrom:
        - secretRef:
            name: foo
            optional: true
      volumeMounts:
        - name: secret-bar
          mountPath: "/rt-secrets/bar"
          readOnly: true


added linear search code for C

added a easy to understand linear search code where the search function is separate to the main function

The code must be pushed to the c branch, the master branch is reserved for c++ implementations. In any case, linear search has been added. Thanks for the help!


removed duplicate 'moment' dev dependency

it's present in regular dependency list

fyi @NuclearRedeye , this got overlooked in the previous bump.
the changes to npm-shrinkwrap.json are the same whether are bump the duplicate or remove it, so I just removed it.


Migrate eSTS Telemetry to common4j

Changes

Migrates EstsTelemetry and related classes to common4j

The old class is kept as an adapter.

EstsTelemetry now consumes interfaces (ICommand, ICommandResult, IBaseException, IServiceException) instead of the actual objects.

This allows us to only migrate this layer, not every objects that it touches.
It's also easier to write unit tests.

Add a couple of null check and loggers in a few methods.
Add a handful of unit tests for eSTS telemetry. (@shahzaibj  please double check if the results are valid 😅 Also let me know if I should add anything.)
Remove @ Deprecated from Logger, so that it doesn't break OneAuth (for now).

Note: MSAL test case is now broken because of this PR. here's the fix.
https://github.com/AzureAD/microsoft-authentication-library-for-android/pull/1394

Do we have ADO tasks reminding us to remove the adapters in MSAL 3.0/Common 4.0?

Note: MSAL test case is now broken because of this PR
Fix is in: AzureAD/microsoft-authentication-library-for-android#1394


vmaxq_f64 in float64x2 detail::insn::i_min

This seems wrong, it should probably be vminq_f64.
https://github.com/p12tic/libsimdpp/blob/39354c3e65b93ebdf26d1d954dcf57b8249e1c33/simdpp/detail/insn/f_min.h#L68

Indeed, thanks' for noticing.

Fixed in 3e5950c80dbbd97f7314476450fb0f01a33dcb26.


auto mtls: need fallback to simple tls instead of plaintext

Describe the feature request
Today, the auto mtls feature falls back to literally plaintext if mtls is not available. This will not work in most enterprises today because almost all their deployments (k8s or vms) use one-way TLS from the ingress LB to pods or VMs. The TLS certs in the app is the same as the TLS certs in the load balancer (this is the poor mans way in the enterprise to ensure that all data in transit remains encrypted). There are two strong use cases for the tls fallback:

With Istio, we are advocating that people switch to plaintext for all outbound communications and let the sidecar do any form of TLS, so that they can get rich telemetry, routing, etc. However, when the sidecar has to talk to non-mesh services, we effectively end up sending plaintext traffic from the sidecar to these non-mesh services [which ends up breaking as these services require TLS]. So this will then force the user to either author destination rules for each and every dependency or simply do HTTPS from the application and let the sidecar do SNI passthrough. If they do the latter, they lose all benefits of Istio - especially in a large deployment where Istio is being tested and gradually rolled out.
The introduction of auto mTLS was really helpful because sidecars could now automatically detect if the other side has a sidecar or not, and if so, it will do mtls. If not, it will do plaintext. however, as described above, in most enterprise setups, these sidecars will need to do one-way TLS rather than plaintext. By allowing automTLS to fallback to one-way TLS we can solve this problem.
The only caveat is that the user wont be able to specify root certs that will verify the server side CA cert. This, while not so great, is still a LOT better than simply doing plaintext to all unknown services, and breaking compliance [all data in transit is not encrypted]. If the user really needs to verify the server side CA, i.e. security is of paramount importance, then they would not be doing PERMISSIVE mode or autoMTLS at all. So, given that these users are willing to relax the security constraints during the migration period (by allowing permissive mode, protocol sniffing, fallback from mtls to literally-plaintext), they are okay with doing unverified https compared to plaintext.

The second use case is incremental migration of these TLS workloads themselves. Imagine the following scenario:

User has a details.vm.internal service that is part of bookinfo app

This service is running on 400 VMs where these VMs have a TLS cert (one-way TLS)
This service receives traffic only from k8s where Istio is installed (i.e. from sidecars).

Initially the user created a destination rule that initiated one-way TLS from the productpage service in kubernetes.
The user is slowly migrating these VMs to use Istio mutual TLS, by adding sidecars on these VMs and letting the sidecar take care of all mutual TLS traffic.
During this migration, some set of VMs now gain the ability to use mutual TLS while the rest are doing one-way TLS. However, the destination rule will force all traffic to use one-way TLS and thus results in 503s. If I remove the destination rule, traffic to VMs using one-way TLs will fail, while traffic to VMs using mutual TLS will succeed due to auto mTLS. This is because the auto mtls implementation defaults to falling back to literally plaintext mode. In this case, it should be falling back to the one-way TLS.

A simple fix to the problem is to stop having a boolean label like security.istio.io/mtlsReady: true. Instead, we should have security.istio.io/tlsMode: plaintext|simple-no-verify|istio. If the endpoint has plaintext label (which is mostly unlikely), we can use plaintext from the client. If it has sidecar-no-verify label, we can use simple one-way TLS from the sidecar. Granted we will not be able to do root ca verification in this case, but keep in mind: this is a migration use case like the permissive mode on server side - i.e. temporary.

Describe alternatives you've considered
Additional context

A nice benefit of this idea is that it will also work for @uk000 and co, where they have mixed mode (k8s pods with mtls, and VMs with one-way TLS), without requiring API changes.
Since we have not made a release yet, we should consider changing this label soon. Otherwise, we will have to support backward compatibility and cause more confusion

A little worried about tlsMode label and the complexity to under these label meanings and apply at the correct times during the transition period.
if they have 400 services on vm, can we ask them to do canary on these 400 services, like 20% add envoy sidecar (auto mTLS) while 80% stay on one way TLS (via destinationrule) and subsets?  can we put the services with sidecar different subset than services without sidecar?  I'd think the service code could be different too as sidecar can reduce lots of non-essential code from the service.

The question here is why can’t we do the same for auto mtls as well ? Why do we need to have this auto mtls option when we can tell end users to do subsets etc ?
If we added auto mtls for simplicity, where end users have to add these labels for non k8s workloads, then we should maintain the same approach for simple tls. Also remember that this will become useful in kubernetes as well if people have migrated their workloads from vms to k8s, but haven’t added sidecars yet. Simply assuming that anything not mtls is plain text is forcing users to an even more insecure system than allowing them to do one way tls as they are doing today.

So security.istio.io/tlsMode:istio is the new security.istio.io/mtlsReady: true?
auto mTLS was introduced when user changes from permissive to strict for services already have the sidecar.  I'm just trying to understand how common your cases is and if it is necessary to cause confusing in the configuration for simple cases:

user enable strict mTLS for a service or a namespace or the mesh.  This is where auto mTLS can become really handy.  Many of our tutorials in istio.io is this simple scenario.
in your specific case, I feel people don't want to declare the services in transition to serve traffic until it is well tested/validated.  That is why I proposed canary or a/b testing for those.

So security.istio.io/tlsMode:istio is the new security.istio.io/mtlsReady: true?

yes and it will be auto injected. So no user configuration in either case.

auto mTLS was introduced when user changes from permissive to strict for services already have the sidecar.
That is no longer true. See below :)

I'm just trying to understand how common your cases is and if it is necessary to cause confusing in the configuration for simple cases:

user enable strict mTLS for a service or a namespace or the mesh.  This is where auto mTLS can become really handy.  Many of our tutorials in istio.io is this simple scenario.

Nope. auto mtls helps to automatically do mTLS when the calling sidecar knows that the destination also has a sidecar, irrespective of whether the destination has strict or permissive mode. This way, anytime you add a sidecar to a service (partial or full), all communications to this sidecar (from other sidecars in the mesh) will automatically do mutual TLS - you dont even have to bother changing from permisssive to strict. Thats the beauty of it. The more sidecars you add, the more mtls you get with zero change.

in your specific case, I feel people don't want to declare the services in transition to serve traffic until it is well tested/validated.  That is why I proposed canary or a/b testing for those.

So my case is exactly same as the auto mtls case. The biggest assumption/mistake in auto mTLS is that the fallback in case there is no sidecar is always plaintext. This is not an acceptable solution in enterprises because almost all their workloads use one-way TLS from the load balancer to their application pods in k8s or VMs. No one in financial sector does clear text traffic [even before istio]. So, our current model of auto mtls will become useless to them because they cannot do plaintext from a sidecar to a non sidecar pod. I have updated my use case above

Thanks for the additional detail and update!  So I think I get there is a need for 3 modes... I was somewhat confused to use these 3 modes.... Here are the 3 modes:

no fall back, always plain text.
Fall back for mTLS (if mTLS doesn't work out):

2.1  plain text
2.2 one way TLS

Are these correct?

auto mtls helps to automatically do mTLS when the calling sidecar knows that the destination also has a sidecar, irrespective of whether the destination has strict or permissive mode. This way, anytime you add a sidecar to a service (partial or full), all communications to this sidecar (from other sidecars in the mesh) will automatically do mutual TLS - you dont even have to bother changing from permisssive to strict. Thats the beauty of it. The more sidecars you add, the more mtls you get with zero change.
I'm concerned with this... How would I know if I'm enforcing mTLS for my sidecars?  Today I can do it via mesh policy or/and auth policy. Users may likely want the flow to fail when the services not using mTLS when it is mandatory to use mTLS for all services communication.

When you enforce mTLS, the server side will be strict completely. So, any client (sidecar or not) that does plaintext will fail to communicate. At that point, on the client side, all sidecars will default to using the mutual TLS because all the destinations will have that label.

As comment in the PR, can we design a proper API for setting fallback TLS options instead? I think it would be more flexible to address all needs (the simple TLS fallback seems to limited to me, for example how does it works if server use self-signed cert), and does not complicate the label semantic.

Today, the auto mtls feature falls back to literally plaintext if mtls is not available.

I think this feature is mixing "service mesh transport security" (which is either mTLS or plaintext) with "workload transport security" (which is either one-way TLS or plaintext; we don't have client certificates to perform workload mTLS from the sidecar, so we treat it as opaque bytes, i.e. plaintext).

we don't have client certificates to perform workload mTLS from the sidecar, so we treat it as opaque bytes, i.e. plaintext).

So yeah, actually, could users continue to let client app terminate the one way tls connection for those not migrated server workloads? This meets the requirements of not falling back to plain-text, even though traffic is opaque bytes in the point of client sidecar.
If the answer is no, is it because users can't use L7/http Istio features if sidecar don't understand the protocol?

Clearly we have an issue with a layerng violation. We would rather have as @PiotrSikora describes a mechanism for transport security that can't collide with application security. The fact that we use TLS on the same port as the application port is causing us pain.
Now in this case there is no reason why the fallback behavior can't be the behavior assigned to the endpoint in the DestinationRule rather than plaintext. It's still a layering violation but at least it keeps TLS if it were declared.
Separately we should look for a more holistic solution to the layering violation.
@rlenglet FYI

I dont think it is a violation. Remember that people adopt a mesh for telemetry as well. When tehy add a sidecar to their app, we advise them that they can do plaintext from their app and we will initiate TLS on their behalf, if they want telemetry, routing, etc.  And that you will not get telemetry as you progress in adoption.
Imagine an org that has 4000 namespaces across VMs and k8s. IF the end user adopts the mesh on a namespace by namespace basis, the first tangible benefit is telemetry (for traffic to in-mesh sidecars and traffic to non-mesh apps as well). So, telling these people that adding a sidecar is going to be of little use until the entire system is on the mesh is not a great thing to do, especially when the competitors are providing the same feature.

what the "security.istio.io/tlsMode:istio" feature?


grace: init at 0.0.1

Description of changes

it's strace, with colours
Closes #200531
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

I also built on aarch64-linux, but the build fails.  For now I have set platforms to x86_64-linux.

Can you please post the failed build logs on aarch64? Someone might be able to debug it.

aarch64-linux build log...
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/dqggbh7dj32cf0fxylzyrq89dckm1i7w-source
source root is source
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Building subPackage .
internal/unsafeheader
internal/race
internal/goos
internal/goexperiment
internal/itoa
encoding
math/bits
unicode/utf8
internal/nettrace
internal/goarch
unicode
sync/atomic
runtime/internal/syscall
internal/cpu
internal/abi
runtime/internal/math
runtime/internal/sys
github.com/rivo/uniseg
math
runtime/internal/atomic
internal/bytealg
runtime
internal/reflectlite
sync
internal/singleflight
internal/testlog
runtime/cgo
errors
sort
strconv
internal/oserror
path
vendor/golang.org/x/net/dns/dnsmessage
io
syscall
bytes
strings
reflect
bufio
regexp/syntax
internal/syscall/execenv
internal/syscall/unix
time
regexp
context
io/fs
internal/poll
internal/fmtsort
encoding/binary
encoding/base64
golang.org/x/sys/unix
os
internal/godebug
github.com/mattn/go-runewidth
os/signal
path/filepath
fmt
internal/intern
os/exec
net/netip
encoding/hex
net/url
flag
encoding/csv
text/template/parse
golang.org/x/term
github.com/aquasecurity/table
text/template
net
github.com/liamg/grace/tracer/netw
github.com/spf13/pflag
github.com/liamg/grace/tracer/annotation
github.com/spf13/cobra
# github.com/liamg/grace/tracer/annotation
tracer/annotation/ptrace.go:18:7: undefined: unix.PTRACE_GETFPREGS
tracer/annotation/ptrace.go:19:7: undefined: unix.PTRACE_SETFPREGS
tracer/annotation/ptrace.go:22:7: undefined: unix.PTRACE_GETFPXREGS
tracer/annotation/ptrace.go:23:7: undefined: unix.PTRACE_SETFPXREGS
tracer/annotation/mmap.go:22:8: undefined: unix.MAP_32BIT

Closing because I no longer use this.  Anyone else is welcome to fork this if they want to maintain it :)


CakePHP3 does not seem to work

Maintainability.................... 67.71 / 100
Accessibility for new developers...    50 / 100
Simplicity of algorithms...........     0 / 100
Volume.............................     0 / 100
Reducing bug's probability.........     0 / 100

Hi,
hum, looking in source code (v3.2), score is coherent:

Bugs by file: 0.91 (good: < 0.07)
Complexity cyclomatic (average) : 17,89 (good: < 2)

And all other indicators are same. I suggest to read the HTML report to see details.
But, indeed, 0 is maybe... a bit naughty. Formula should be improved.


Document how to tune storage specs

Currently there is no preferred approach to tune storage configuration for benchmark environment. Documenting the same would help the user to play with difference storage types(gp2, gp3, io,.. etc), IOPS and throughput.

Hello! The general recommendations are the same as here https://docs.victoriametrics.com/Troubleshooting.html#slow-data-ingestion
Even when VIctoriaMetrics is optimized for running on slow disks, using SSDs will just make performance better.
IOPS and throughput are vary only for clouds and each has its own specifications. A general recommendation is to have bigger disks because most of the cloud providers correlate IOPS allowance with disk size.


not able to connect cosmos database through managed identity

Describe the Bug with repro steps
go to designer and add an action.
search cosmos database query operation.
the UI just provides the connection string to be filled out.
I have already added logic app's managed identity into the cosmos db. not sure is it working and how to work it for managed identity.
https://ms.portal.azure.com/#view/Microsoft_Azure_EMA/DesignerEditor.ReactView/id/%2fsubscriptions%2fe5874924-875c-48d4-874b-1d410e6514c5%2fresourceGroups%2fmediavalidation-ppe%2fproviders%2fMicrosoft.Web%2fsites%2fmvsteststatus%2fworkflows%2fmvsteststatusmonitor/location/East%20US/isReadOnly~/true/isMonitoringView~/true/runId/08584748190643678493686429590CU00
What type of Logic App Is this happening in?
Standard (Portal)
Which operating system are you using?
Windows
Are you using new designer or old designer
New Designer
Did you refer to the TSG before filing this issue? https://aka.ms/lauxtsg
Yes
Workflow JSON
No response
Screenshots or Videos
No response
Browser
edge
Additional context
No response

Hi @hsiangtingyang, from what I can tell the only connection authorization available for the Cosmos DB connector is via connection string, MI is not supported as far as I'm aware

Are there any future plans to support it in Logic Apps?


Step-size methods and preconditoners for GD

Currently a draft for discussion purposes
TODO:

[x] testing the preconditioners
[x] Unit tests for the preconditioners
[x] Documentation for the preconditioners
[x] Unit tests for the step-size methods
[x] Think about efficiency for the Armijo rule
[x] Think about efficiency for safe division in preconditioners
[x] Documentation for the step-size rules
[x] Testing for ISTA and FISTA with step sizes and preconditioners
[x] Documentation for ISTA and FISTA
[x] Think of a new name for the preconditioner class - it does more than precondition, it can modify the gradient (or anything else) in any way it wants

Changes
Added step-size methods and preconditioners for GD
Testing you performed

Please add any demo scripts to https://github.com/TomographicImaging/CIL-Demos/tree/main/misc

Related issues/links

Closes #1762,
closes #1592,
closes #1588
closes #1359
closes #1817

Checklist

[x] I have performed a self-review of my code
[x] I have added docstrings in line with the guidance in the developer guide
[x] I have updated the relevant documentation
[x] I have implemented unit tests that cover any new or modified functionality
[x] CHANGELOG.md has been updated with any functionality change
[x] Request review from all relevant developers
[x] Change pull request label to 'Waiting for review'

Contribution Notes
Please read and adhere to the developer guide and local patterns and conventions.

[x] The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License
[x] I confirm that the contribution does not violate any intellectual property rights of third parties

--->

Preconditioner -> SearchDirectionUpdater or something?

Closes #1817

Jenkins is happy

Thanks a lot all! Yet another major milestone


Popped Regen gBalloons shouldn't reset regen timer

Currently, after a Regen gBalloon is popped, its children will have their regen timers reset, regenerating a layer only after 2 seconds.

Resolved via commit f561f7cb583760eb7da0d0f36da1eabc08bed6ab.


Allow Interceptors to add EventListeners.

#fixes https://github.com/square/okhttp/issues/7164

@oldergod related to https://github.com/square/okhttp/issues/7297 somewhat, I wonder if we want to allow adding an EventListener to a Call before it's executed or cancelled?

@swankjesse any thoughts on this?

@swankjesse ping, it would be good to get this into an alpha for bridge integrations.

I’m not sold on this.
I think the top-level goal is reasonable, but this implementation isn’t consistent with our other with methods on interceptors – this collects all events; not the events downstream of proceed. In particular I think the with method would at minimum need to be add.
But then I don’t like other side-effects this can trigger. Like if the interceptor runs 2x, it’ll add the listener 2x.
I think if we really wanted this behavior, we’d be better off adding interceptors to Call objects. And we practically already have that already in OkHttpClient.
Perhaps we start with some use cases and build the API to address them, rather than taking interceptors and trying to make them more powerful.

Ok, will rework.


how to show local images in description in swaggo

I want to show some local images in description.

I managed by embedding image project and exposing API endpoint (I am using Gin), which then reads from embedded image.
I created following folder structure
  static
  ├── main.go
  └── images
          └── image1.png

In static/main.go:
package static
import "embed"

//go:embed *
var Files embed.FS

In function, which gets called by gin. Note that ReadFile will return byte array.
func (a *Controller) getSwaggerLocalFile(c *gin.Context) {
  b, err = static.Files.ReadFile("images/image1.png")
  if err != nil {
      return
  }

  c.Data(http.StatusOK, "image", b) // <- if you have any type other than image, you'll have to detect ContentType on your own
}

In Swagger documentation:
// My API route is called api/swagger_local_file/image1.png, where image1.png is image name which I'd like to show in swagger. In code, which reads from static.Files.ReadFile I hardcoded this parameter
// @Description ![Image 1 Example](api/swagger_local_file/image1.png)


[1987] Ensure that Sub Commands contain Provider Recruitment Cycle year

Context
When using bin/mcb provider edit ABC -r 2020 argument to specify a Recruitment Cycle year and then editing a course from within that Provider. The Courses loaded are for the Current Recruitment Cycle.
Changes proposed in this pull request

Always specify the Recruitment Cycle

Guidance to review
Is there a better way to test this?
Checklist

[x] Make sure all information from the Trello card is in here
[x] Attach to Trello card
[x] Rebased master
[x] Cleaned commit history
[x] Tested by running locally

 MCB isn't cycle aware for 'providers edit'

Question how useful is this warning from CodeClimate?

Class ProviderEditor has 21 methods (exceeds 20 allowed)

@misaka how useful is this warning from CodeClimate?

Class ProviderEditor has 21 methods (exceeds 20 allowed)

Enh. I mean, tbh, looking at that module, it's actually quite big and probably could benefit by pulling a couple methods out. The simplest way forward may be to remove the update one-liner:
    def update(attrs)
      @provider.update(attrs)
    end

Does this actually serve a purpose?


[2020-02-27]: [Notebook conversion to blog post not showing up]

Prerequisites for filing a bug

[ ] Did you run the local development server as indicated in the development guide?

Please include any useful logs or error messages from that

[x] Have you searched opened / closed issues on this repo for a similar problem?
[ ] Have you searched the fastai forums for a similar problem?

Please include the following information

Steps to reproduce the problem
I placed 2020-0227-*.ipynb file in _notebook which has  791 KB capacity. It failed to show up in the blog page. Am I missing something here are there any limit to notebook capacity?

I see that it has skipped because of future date, I have put the naming convention as per my local time IST. I guess the github action might be running in different time zone than mine. Any resolution for this ?
  Skipping: _posts/2020-02-27-Diagram-as-Code.md has a future date

Reference log
https://github.com/byteshiva/blog/runs/470708733?check_suite_focus=true

@byteshiva can do me a favor
Set the following in your_config.yml file and let me know if it works
future: true

This is fixed by https://github.com/fastai/fastpages/pull/130

@byteshiva can do me a favor
Set the following in your_config.yml file and let me know if it works
future: true

Thanks @hamelsmu, it worked after settings future: true
Ref:
https://github.com/byteshiva/blog/runs/470888228?check_suite_focus=true


add flair SegtokSentenceSplitter

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.83%. Comparing base (b8256c8) to head (864b31e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #105      +/-   ##
==========================================
+ Coverage   95.82%   95.83%   +0.01%
==========================================
  Files          57       57
  Lines        4524     4543      +19
==========================================
+ Hits         4335     4354      +19
  Misses        189      189

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Error upon non-existing places are unintuitive

Consider the following example;
package main

import (
	owm "github.com/briandowns/openweathermap"
	"log"
)

func main() {
	w, err := owm.NewCurrent("C", "en", "<redacted>")
	if err != nil {
		log.Fatal(err)
	}

	err = w.CurrentByName("St Petersburg")
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("Temperature in %s: %.1f °C\n", w.Name, w.Main.Temp)

	err = w.CurrentByName("fff")
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("Temperature in %s: %.1f °C\n", w.Name, w.Main.Temp)
}

This outputs json: cannot unmarshal string into Go struct field CurrentWeatherData.cod of type int as an error for the second CurrentByName call.
Perhaps a more descriptive error message could be used here?

The response of the particular query is 404 Not Found. Do you think we need to just map the response codes to messages?

I would think that returning the provided location with the error would provide necessary context. Would more be needed?


Support case sensitive query

Hi, is it possible to support case sensitive queries?

I see https://github.com/mongoid/mongoid_search/blob/master/lib/mongoid_search/util.rb#L41. So not without changes.
I do recommend you use something more advanced than this gem though, like mongoid_fulltext.


Add FsGroup in podSecurityContext

Description
This PR adds FsGroup setting in PodSecurityContext for container based instances (for example cinder volumes begin with root user write access only).

/merge


Login error - Something went wrong during login to connectycube

I have tried to use the entire code from this repository
I got these error:
W/vibeland_clien( 7109): Accessing hidden method Lsun/misc/Unsafe;->putInt(Ljava/lang/Object;JI)V (greylist, linking, allowed)
I/flutter ( 7109): CB-SDK: : =========================================================
I/flutter ( 7109): === REQUEST ==== acc46ab2-6ded-44fc-9b00-c6d41f98d46e ===
I/flutter ( 7109): REQUEST
I/flutter ( 7109):   POST https://api.connectycube.com/session
I/flutter ( 7109): HEADERS
I/flutter ( 7109):   {Content-type: application/json, ConnectyCube-REST-API-Version: 0.1.1, CB-SDK: Flutter 1.1.3, CB-Token: }
I/flutter ( 7109): BODY
I/flutter ( 7109):   {"application_id":"4350","auth_key":"6O63DQYfGsAXvm2","nonce":"2186220645","timestamp":"1614934812","signature":"8789a00d63824aedb37a760770a73650473cf1b4","user":{"login":"call_user_1","password":"6yE4GzKvoKN2Q2DsGiZC"}}
I/flutter ( 7109):
I/flutter ( 7109): CB-SDK: : *********************************************************
I/flutter ( 7109): *** RESPONSE *** 401 *** acc46ab2-6ded-44fc-9b00-c6d41f98d46e ***
I/flutter ( 7109): HEADERS
I/flutter ( 7109):   {connection: keep-alive, set-cookie: _mkra_ctxt=aafcd78dbbae13d73233439207148309--401; path=/; max-age=5; HttpOnly; secure, cache-control: no-cache, status: 401 Unauthorized, transfer-encoding: chunked, date: Fri, 05 Mar 2021 09:00:13 GMT, access-control-allow-origin: *, strict-transport-security: max-age=31536000, content-type: application/json; charset=utf-8, x-xss-protection: 1; mode=block, server: nginx/1.16.1, x-request-id: fc7d0cde-55ba-4e99-9da5-c6d5eecb43aa, connectycube-rest-api-version: 0.1.1, x-runtime: 0.026810, x-frame-options: SAMEORIGIN, x-content-type-options: nosniff}
I/flutter ( 7109): BODY
I/flutter ( 7109):   {"errors":["Unauthorized"]}
I/flutter ( 7109):
I/flutter ( 7109): CB-SDK: LoginScreen.BodyState: Login error ResponseException: 401: {"errors":["Unauthorized"]}

const String APP_ID = "xxxx";
const String AUTH_KEY = "xxxxx";
const String AUTH_SECRET = "xxxxxx";
const String ACCOUNT_ID = "xxxx";
const String DEFAULT_PASS = "mypasswordondashboard";

when I click on User 1 or User 2.... I got dialog which tell me login Failed

No activity for a long time, so close...


Bug: Unexpected drawer behavior

Describe the bug
I'd expect the drawer that you open to access home, repo insights, contributor insights, etc. to close upon clicking a link, however, it stays open.
If this is the intentional behavior, please close this out.
Steps to reproduce

Go to your workspace.
Click the button to open the drawer.
Click a link in there.

@nickytonline I absolutely see where you're coming from. Maybe other users don't expect a drawer to close upon clicking a link inside of it, thus why I didn't see another issue open for this. I agree it's a lot to add without real ROI; just food for thought!


made the example point to the default branch of the repo

the latest version (v1.2.0) isn't compatible with the example in the readme, updated the example to target the latest commit of the default branch of the repo

I understand how this can be confusing. Thanks for your addition. LGTM!


Question of the network.

Is there any way can play the game without port forwarding?

Not at the moment. Dedicated servers have been requested in #47, but I don't have any ETA on that

You can use OpenVPN
create a server
all players connect it
DCHP server must open


Allow to config Sasl configs in Kafka sink.

Motivation
Allow pulsar io to push messages to sasl kafka cluster.
Modifications
Add several kafka sasl configs, make them configable.
Documentation
I think the pulsar-io can automatically generate the doc for added fields. So we don't need add any docs.

for the tests simple unit tests around KafkaAbstractSink#open() is enough.
We do not need integration tests (they will be very heavyweight)


Selector for CSS class needed

Hi,
is there a button that makes it possible to assign a css class to the current selection?
I could only find the button "formatting" that lists tags like h1 and p. I'd need a button that provides a list of css classes.
Or has anyone a plugin for this?
Thanks
Hyper

There is no such plugin for now


restricting access for MM only

Hello All:
At this point jitsi is setup and is working within mattermost and outside mattermost.
I would like to limit access to within mattermost only - thereby disabling direct access via the url: (example: meeting.johnson.com).
Any ideas or helpful hints.

Does the JWT authentication solve your use case?


azure-cli - osx - segmentation fault -

Describe the bug
getting segmentation fault errors when running az
/usr/local/bin/az: line 8: 12762 Segmentation fault: 11  python3 -m azure.cli "$@"

To Reproduce

Install azure-cli as per documentation,
run az login from cli
Browser open for login
Obtain "You have logged into Microsoft Azure!"
Segmentation fault error

Expected behavior
I expect az cli to work
Environment summary
installed as per docs
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-macos?view=azure-cli-latest
brew update && brew install azure-cli
osx 10.13.6
python3
└─[$]> az --version
azure-cli (2.0.46)

acr (2.1.5)
acs (2.3.4)
advisor (0.6.0)
ams (0.2.3)
appservice (0.2.4)
backup (1.2.1)
batch (3.4.0)
batchai (0.4.3)
billing (0.2.0)
botservice (0.1.1)
cdn (0.1.1)
cloud (2.1.0)
cognitiveservices (0.2.3)
command-modules-nspkg (2.0.2)
configure (2.0.18)
consumption (0.4.0)
container (0.3.4)
core (2.0.46)
cosmosdb (0.2.1)
dla (0.2.3)
dls (0.1.3)
dms (0.1.1)
eventgrid (0.2.0)
eventhubs (0.2.4)
extension (0.2.1)
feedback (2.1.4)
find (0.2.12)
interactive (0.3.30)
iot (0.3.2)
iotcentral (0.1.2)
keyvault (2.2.3)
lab (0.1.1)
maps (0.3.2)
monitor (0.2.3)
network (2.2.5)
nspkg (3.0.3)
policyinsights (0.1.0)
profile (2.1.1)
rdbms (0.3.2)
redis (0.3.2)
relay (0.1.2)
reservations (0.4.0)
resource (2.1.4)
role (2.1.5)
search (0.1.1)
servicebus (0.2.3)
servicefabric (0.1.3)
signalr (1.0.0)
sql (2.1.4)
storage (2.2.2)
telemetry (1.0.0)
vm (2.2.3)

Python location '/usr/local/bin/python3.7m'
Extensions directory '/Users/jdenis/.azure/cliextensions'

Python (Darwin) 3.7.0a2 (v3.7.0a2:f7ac4fe52a, Oct 16 2017, 21:11:18)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

Fix by modifying /usr/local/bin/az

Commenting PYTHONPATH

#!/usr/bin/env bash
#export PYTHONPATH="/usr/local/Cellar/azure-cli/2.0.46/libexec/lib/python3.7/site-packages"
if command -v python3.7 >/dev/null 2>&1; then
  python -m azure.cli "$@"
else
  /usr/bin/python -m azure.cli "$@"
fi


Remove generated proto files

Keeps the proto definitions themselves in Cirq.
Also a variety of cleanup of a pattern where a method both mutated and returned an object.  This was slightly useful for the old protobuf python interface (which was clunky), but is not as useful when just using straight dictionaries.

Very happy to see this finally going away.


Correct SHA may not being chosen when used in a pre-merge commit

Issue: My notifications are not working when using the CLI.
In the case that the incorrect SHA is picked, Codecov will show the commit message as merge <SHA>
Workaround: You can manually pass the correct SHA to the CLI with the --sha flag.
If you are using the CircleCI orb, or the GitHub action, please see that wrappers's docs for the method to manually pass the SHA.
We are aware of this issue.

The Uploader uses the following logic to detect the correct SHA in a pre-merge commit:
https://github.com/codecov/uploader/blob/1e505ba8a27a1e3311a7a8db7f314e3705a2ee08/src/ci_providers/provider_githubactions.ts#L97-L122

@drazisil-codecov can you check if the following are related

https://github.com/codecov/feedback/issues/126#issuecomment-1931862637
https://github.com/codecov/codecov-action/issues/1274#issuecomment-1934469452


New dataset and metadata integration

Todo's

[ ] @aaravpandya Verify with Madrona viewer
[ ] @aaravpandya Review PR
[x] @daphnecor Test run on 10 scenarios + videos to verify correctness
[x] @nadarenator Fix metadata design
[x] @daphnecor Update tutotrials

This PR has breaking changes with the datasets. Please change the dataset before merging in this PR.
Also, can you change the file in tests/pytest_data with the new format? Tests wont work with the old file format.

Todo @daphnecor :

[] Change dataset files
[] Change test file

Todo @aaravpandya :

[] Test code

This PR has breaking changes with the datasets. Please change the dataset before merging in this PR.
Also, can you change the file in tests/pytest_data with the new format? Tests wont work with the old file format.

Yes, we will do that this week. Since this PR cannot be merged until we fix the data, I have taken your code and applied it to our integrations/vbd branch temporarily.

@nadarenator Let's see if we can update the dataset links in the readme today so we can merge this PR

I think we should hold off on regenerating the dataset until we finish vbd alignment, unless you're confident that the current processing script extracts all required info

Is regenerating that expensive? We can just regenerate a small number of files for testing

Yes that's what I meant, using a small set for our purposes. But before merging this pr we'd need updated links, which would involve regenerating the whole dataset. And if we find out during vbd integration that we need an additional piece of info from the raw tfrecords we'd have to regenerate the full dataset again.

Only thing remaining: Regenerate dataset and upload to hf, add link to readme.

Only thing remaining: Regenerate dataset and upload to hf, add link to readme.

Can you send me a link to the new data once it's up? I'd like to do a test PPO run

It should be available now on my public directory on greene (the same one I shared before).

Would be great if we could redo some of the tutorial notebooks, most of the indexing is outdated, and there's also datatypes now that handle the indexing internally

I am approving this because I am unable to fix the tests. However, I have done a smell test, visualized the scenarios and reverified the code. I am confident that everything should be working fine. But considering that this feature is blocking work on vbd integration, I think its fine to merge it in.
I will be working on getting the tests to work in a separate PR.
Also, @daphnecor since many people are using GPUDrive, I would recommend bumping the version number of our current release, and putting a "warning/info" card on the readme to notify people that this new version has breaking changes with the previous dataset and that they should download a new version.

I am approving this because I am unable to fix the tests. However, I have done a smell test, visualized the scenarios and reverified the code. I am confident that everything should be working fine. But considering that this feature is blocking work on vbd integration, I think its fine to merge it in.
I will be working on getting the tests to work in a separate PR.
Also, @daphnecor since many people are using GPUDrive, I would recommend bumping the version number of our current release, and putting a "warning/info" card on the readme to notify people that this new version has breaking changes with the previous dataset and that they should download a new version.

Great suggestion. I will add the release and description


Shibboleth auth UI Rancher API Host field is disabled and certs textboxes missing when trying to configure it.

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):

Install Rancher 2.4
Navigate to Security -> Authentication -> Shibboleth

Result:
The UI is displayed to the user but the Rancher API host field is disabled.

It seems that the Certificates IDs textboxes are also missing, because clicking Authenticate with Shibboleth those fields are required but are not visible:

Other details that may be helpful:
This was previously working before this other issue was fixed: https://github.com/rancher/rancher/issues/25426
Environment information

Rancher version: 2.4 master-head (02/18/2020) commit id: 580ab5bb4
Installation option (single install/HA): single

Available in latest2

Rancher version: 2.4 master-head (02/18/2020) commit id: 580ab5bb4
I'm able to configure Shibboleth auth now (with and without OpenLDAP), the fields are visible and editable in the UI. Also navigating to other SAML auth providers didn't reproduce #25426 either.


Design & implement system that reduces storage pressure and provides client feedback when PUSH is throttled

This is fork from https://github.com/microsoft/FluidFramework/issues/6596. That issue will continue to track handling of the nack with retryAfter, while this issue will track the following issues:

The lack of feedback to client when PUSH is throttled due to flushing ops. Currently, 99% of that is due to new client joining, as ops flushing is coalesced with new summary commitment, and thus has direct way to provide client feedback in the form of nack.
Optional - reduction or complete elimination of a need to flush ops by PUSH to storage on client joining.
Optional - exposing a way to fetch ops from PUSH (redis).

The key goal should be to push throttling up the stack, such that if the issue is - too high ops rate, then client has enough information to learn about it and possibly adjust its behavior. While long term we may want to even disallow users to make changes, for now we will focus consuming this signal in summarization workflow, i.e. make sure client does not keep piling uncommitted summaries while PUSH has trouble and thus creates positive feedback loop.
The discussion of signals in #6596 is directly related to this problem that we fork into this issue.

Here is the summary of discussion between @GaryWilber, @marcmasmsft  & me:

We commit to a plan for SPO fetch ops API to be single source of truth for ops, and thus fetch ops (on as needed bases) from PUSH / redis

the details needs to be designed. It's likely that client would need to provide some hint for this flow to be engaged, both as a staging mechanism, but also as a way to reduce extra work on the server (later might be indirect, for example we should disambiguate cases where client asked for specific range, vs. client proactively asks for op tail with no specific expectation on how many ops are available).
There is some uncertainty here due to 3 entities involved (and service to service calls), so we need to test feasibility of it.
Ops commit while submitting summary needs more discussion. It's possible that we will stay with existing flow (i.e. push model, not pull) which would require client to ask push to flush ops before client can proceed with submitting summary in single commit flow.

Marcelo to work on design doc, publish it, and figure out execution plan.
This will take substantial time to evolve, so if we need some shorter mechanism to solve immediate scalability problems, then we should address it independently.

Temp solution in this space should be staged in a way where service can retire such temp capability easily, in other words - client (and whole system) should behave correctly when such new capability is missing or become missing in the future.

Here is a proposal from me on how to address this problem (or rather consideration points). I start with just flush() API as consideration, but look into getOps() type of API as well (it has very similar requirements RE implementation).

PUSH to implement flush() API (available via via socket messages) to flush some set (current?) of ops to SPO.
Client to provide indication in connection message if client supports flushing and will use it if server supports it (enlightened clients).

server would not flush ops on connection for such client, otherwise it keeps old behavior.

bto communicate back to client if it supports flushing

Client should keep existing behavior (not calling this API) if server does not support it, such that PUSH can revoke this API in the future without notice.

Client usage of flush:

not flushing on summary (at least for now with 2-phase summary commit), with assumption that ops flushing is bundled with summary upload by PUSH and nack is used for communicating response.
- we can revisit this in the future.
Client will ask to flush ops on hitting gap in ops (after preforming first successful storage ops fetch call but not filling in the gap)

Providing feedback (success / failure) is optional from POV of client that tries to fetch missing ops (as such client can just keep asking SPO for ops for 30 seconds, though feedback would reduce frequency and make telemetry more actionable).
But it's not optional from POV of summarizer client who is blocked by this operation. So there is a need to provide that feedback (at least 429 / any other recoverable failures) to all clients.
instead of flush, we can use getOps() here with similar impact to client. In such case feedback communication is part of API. Perf characteristics are slightly different. It will result in fewer SPO calls, and improved latencies, so it definitely needs to be considered, but flush() is more versatile as it maybe used in the future as a pre-step for summary upload.

Benefits (assuming all enlightened clients):

PUSH does not need to flush ops on connection, substantially reducing number of SPO calls (in some situations, including expiring tokens), as well as provide recovery mechanism for fetching ops that were lost through pipeline.

Gap:

if we have a mix of clients, then enlightened clients need to know about anybody (clients, PUSH) doing flushes and hitting problems. But that's not enough - summarizer may be non-enlightened client and thus not aware why summaries are not coming through and thus continue to post summaries and make things worse (positive feedback loop).

With that, I think there are two possible practical workflows:

flush() and signaling to when PUSH has issues on any flush().

Pros:

has slightly better characteristics in terms of client awareness of what's going on in a mix of enlightened & non-enlightened  clients (if summarizer is enlightened).
possibly reusable for other workflows

Cons: has more implementation nuances (signals need to be re-broadcast to any new clients that joins, signals can be lost, etc.)

getOps()

Pros:

More localized (it's just single client activity)
Better perf profile / latency

Cons: Mix of enlightened & non-enlightened clients will behave worse in terms of scaling down on throttling failures.

My choice would be to go with getOps() kind of API, which I'd guess will look like this:

"getOps" message from client
ops are returned through normal (socket) flow, maybe have some marker that they are result of prior getOps call (it may help client to assert that normal ops have to come sequentially).
some message back to client indicating completion of the call (client can use it to understand if gap was filled in or not). If failures can occur, they are communicated on same message.

I wonder if this is all necessary.
Why does push need to flush on each join?
We support IDocumentDeltaConnection.initialMessages? Does this not include all unflushed ops? Isn't that its purpose? is it not sufficient?
Even with IDocumentDeltaConnection.initialMessages there is still the problem of feedback. There is a server to client feedback that pushing ops to storage is behind so we should hold off summarizing, and maybe in the future take more drastic action like going readonly.  The other feedback is client to server, basically that it cannot find some op gap. If we only have to solve this, and not the above i wonder if other solutions are possible.

Gary would have more precise description here, there is no guarantee on ordering / lack of gaps. I do not remember precise reasoning here, but it was something to do with fetching these ops being async and potentially some ops being added to redis after this call is satisfied, and such ops being not broadcast earlier to that (connecting) client. So they exists only in redis past the point of connection (from client perspective) and are not available to client in any other form.

I'm not super familiar with this layer of the code, but i remember it working like this. The driver connects to the socket, which quickly starts sending ops, which the client should keep. The server then sends connect document success which includes outstanding ops as of that point, but only needs to contain up to the point of when the client started receiving ops.  Older ops come from the server.
It's possible push started flushing on each join, as it wasn't possible to reliably do the above. @GaryWilber and @tanviraumi is this a push only behavior? AFIK r11s writes these directly to mongo, so no need to flush anything. It seems other solutions are possible here, like tracking the first seq on each conn, and ensuring server side, that messages up to that seq are either flushed, or in initial messages.
also, @vladsud  do we envision this all being encapsulated in the driver? Seems like it could be hard to encapsulate with the separation of delta stream and delta storage

@anthony-murphy Yes it's a push only behavior.
For r11s ops are inserted into mongodb. When clients want to get ops it ends up checking mongodb. So there is a single source of truth for that data.
For push, ops are either in SPO or Push. And technically ops could be in 2 separate places within Push.
Clients can access SPO ops via the get ops api and clients get the Push Redis ops via initialMessages. But due to how we are broadcasting messages (we broadcast the op before inserting it into Redis), there is point in time when the op is technically inaccessible. The op could be in a kafka topic, waiting to be consumed and inserted into Redis. For docs with high op rates, clients are likely to not get all the ops when they connect due to this. This is why we always post ops to SPO when clients join.
I have prototyped the solution you described, where we track the join op seq# and send the missed ops to the client. However that solution cannot work for read-only connections because there is no join op for them, so I dropped working on that further because I think we need a more general solution for this.
One other solution is to insert ops into redis before broadcasting them. This would ensure clients joining will always get all the ops. However this would increase latency because we would now be waiting for that redis insert call before broadcasting - we didn't want that to be a thing.. but maybe it would actually be worth doing. I'm conflicted about it.

@vladsud Here's what the "get_ops" and "flush_ops" websocket apis look like right now.
Both apis have a request and a response and take in a client id & and object as params. Both objects contain a client provided nonce.
Get ops - only returns ops from Push's Redis
If all ops in the inclusive range based on the from & to params was found, code is 200. The ops are returned in the messages property.

If no ops were found, code is 404

If only some ops were found in the range, code is 206

Flush ops:

It returns a 200 if it flushed ops and includes the last persisted sequence number.
If no ops are flushed, it returns a 204.
In error cases the apis will return a code & retryAfter property (if available).
I could make the get_ops api broadcast the ops normally instead of making them only be returned in "get_ops_response" if you would prefer that. I would probably rename the event to "request_ops" instead of "get_ops" in that case.

@GaryWilber I don't know enough about how the server code is structured, but ideally, we can know the first op sent to any client, and then the server knows it needs to send the gap from that op down to the last flushed op in initial messages which is part of connect document success. This should be agnostic to write vs read.

@anthony-murphy That is a great idea. Originally the frontends were "dumb". They did not track what it was broadcasting to clients - they simply broadcasted whatever it was told and never inspected the payloads.  This is how r11s works too.
Due to some issues seen in the scale tests, I recently had to make the frontends smarter. They now parse the ops being broadcasted to clients. It tracks the sequence numbers per-document in order to detect issues (op gaps). Now that we have that capability, it should be possible to extend that logic further for per-client tracking too.
@vladsud Would a client recover correctly if this situation occurs:

Client gets initialMessages [1, 2, 3].
Server broadcasts 6.
Server broadcasts 7.
One second later, the server broadcasts 4, 5 to that client.

I think that was supported at some point but I forget if that's still okay or not. Today the per-document op gap recovery logic will not do that. But adding per-client op gap recovery logic may result in that sequence of events.

That's right. I think both scenarios can happen (the one I listed and yours).  The main issue is what you described:

So if PUSH wants to fill in gaps, in needs to detect gap between very first op it sent and last op in initialMessages and attempt to fill it in.

Push should be able to detect that and fetch the gap ops and send them (per-client). I am going to try testing this in spdf this week.

awesome, it would be great it this works, and doesn't necessitate any protocol changes. This still leaves the issue of client feedback in general if push is throttled for whatever reason. I'm really curious if @tanviraumi or someone he knows had to solve a similar issue around their throttling design. Does it just used NACKs? or are there other mechanisms?

I briefly talked to Gary and suggested authoring a document describing current behavior of PUSH (in respect to interactions to redis, kafka, etc., as well as document and client state tracking and processes involved around filling in ops gaps).
As for next steps, client would need to make a change to not immediately hit storage on op gap as PUSH will fill in a gap in next 0-20ms (with this new design), but ops coming out of order today result in immediate request to storage, which will be unnecessary with these changes. We already have enough telemetry to notice when we do unneeded storage calls, as well as cases where client can't fill in op gap in reasonable amount of time (30 seconds), so PUSH changes in this space could be validated by observing client telemetry.
The staging will look like this:

PUSH implements logic to fill in the gaps
Client implements delaying calling storage when gap is detected, in anticipation of PUSH filling in the gap in next N ms.
We validate via telemetry that client never does unneeded requests to storage (i.e. if client does request ops from storage to fill in the gap, then ops are always actually in storage). This tells us that # 1 is actually working and that threshold we chose is reasonable (it's probabilistic, so the goal of # 2 is reduce, but not completely eliminate calls to storage).
PUSH stops flushing ops to storage on client connection.

if there is a way to flight this change, we should, and validate it in our stress tests (likely by client having local change that provides some hint on connection to enable that behavior)
Validation here includes also ensuring that client never fails due to not being able to find ops anywhere for 30 seconds.

Based on discussion at PUSH review meeting:
PUSH has a lot of logic to resolve various issues with reliability of op delivery and ordering guarantees. And even after set of latest fixes, ops are not ordered all the time, resulting in client needing # 2 from above.
It feels like we need to move further to one of the extremes:

PUSH continues on this path and provides strong ordering guarantees, ensuring that all ops are delivered in order at all times, including initial ops on connection (this is almost the case, but not guaranteed, ops can come out of order).
We remove most of the logic that helps with above from PUSH and instead force client (driver) to fetch ops from PUSH through getOps type of API (latest maybe moved to be brokered by SPO).

Solution is described at the beginning of this issue.
The biggest con of that approach is additional network latency. So feasibility of that path depends on how often it happens.

The main telemetry event that is telling is fluid:telemetry:DeltaManager:enqueueMessages event (reason & previousReason properties).
Query:

Office_Fluid_FluidRuntime_Performance
| where Data_eventName == "fluid:telemetry:DeltaManager:enqueueMessages"
| summarize count(), countif(Data_initialGap > 0), countif(Data_gap > 0), countif(Data_duplicate > 0) by Data_eventName, Data_reason, Data_previousReason
| order by count_

First, scale - appropriate set (limiting only to 0.4x loader versions as that's when this event was added) corresponds to

205K container sessions
563K (re)connections.

Terms mean:

ReconnectOps means processing initial ops on some reconnect (not first connection for this container).
InitialOps - same as above, but very first connection in container.
DocumentOpen_pending means we fetched ops from storage on document open, but there was some gap or duplciate
opHandler - op is coming from socket
reason (one of the above) - where op came from when we detected gap or duplicate ops
previousReason - what was the source (reason) of previously successfully processed op (before gap/duplication occured)

What this data tells us:

Ranges do not have that many gaps. I.e. there are cases where initial ops as observed by DeltaManager would have a gap due to establishing early op handler and potential for ops to trickle in while we wait for "connect_document_success" message to bring initial ops - such early ops are merged into initial ops set by driver. So number of gaps for reason == "InitialOps" or reason =="ReconnectOps" is at 300.
There are a lot of duplicates in initial ops (ReconnectOps) and ops coming from web socket (on previous connection), i.e. there are a ton of fast reconnects where initialOps bring same content that client saw before (over 30K cases)
Most interesting is initialGap property when reason == "opHandler", i.e. when we consume an op from socket and realize there is a gap. Here are the top offenders:

previousReason       count
opHandler	     10066
ReconnectOps	     6736
DocumentOpen_fetch   751

So in 10K cases we do see a gap in op stream itself (that's 2% of connections). I believe this is the issue with lost redis connection by front-end.
And in 6K cases (1.2% of connections) we see a gap between first op and initial ops on (presumably) same connection.

And here is Gary's description of current system:
To recap the 2 things causing the gaps:

the fact that Push broadcasts to clients before inserting the op into redis. this results in ops are sometimes not accessible to newly joining clients
redis pubsub issues, so frontends sometimes lose ops

And the workarounds:

Push posting ops to spo whenver a client joins. This results in the client asking SPO for ops after joining and it gets the successfully
Added extra logic to detect these gaps, fetch the ops from redis, and broadcast them to the client.

With my change in SPDF, both are now fixed with the same logic.

The frontend detects gaps on a per document level and sends the ops to the clients
The frontend detects gaps on a per client level and sends the op to the client (the gap ops may be sent up to 5 seconds later after detection, so the client needs to handle non-sequential ops)

@GaryWilber, some observations based on my prototyping (in priority order):

"If no ops are flushed, it returns a 204.". This is not very useful as it does not tell client the latest sequence number that SPO has. So client has no ability to make decision on what to do next (in situation where it needs to ensure that certain op made it to SPO in order to proceed with summarizaiton).
It is very useful for IConnected response to have supportedFeatures property listing all optional features like "api_flush_ops" & "api_get_ops" such that client can make decision on whether to use them, and whether to assert if it does not get proper response from server when such features are supported (i.e. defense in depth). For example, client for most part ensures that it will not have overlapping ops requests, so it could assert that responses indeed come back before next request shows up. Currently I can't do it as I do not know if particular PUSH box supports that feature.
Payloads do not have enough context on multiplexed socket for clients to understand if given response is applicable to given document. Similar, client can't tell if it's response for its request unless it tracks all nonces. In both cases it's possible to format nonce in a way where it has that data (i.e. it is formed in format of documentId-clientId-randomId, but it would be great for responses in both cases to contain documentId & clientId.

@vladsud Great observations.

If no ops are flushed, it would hopefully mean that the latest op the client processed is already in SPO. However we know that ops are broadcasted before being inserted into Redis so that may not be the case.

This is a bit tricky to solve because Push does not explicitly track the last persisted op to SPO. It is tracked by the deli lambda but it's not exposed to clients and it's not exposed in a way that's easy for me to grab during the flush_ops call. Is this a blocker for the single stage post summary work?

That does sound useful. For returning supportedFeatures, would this work:
export type FluidSupportedFeatures = "api_get_ops" | "api_flush_ops"
and add supportedFeatures?: FluidSupportedFeatures[]; to IConnected.

I will add tenantId, documentId & clientId to the responses.

For # 1, I'll go for now with assumption that "all ops are flushed", but if we see SPO rejecting summary due to lack of ops in such cases, we might need to revisit it.
I'd prefer to go with connected.supportedFeatures = { api_get_ops: true, api_flush_ops: true };
Mostly to allow more extensible format (i.e. maybe some future features would want to communicate level of support, or some other properties).

@vladsud

retryAfter will be there. Let me know if it's not.
Yes it's because the flushes are async between the two. The flushes clients run can happen at the same time as the ones Push run. The client initiated flushes do not run through the same queue system so they can happen concurrently.
I looked at the 409s. From what I see, every single conflict was from Push posting ops due to client joins at the same time. So once we stop posting ops when clients joins, you should see a large reduction in 409s

Everything covered in this issue has been implemented, including get_ops & flush_ops flows.
Next step - we should start re-evaluating places where we added complexity on PUSH side that are no longer required with get_ops flow.
Specifically:

We can deprecate initialMessages flow - see https://github.com/microsoft/FluidFramework/issues/7132
PUSH can stop flushing ops to SPO on client connection
We should evaluate any other places where PUSH can be simplified.

@GaryWilber - if you have any PBIs tracking any future work in this domain, it would be great to list them.
Otherwise I'm, not aware of any work on client in this domain, so closing it as fixed.


React Native: Avoid adding px to unit values

What
Removes px units when using with React Native.
Why
React Native does not accept px as unit values.

How
Based on this comment from #142, I've just added an environment check on the px function and if the user is using React Native, then return without adding the px suffix.

Codecov Report

Merging #279 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #279   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           5      5
  Lines         183    184    +1
=====================================
+ Hits          183    184    +1

Impacted Files
Coverage Δ

src/util.js
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72f2683...53a6981. Read the comment docs.

Thanks! I don't think I'd want to add conditional logic in the runtime for something like this. Would you be willing to look into a babel-based solution? i.e. something where a build is compiled during publish to support this?

I've never implemented a babel solution that solves that issue, do you have any example for that so I can use it and adapt?

I don't know of anything off the top of my head. It would be similar to how you've implemented it, but change the code to either include or not include the 'px' string concatenation during the prepare step – I think a good first step would be to search for a babel plugin that can pre-evaluate the code in the px function based on the environment

Got it. Still, I have the doubts of why you don't want to include 1 line of code that can help a lot of developers that may be experiencing the same issue I have on the future?

It's a trade-off. I do want this to be compatible with RN and useful to those developers, but I don't want added complexity in the runtime part of the codebase to support things like this. The px might be the only thing that needs changed, but there's also a lot of code in styled-system for media queries that is completely unnecessary for RN, and I'd rather have something set up to support changes like that when needed so that different platforms have more optimized/custom tailored options


Support pytest 6

ERROR: pytest-homeassistant 0.1.1 has requirement pytest<6,>=5.4.0, but you'll have pytest 6.0.1 which is incompatible.

It would be great if pytest 6 would be supported. Otherwise, pytest should be pinned on lower than 6 in https://github.com/boralyl/github-custom-component-tutorial/blob/master/requirements.test.txt.

Thanks for creating the issue!  I pinned pytest in my other project.  I haven't had any time to look into what's new and if there are any backwards incompatibilities with 6.0.  When I have time I'll look into this.  Excited to see what's new!

It looks like there are no breaking changes that affect this project. (https://docs.pytest.org/en/latest/changelog.html)
Warnings of type PytestDeprecationWarning will now be throwns as errors, however I didn't face them during testing.
I pinned my version also on lower than <6, but it looks like this one could be updated easily.

Thanks for the heads up.  I'll try upgrading pytest on a few of my custom components with tests to ensure everything looks good.  If so I can loosen the pins.

v0.1.3 now allows pytest up to version 6.0.x


NerdTree won't open if there is ~ in the folder name

Environment:

Operating System: Mac
Vim version :version: 8.0.#
NERDTree version git rev-parse --short HEAD: HEAD
NERDTree settings applied in your vimrc, if any: N/A

Description:

Mac's iCloud Drive has ~ in their directory name causing Nerdtree not to open properly.

Process:
cd /Users/rickyc/Library/Mobile Documents/com~apple~CloudDocs/
open up vim and open nerdtree

Current Result:
Error detected while processing function 184[2]..185[27]..188[7]..151[9]..149[31]..262[4]..NERDTreeGitStatusRefreshListener[5]..NERDTreeGetGitStatusPrefix:
line   11:
E33: No previous substitute regular expression
E33: No previous substitute regular expression
Error detected while processing function 184[2]..185[27]..188[7]..151:
line    9:
E171: Missing :endif

😿
Expected Result:

Nerdtree should toggle open.

Just dug a bit deeper, wrong plugin!


Merge development into main

coverage: 81.053% (+6.1%) from 74.968% when pulling 499b80be5637ae05d390547c78d3e3d8c6a5ab56 on development into 9abeac6d79e0667fc0799f3f314f7290f29829c1 on main.


Nested keys required?

Hello, with React query i naturally have this scenario:
-list of customers by account  filtered by order by
how i map this to cache keys is:
`function customersByAccountId(accountId: number) {
return ["customerListByAccountId", accountId];
}
function customersByAccountIdFiltered(accountId: number, queryFilter: QueryFilter) {
return [...customersByAccountId(accountId), "list", { queryFilter }];
}`
This way, when i edit an attribute for a customer, i can:
update the in memory query for customersByAccountIdFiltered (which will change the attribute on the current screen, without reload)
and invalidate customersByAccountId (with refetchActive: false,) which will invalidate all other cached filtered entries.
How do i map this with query-key-factory?

@kimbuba can you share a little more context about your use case? especially about:

and invalidate customersByAccountId (with refetchActive: false,) which will invalidate all other cached filtered entries.

I can see ["customerListByAccountId", accountId, 'list'] being enough to invalidade the "other cached filtered entries", so I want to understand a little more about how you use ["customerListByAccountId", accountId] connected to the ["customerListByAccountId", accountId, 'list'] queries, so that I can give you a proper solution or really understand if that's a use-case that the lib does not yet support

Hello Luke!
Yes sure.
I need to support the following keys structure
["customerListByAccountId", accountId, 'list', {orderBy:"create_date"}]
["customerListByAccountId", accountId, 'list', {orderBy:"id"}]
["customerListByAccountId", accountId, 'list2', {orderBy:"create_date"}]
how do i clean this using your library? accountId is a variable number.
Basically i need to do the following, clean:
["customerListByAccountId", accountId]
how do you translate it with createQueryKeyStore?

What's the difference between list and list2? Is that also a dynamic parameter? Maybe pagination?

thanks, i see.
Would it be possible to go more deeper in the hierarchy?
Say:
customer/10/stakeholders/213/load
that would easily translate into
[customer,10,stakeholders,213,load]
so i could easily invalidate customerId 10 and everything under it with this key [customer,10]

that would be to support a hierarchy structure.
But i'm trying to adapt my current structure to your library.
Btw is this
queryClient.invalidateQueries([...store.customers.accountId._def, accountId], { refetchActive: false });
actually
queryClient.invalidateQueries([...store.customers. listByAccountId._def, accountId], { refetchActive: false });
or am i missing something?

that would be to support a hierarchy structure.

Basically scoping some keys to an entity ID, or any other use cases?  I remember thinking something like this when designing this first api but couldn't think about any examples at the time. So, as much information for use cases like these as I can get will help me think about how to support it

Btw is this
queryClient.invalidateQueries([...store.customers.accountId._def, accountId], { refetchActive: false });
actually
queryClient.invalidateQueries([...store.customers. listByAccountId._def, accountId], { refetchActive: false });

Yes you're correct, I totally miswrote that 😓

np,
here is another example:
I have a list of accounts and each account has customers
i want to do different lists per customers, as they are calling different endpoints.
I could put everything as in a filter for this simple example but i may not want to.
API:
- customersListByAccountInvited(1, "desc");
- customersListByAccountNotInvited(2, "asc");

The first gives the list of all customers non invited ordered by desc.
The second gives the list of all customers non invited ordered by asc.
This would naturally map as: ["customers",1,"invited", {sort:"desc"}]
And: ["customers",1,"non_invited", {sort:"asc"}]
With 1, the accountId
Now say i'm adding a customer to the first cache.
I would setQuery on["customers",1,"invited", {sort:"desc"}]and add a user "John"
I now need to invalidate the following
queryClient.invalidateQueries(["customers",1], { refetchActive: false });
This way:

The current list has updated data without refetching
The second list is invalidated and will be reloaded

Is there a way to map this?
Something like this would not give me what i want to accomplish:
customers: {
    listInvited: (accountId: number, sort: {"asc"|"desc"}) => [accountId, { sort }],
    listNonInvited: (accountId: number, sort: {"asc"|"desc"}) => [accountId, { sort }],
  },

As there would be no way to invalidate
queryClient.invalidateQueries(["customers",1], { refetchActive: false });

Thanks for the information @kimbuba, I'll start some exploration towards that use case

@kimbuba just letting you know, v1.0 has been released with support for nested keys: https://github.com/lukemorales/query-key-factory/releases/tag/v1.0.0


joystick: add a joystick naming compatibility hint and API call

Starting with 2.0.12, SDL aplies some normalization to the joystick names, which results in the naming being different from what Linux's event/udev APIs report.
This causes issues for the EmulationStation's input mapping scripts, which assume the names (SDL vs. Linux) are identical (see for instance RetroPie-Setup#3398.
The PR adds 2 commits:

introduce a hint for SDL to use the old/pre-2.0.12) compatible joystick naming (SDL_HINT_JOYSTICK_USE_OLD_NAME, 0=false, 1=true).  By default, the hint is set to true (i.e. the compatible naming is used by default).
add a SDL_JoystickCompatName function that will always return the pre-2.0.12 joystick name, regardless of the value of the hint above. This helps application retrieve the previously used joystick name.

OK, fixed a build failure on non-Linux, should be ok.
I've modified test/testjoystick.c to show both the new and old names during testing:
Old naming :
SDL_JOYSTICK_USE_OLD_NAME=1 ./testjoystick
INFO: Joystick device 0 added.
INFO: Joystick
INFO:        name: 'USB,2-axis 8-button gamepad  '
INFO:    old name: 'USB,2-axis 8-button gamepad  '
INFO:        type: Game Controller
INFO:        axes: 2
INFO:       balls: 0
INFO:        hats: 0
INFO:     buttons: 8
INFO: instance id: 0
INFO:        guid:<PHONE_NUMBER>0500006020000010010000
INFO:     VID/PID: 0x0583/0x2060

New/Upstream naming :
SDL_JOYSTICK_USE_OLD_NAME=0 ./testjoystick
INFO: Joystick device 0 added.
INFO: Joystick
INFO:        name: 'USB,2-axis 8-button gamepad'
INFO:    old name: 'USB,2-axis 8-button gamepad  '
INFO:        type: Game Controller
INFO:        axes: 2
INFO:       balls: 0
INFO:        hats: 0
INFO:     buttons: 8
INFO: instance id: 0
INFO:        guid:<PHONE_NUMBER>0500006020000010010000
INFO:     VID/PID: 0x0583/0x2060

Some notes on the impact of the SDL_JoystickName and the changes in the PR:

the SDL Gamecontroller subsystem shouldn't be impacted, the mapping db uses the device's GUID.
older ES configs would also not be affected on upgrade when using the 'new' names, since the device's GUID is used for identification.
the retroarch configs with new names would still work if the vendor/product IDs are added to the generated .cfg. We could update ES to include the PID/VID and adjust the input configuration scripts to include it. Other input mapping scripts may need also adjusting to see which are using SDL or 'Linux' names (e.g. mupen64plus , reicast etc.)

I'm thinking the second patch is going to a pain to maintain as this code may change a lot in the future.
The patch needs some work for 2.0.18 due to the addition of FreeJoylistItem / RemoveJoylistItem although FreeJoylistItem would enable less need for the additional frees.
Why do we need a function to get the compatible name ? I wonder if really it's needed vs the maintainability.

Why do we need a function to get the compatible name ? I wonder if really it's needed vs the maintainability.

The reasoning was that we could switch off the compatibility hint and use the new SDL names, but we'd need the old/compat name for:

migrating existing ES configuration, when the SDL version changes with the new default
obtaining the udev compatible name for auto-configuration scripts which need it (i.e. RetroArch)

total noob learning a lot here.  is there  a script you run to make these changes? or do they have to modified individually? if there is where do you run it ? lol terminal? or ES environment?


[BUG] UTF-8 Character Encoding Issues in opensearchproject/data-prepper container

Bug Description
opensearchproject/data-prepper container image incorrectly handles UTF-8 characters when streaming data from DynamoDB to S3 buckets in NDJSON format. Non-ASCII characters are replaced with question marks (?) in the output files.
Steps to Reproduce

Set up data-prepper using the opensearchproject/data-prepper container image
Create a DynamoDB table with items containing strings with non-ASCII characters (e.g., Mandarin, Tamil)
Configure data-prepper to stream changes from the DynamoDB table to an S3 bucket using NDJSON format
Observe the resulting S3 objects

Actual Behavior
All non-ASCII characters in the original DynamoDB data are replaced with question marks (?) in the S3 output files.
Expected Behavior
All UTF-8 characters, including non-ASCII characters, should be preserved in the output NDJSON files exactly as they appear in the source DynamoDB table.
Workaround
Adding the environment variable LC_ALL=C.UTF-8 to the container configuration resolves the issue. This environment variable should be set by default in the container image to ensure proper UTF-8 handling.

We have had similar issues with the DynamoDB source and control characters. These might be related: #5027


Upgrading kubb/cli from 1.6.1 to 1.6.2 presents error about requring @swc/core or @swc/wasm

What version of kubb is running?
1.6.2
What platform is your computer?
Windows
What steps can reproduce the bug?
Upgrade @kubb/cli from 1.6.1 to 1.6.2
Receive error on execution

> kubb generate --config config.ts

✖ swc compiler requires either @swc/core or @swc/wasm to be installed as a dependency.
See https://typestrong.org/ts-node/docs/transpilers
  Error: swc compiler requires either @swc/core or @swc/wasm to be installed as a dependency.
See https://typestrong.org/ts-node/docs/t  ranspilers

kubb/cli  1.6.1

Reverting to 1.6.1 resolves the issue.
What is the expected behavior?
No response
Swagger/OpenAPI file?
No response
Additional information
Diffing the commits for 1.6.2:
In packages/cli/package.json, it appears the below dependency was removed (perhaps related)
"dependencies"
"@swc/core": "^1.3.80",

@jafin Oeps, removed it by accident, I will release a new version :)

@stijnvanhulle thanks for the prompt fix, 1.6.3 resolved the issue


Provide friendly error message for db connection failure

Just a little message to help others avoid confusion like #9

Thanks :)

Thank you for following through on this auth experiment. I think it's going to be the missing piece to add basic auth to database-driven APIs. You've thought a lot of things through and have features for real-world user management. Actually I'd be honored if you would join the postgrest chat room where we discuss using pure sql to build APIs.


Fix command errors in Nebari server run for templates

Description
This PR is a follow up PR for the fixes in the code execution while testing in the Nebari server. Particularly we faced two issues, first if we open a template in Nebari, its default folder will be root and if it is the second template opened, then it will try to download and overwrite the previous one. This can be fixed by fixing the nbuid-> nbuid + '-nebari'.
Also we add some extra commands that can help in creating a new directory and store templates files in that specific directory.
Fix #314

Additional context
This issue was discussed in Discord this week.
What is the purpose of this pull request?

[x] Bug fix
[ ] New feature
[ ] Other

Hey @vfdev-5 seems to work as expected now, can you check on your server as well once?

Is it expected to have this ? How did you test it ?
@guptaaryan16 this does not work!

Is it expected to have this ? How did you test it ?
@guptaaryan16 this does not work!

Actually for other configurations like text-segmentation, it seems to work, I guess we need to delete some previous notebooks generated for nebari.js like the one you mentioned

Actually for other configurations like text-segmentation, it seems to work, I guess we need to delete some previous notebooks generated for nebari.js like the one you mentioned

It's very annoying that I have to discover during the review. You could have tested the first template at least

It's very annoying that I have to discover during the review. You could have tested the first template at least

Actually I was testing some solutions for this problem, and it seems like updating nbuid-> nbuid + '-nebari' + <last_commit_hash> may work. Let me test this and then we can merge this PR.

Actually I was testing some solutions for this problem, and it seems like updating nbuid-> nbuid + '-nebari' + <last_commit_hash> may work. Let me test this and then we can merge this PR

Commit suffix is not related to nebari option but have to be considered in general. But in general appending a commit is not a good solution as we would generate new uuid everytime someone pushes to main (even unrelated change).
We may want distinguish cases when the app is run from netlify PR deployement vs from code-generator.pytorch-ignite.ai/ and append commit from PR only, IMO

Commit suffix is not related to nebari option but have to be considered in general. But in general appending a commit is not a good solution as we would generate new uuid everytime someone pushes to main (even unrelated change).
We may want distinguish cases when the app is run from netlify PR deployement vs from code-generator.pytorch-ignite.ai/ and append commit from PR only, IMO

Let me revert the changes for now and make a second PR for this deployment vs PR build case