Ping test s4 fix

This commit is not generic enough but in order to not make too many changes
not critical for S4 to work, it is being pushed as is.
Next test (S5) will do all the cleanup.

Codecov Report

Merging #629 (45e2a99) into master (5c071c7) will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #629      +/-   ##
============================================
+ Coverage     31.91%   31.95%   +0.04%
- Complexity     1249     1256       +7
============================================
  Files           525      525
  Lines         13491    13491
  Branches       1669     1669
============================================
+ Hits           4305     4311       +6
+ Misses         8598     8596       -2
+ Partials        588      584       -4

Impacted Files
Coverage Δ

...lcor/portmanager/processor/DataPlaneProcessor.java
70.27% <0.00%> (+0.54%)
:arrow_up:

...rewei/alcor/portmanager/processor/PortContext.java
71.42% <0.00%> (+3.89%)
:arrow_up:

...i/alcor/portmanager/processor/RouterProcessor.java
85.29% <0.00%> (+5.88%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c071c7...45e2a99. Read the comment docs.


mpi_cmd

MPI execution now controlled by MPI_PREP_CMD and MPI_CMD.
MPI_PREP_CMD: command necessary to prepare for running MPI (loading of environment variables or somesuch)
MPI_CMD : command to execute MPI

per discussion on other pull request and in standup meeting, merging into ml_structure_s13


Uncaught exception when using sink and fopen, closing resource before saving it

Guzzle version(s) affected: 6.3-dev
PHP version: 7.3.14
cURL version: 7.66.0
Description
Getting Uncaught Exception: Invalid resource type: resource (closed) when trying to save the response of a request directly to a file, one of the issues is that it only happens some times, this code works fine for most of the time.
The resource(closed)  should be the gettype only after fclose() so I am not sure what is going on.
How to reproduce
        $client = new Client();
        $fn      = tempnam(sys_get_temp_dir(), 'some-prefix-');
        $fp      = fopen($fn, 'r+');
        $res = $client->request('GET', $url, ['sink' => $fp]);
        $stream = $res->getBody()->detach();
        $stat   = fstat($stream);
        fclose($fp);

Possible Solution
Maybe it is related to something asycronous happening?
Additional context
This is a code I have to upload files directly from a URL

I don't think this is a bug in Guzzle. It's a bug in your code's 3rd line. You are not checking if fopen fails.

@GrahamCampbell cheers I will do that but having said that it works 99.9% of the time, can you foresee any reason why fopen wouldn't work here sporadically ? Also, if it fails what should be the correct handling? Just keep trying to open the file until it does? Seems odd

I get the "Invalid resource type: resource (closed)" message when using the 'sink' option and the requested URL returns a 301 redirect.

Can Guzzle throw more meaningful exception in this case?
I would actually consider this as a bug, because per contract \GuzzleHttp\ClientInterface::request can only throw GuzzleException not InvalidArgumentException.


New sortable methods

Including new sortable methods params to decide programatically if field will be sortable

@ChStark thanks for the PR, can you please add tests for it?


plex-mpv-shim not working

error message :
one running this command ; "plex-mpv-shim" i got this under writter message
2022-01-24 23:28:46,757 [    INFO] conf: Loaded settings from json: /home/tarun/.config/plex-mpv-shim/conf.json
Traceback (most recent call last):
File "/usr/local/bin/plex-mpv-shim", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/plex_mpv_shim/mpv_shim.py", line 47, in main
from .player import playerManager
File "/usr/local/lib/python3.6/dist-packages/plex_mpv_shim/player.py", line 23, in
import mpv
File "/usr/local/lib/python3.6/dist-packages/mpv.py", line 531, in
_handle_func('mpv_detach_destroy',          [],                                         None, errcheck=None)
File "/usr/local/lib/python3.6/dist-packages/mpv.py", line 478, in _handle_func
func = getattr(backend, name)
File "/usr/lib/python3.6/ctypes/init.py", line 361, in getattr
func = self.getitem(name)
File "/usr/lib/python3.6/ctypes/init.py", line 366, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/local/lib/libmpv.so.2: undefined symbol: mpv_detach_destroy
System configuration:
ubuntu: 18.04
ram 8gb
Installing steps:
I followed all the steps mentioned in  Documentation for installation

sudo pip3 install --upgrade plex-mpv-shim
sudo pip3 install pystray
sudo apt install python3-tk
sudo pip3 install --upgrade python-mpv
sudo apt install autoconf automake libtool libharfbuzz-dev libfreetype6-dev libfontconfig1-dev libx11-dev libxrandr-dev libvdpau-dev libva-dev mesa-common-dev libegl1-mesa-dev yasm libasound2-dev libpulse-dev libuchardet-dev zlib1g-dev libfribidi-dev git libgnutls28-dev libgl1-mesa-dev libsdl2-dev cmake wget python g++ libluajit-5.1-dev
git clone https://github.com/mpv-player/mpv-build.git
cd mpv-build
echo --enable-libmpv-shared > mpv_options
./rebuild -j4
sudo ./install
sudo ldconfig

The latest MPV master branch has had breakage with the libmpv1 library. Use the last stable version or switch to external MPV mode, which should not be affected.

See here for another fix: https://github.com/iwalton3/plex-mpv-shim/issues/72#issuecomment-1365374854


Long Out_channel shrinking runs cause timeout

Adding the missing bytes-involving bindings introduced a sister-issue to #373 albeit for Out_channel.
This was observed by @shym in https://github.com/ocaml-multicore/multicoretests/issues/373#issuecomment-1630881135
I've also observed it on #376:
For Cygwin trunk/5.2 part1
https://github.com/ocaml-multicore/multicoretests/actions/runs/5553343778/jobs/10141775211
random seed: 186839348
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain (generating)
[ ]   18    0    0   18 / 1000    62.9s Lin DSL In_channel test with Domain
[✓]   20    0    1   19 / 1000   107.3s Lin DSL In_channel test with Domain

[ ]    0    0    0    0 / 5000     0.0s Lin DSL Out_channel test with Domain
[ ]    4    0    0    4 / 5000    17.7s Lin DSL Out_channel test with Domain
[ ]   17    0    0   17 / 5000    89.4s Lin DSL Out_channel test with Domain
[ ]   30    0    0   30 / 5000   152.8s Lin DSL Out_channel test with Domain
[ ]   37    0    0   37 / 5000   213.0s Lin DSL Out_channel test with Domain (shrinking:   11)
[ ]   37    0    0   37 / 5000   275.9s Lin DSL Out_channel test with Domain (shrinking:   15.0005)
[ ]   37    0    0   37 / 5000   341.2s Lin DSL Out_channel test with Domain (shrinking:   18.0004)
[ ]   37    0    0   37 / 5000   407.6s Lin DSL Out_channel test with Domain (shrinking:   20.0004)
[ ]   37    0    0   37 / 5000   469.6s Lin DSL Out_channel test with Domain (shrinking:   21.0007)
[ ]   37    0    0   37 / 5000   533.5s Lin DSL Out_channel test with Domain (shrinking:   22.0007)
[ ]   37    0    0   37 / 5000   598.2s Lin DSL Out_channel test with Domain (shrinking:   23.0007)
[ ]   37    0    0   37 / 5000   663.0s Lin DSL Out_channel test with Domain (shrinking:   24.0006)
...
[ ]   37    0    0   37 / 5000  3898.2s Lin DSL Out_channel test with Domain (shrinking:   73.0009)
[ ]   37    0    0   37 / 5000  3963.1s Lin DSL Out_channel test with Domain (shrinking:   74.0008)
[ ]   37    0    0   37 / 5000  4023.3s Lin DSL Out_channel test with Domain (shrinking:   75.0007)
[ ]   37    0    0   37 / 5000  4085.3s Lin DSL Out_channel test with Domain (shrinking:   76.0007)
[ ]   37    0    0   37 / 5000  4147.7s Lin DSL Out_channel test with Domain (shrinking:   77.0007)
[ ]   37    0    0   37 / 5000  4210.4s Lin DSL Out_channel test with Domain (shrinking:   78.0007)
[ ]   37    0    0   37 / 5000  4276.4s Lin DSL Out_channel test with Domain (shrinking:   79.0006)
[ ]   37    0    0   37 / 5000  4339.0s Lin DSL Out_channel test with Domain (shrinking:   80.0005)
[ ]   37    0    0   37 / 5000  4399.3s Lin DSL Out_channel test with Domain (shrinking:   81.0004)
[ ]   37    0    0   37 / 5000  4466.2s Lin DSL Out_channel test with Domain (shrinking:   82.0004)
[ ]   37    0    0   37 / 5000  4528.5s Lin DSL Out_channel test with Domain (shrinking:   83.0003)
Error: The operation was canceled.

For Windows MingW 5.1
https://github.com/ocaml-multicore/multicoretests/actions/runs/5553343781/jobs/10141774703
random seed: 163892320
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain (generating)
[ ]    1    0    0    1 / 1000    63.0s Lin DSL In_channel test with Domain (shrinking:   29.0006)
[✓]    2    0    1    1 / 1000   108.9s Lin DSL In_channel test with Domain

[ ]    0    0    0    0 / 5000     0.0s Lin DSL Out_channel test with Domain
[ ]   11    0    0   11 / 5000    21.4s Lin DSL Out_channel test with Domain
[ ]   11    0    0   11 / 5000    86.6s Lin DSL Out_channel test with Domain (shrinking:    3.0004)
[ ]   11    0    0   11 / 5000   148.9s Lin DSL Out_channel test with Domain (shrinking:    5.0009)
[ ]   11    0    0   11 / 5000   212.4s Lin DSL Out_channel test with Domain (shrinking:    7.0010)
[ ]   11    0    0   11 / 5000   273.5s Lin DSL Out_channel test with Domain (shrinking:    8.0005)
[ ]   11    0    0   11 / 5000   336.8s Lin DSL Out_channel test with Domain (shrinking:    9.0011)
[ ]   11    0    0   11 / 5000   401.0s Lin DSL Out_channel test with Domain (shrinking:   12.0007)
....
[ ]   11    0    0   11 / 5000 10347.9s Lin DSL Out_channel test with Domain (shrinking:   87.0033)
[ ]   11    0    0   11 / 5000 10409.4s Lin DSL Out_channel test with Domain (shrinking:   88.0009)
[ ]   11    0    0   11 / 5000 10472.8s Lin DSL Out_channel test with Domain (shrinking:   88.0014)
[ ]   11    0    0   11 / 5000 10532.9s Lin DSL Out_channel test with Domain (shrinking:   89.0007)
[ ]   11    0    0   11 / 5000 10601.2s Lin DSL Out_channel test with Domain (shrinking:   89.0014)
[ ]   11    0    0   11 / 5000 10670.7s Lin DSL Out_channel test with Domain (shrinking:   90.0009)
[ ]   11    0    0   11 / 5000 10735.0s Lin DSL Out_channel test with Domain (shrinking:   90.0016)
[ ]   11    0    0   11 / 5000 10797.9s Lin DSL Out_channel test with Domain (shrinking:   91.0013)
[ ]   11    0    0   11 / 5000 10862.2s Lin DSL Out_channel test with Domain (shrinking:   91.0020)
[ ]   11    0    0   11 / 5000 10928.2s Lin DSL Out_channel test with Domain (shrinking:   91.0028)
[ ]   11    0    0   11 / 5000 10994.4s Lin DSL Out_channel test with Domain (shrinking:   92.0012)
[ ]   11    0    0   11 / 5000 11057.5s Lin DSL Out_channel test with Domain (shrinking:   93.0009)
[ ]   11    0    0   11 / 5000 11117.7s Lin DSL Out_channel test with Domain (shrinking:   93.0015)
[ ]   11    0    0   11 / 5000 11180.9s Lin DSL Out_channel test with Domain (shrinking:   94.0003)
[ ]   11    0    0   11 / 5000 11242.1s Lin DSL Out_channel test with Domain (shrinking:  102.0003)
[ ]   11    0    0   11 / 5000 11302.1s Lin DSL Out_channel test with Domain (shrinking:  108)
[✓]   12    0    1   11 / 5000 11331.5s Lin DSL Out_channel test with Domain

....
                                                              |
                                     Out_channel.output_substring t "bceY\145CQx\156" 1 6
                                                              |
                                   .-----------------------------------------------------.
                                   |                                                     |
                       Out_channel.close_noerr t                                Out_channel.length t

+++ Messages ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Messages for test Lin DSL Out_channel test with Domain:

  Results incompatible with sequential execution

                                                                   |
                                     Out_channel.output_substring t "bceY\145CQx\156" 1 6 : Ok (())
                                                                   |
                                   .---------------------------------------------------------------.
                                   |                                                               |
                     Out_channel.close_noerr t : ()                                  Out_channel.length t : Ok (6)

================================================================================
success (ran 2 tests)

Merging #376 into main triggered this - on Windows bytecode trunk/5.2:
https://github.com/ocaml-multicore/multicoretests/actions/runs/5558724000/jobs/10154146152
random seed: 519895651
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin DSL In_channel test with Domain (generating)
[✓]    2    0    1    1 / 1000    17.6s Lin DSL In_channel test with Domain

[ ]    0    0    0    0 / 5000     0.0s Lin DSL Out_channel test with Domain
[ ]   25    0    0   25 / 5000    43.8s Lin DSL Out_channel test with Domain
[ ]   39    0    0   39 / 5000   407.2s Lin DSL Out_channel test with Domain
[ ]   64    0    0   64 / 5000   475.1s Lin DSL Out_channel test with Domain
[ ]   90    0    0   90 / 5000   541.0s Lin DSL Out_channel test with Domain
[ ]  114    0    0  114 / 5000   601.7s Lin DSL Out_channel test with Domain
[ ]  125    0    0  125 / 5000   691.6s Lin DSL Out_channel test with Domain (shrinking:    0.0003)
[ ]  125    0    0  125 / 5000   765.6s Lin DSL Out_channel test with Domain (shrinking:    0.0005)
[ ]  125    0    0  125 / 5000   830.0s Lin DSL Out_channel test with Domain (shrinking:    0.0008)
[ ]  125    0    0  125 / 5000   892.9s Lin DSL Out_channel test with Domain (shrinking:    0.0010)
[ ]  125    0    0  125 / 5000   956.3s Lin DSL Out_channel test with Domain (shrinking:    0.0014)
[ ]  125    0    0  125 / 5000  1019.9s Lin DSL Out_channel test with Domain (shrinking:    0.0016)
[ ]  125    0    0  125 / 5000  1086.4s Lin DSL Out_channel test with Domain (shrinking:    0.0019)
[ ]  125    0    0  125 / 5000  1168.6s Lin DSL Out_channel test with Domain (shrinking:    0.0022)
[ ]  125    0    0  125 / 5000  1257.4s Lin DSL Out_channel test with Domain (shrinking:    0.0025)
[ ]  125    0    0  125 / 5000  1342.9s Lin DSL Out_channel test with Domain (shrinking:    0.0027)
[ ]  125    0    0  125 / 5000  1418.2s Lin DSL Out_channel test with Domain (shrinking:    0.0029)
[ ]  125    0    0  125 / 5000  1494.0s Lin DSL Out_channel test with Domain (shrinking:    0.0031)
[ ]  125    0    0  125 / 5000  1571.8s Lin DSL Out_channel test with Domain (shrinking:    0.0033)
...
[ ]  125    0    0  125 / 5000  7034.0s Lin DSL Out_channel test with Domain (shrinking:   13.0057)
[ ]  125    0    0  125 / 5000  7098.6s Lin DSL Out_channel test with Domain (shrinking:   14.0003)
[ ]  125    0    0  125 / 5000  7166.9s Lin DSL Out_channel test with Domain (shrinking:   14.0008)
[ ]  125    0    0  125 / 5000  7238.2s Lin DSL Out_channel test with Domain (shrinking:   14.0014)
[ ]  125    0    0  125 / 5000  7301.6s Lin DSL Out_channel test with Domain (shrinking:   14.0018)
[ ]  125    0    0  125 / 5000  7383.7s Lin DSL Out_channel test with Domain (shrinking:   14.0021)
[ ]  125    0    0  125 / 5000  7464.1s Lin DSL Out_channel test with Domain (shrinking:   14.0024)
[ ]  125    0    0  125 / 5000  7539.0s Lin DSL Out_channel test with Domain (shrinking:   14.0027)
[ ]  125    0    0  125 / 5000  7627.5s Lin DSL Out_channel test with Domain (shrinking:   14.0030)
Fatal error: exception User interruption
[ ]  125    0    0  125 / 5000  7690.1s Lin DSL Out_channel test with Domain (shrinking:   16.0002)Terminate batch job (Y/N)?
^C
Error: The operation was canceled.

I'm adding the ocaml5-issue label since this seems be caused by a GC performance regression in 5.1.
In fact, we have never observed this on 5.0 - only on 5.1 and trunk.
Locally my previous perf top testing further shows that the caml_alloc_custom_mem PR https://github.com/ocaml/ocaml/pull/12318 improves GC times reported by perf top to something like the following:
8.98%  lin_tests_dsl_domain.exe  [.] pool_sweep
1.75%  lin_tests_dsl_domain.exe  [.] do_some_marking

which is a significant improvement over pre-12318 (and pre-#387) observations:
51.46%  lin_tests_dsl_domain.exe            [.] pool_sweep
12.13%  lin_tests_dsl_domain.exe            [.] do_some_marking

With 12318 merged into OCaml trunk I don't expect us to see this issue triggered on trunk anymore.


Unify preds, target input arguments for text metrics [6of12]

What does this PR do?
This PR unifies preds, target input arguments for text metrics + also unifies some minor related stuff
Part of #716
Fixes:

[x] bert
[x] bleu
[x] chrf
[x] sacre_bleu
[x] wip
[x] wil

Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure to update the docs?

PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

@stancld made the metrics as a checkbox also here so you can check progress and could you also update the original issue?

still WIP? :rabbit:

still WIP? 🐰 fyi, #727 (review)

Yes a bit, I'll need to handle breaking changes add some deprecation warnings as you pointed out in #727

still need warnings and back compatibility 🐰

Currently adding these... Though, still thinking about if we shouldn't go with None both for predictions and preds. 🤔

@stancld is it ready? :]

@stancld is it ready? :]

@Borda I need to find out the way of how to comply with mypy when preds and predictions are both None by default.. 🤔

@stancld is it ready? :]

@Borda I need to find out the way of how to comply with mypy when preds and predictions are both None by default.. thinking

I would set the line as ignore and add a todo note to remove the ignore after we remove this deprecation
there is no need to waste too much tie on finding a solution for something which is not critical and also just for one release :ice_cream:

@stancld is it ready? :]

@Borda I need to find out the way of how to comply with mypy when preds and predictions are both None by default.. thinking

I would set the line as ignore and add a todo note to remove the ignore after we remove this deprecation there is no need to waste too much tie on finding a solution for something which is not critical and also just for one release 🍨

I was afraid of suggesting that :D But I agree it's not worthy to overcomplicate it for now. I'll add ignore statements and open an issue so that we will remove these statements afterwards.

just the warning stream, otherwise it looks good

@Borda Actually, I don't know why, but I don't get any warnings with a default stream value 🤔


Cross-request state pollution with Nuxtjs SSR

Describe the bug
First of all, thank you for this awesome library.
The bug seems to be some kind of cross-request state pollution when using Vuex-ORM on Nuxtjs with SSR. Its kinda hard to reproduce. I made an example with a fresh Nuxt project repository and also catch the behavior on video (see Additional context below).
Steps to reproduce the bug

Clone repo
Install dependencies (yarn install)
Build project (yarn build)
Open 2-3 browser windows each one using a different slug.

For example:

http://localhost:3000/question/first-question
http://localhost:3000/question/second-question
http://localhost:3000/question/third-question

Reload the page of each window one by one before they all finished loading to simulate multiple requests going on simultaneously.

Expected behaviour
Expected all browser windows to have their respective title related to the slug in the url parameter.
Versions

Vuex ORM: 0.36.3
Nuxt: 2.15.3

Link to minimal reproduction
Repo: https://github.com/jayrchamp/vuex-orm-nuxtjs-cross-request-state-pollution
Additional context
Video: https://storage.googleapis.com/ayourp-examples/vuex-orm-nuxtjs-cross-request-state-pollution.mp4

Got the same issue. Have you installed Vuex ORM for Nuxt correctly? https://vuex-orm.org/guide/model/retrieving-models.html#import-statement + https://vuex-orm.org/guide/digging-deeper/server-side-rendering.html
However, correct installation did not helped me. I solved it the following way.
Instead of
// data/database.ts
import VuexORM from '@vuex-orm/core';
import ... // import models

const database = new VuexORM.Database();
database.register(...); // lot of my models

export default database;

I wrote:
// data/database.ts
import VuexORM from '@vuex-orm/core';
import ... // import models

export default function() {
    const database = new VuexORM.Database();
    database.register(...); // lot of my models

    return database;
}

And in plugin file:
// plugins/vuex-orm.ts
import VuexORM from '@vuex-orm/core';
import database from '~/data/database';

export default ({ store }: any) => {
    VuexORM.install(database())(store);
};

@ArtemGrachov Thanks for sharing your solution. I tried it on the minimal repo and it seems to work fine.
However, in my case, I have already replaced VuexOrm with another solution to make sure that I don't end up finding other major issues like this one in the future.
Hope your solution helps others!

I had to use the option fetchOnServer: false (Nuxt docs) to circumvent this problem.
I guess it's not really helpful if you want a pure SSR and fully optimized SEO experience, but I preferred implementing this over clients getting polluted or semi-broken pages when they open a lot of tabs at once.


Registering the package

@JuliaRegistrator register()

Do you anticipate to provide an interface for other compression libraries as well? Because at the moment the ZfpCompression.jl and our interface is hardcoded in (which is fine!) but then I would suggest to maybe rename this package to SequentialZfpCompression.jl? Otherwise users might be confused that it's not about any compression?

@JuliaRegistrator register()

also you have to write register without ()

Hey @milankl  I created another repo to avoid breaking the link to this one from the juliacon schedule. Here it is https://github.com/AtilaSaraiva/SequentialZfpCompression.jl. I am archiving this one.


Dvo docs

Description
Add OpenAPI spec + additional documentation regarding new REST API endpoints (1, 2)
Fixes # (issue)
Type of change
Please delete options that are not relevant.

Documentation update

Testing steps
n/a
Checklist

[x] make before_commit passes
[x] updated documentation wherever necessary
[x] added or modified tests if necessary
[x] updated schemas and validators in insights-data-schemas in case of input/output change

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b26cc43) 69.76% compared to head (c745779) 69.76%.

:exclamation: Current head c745779 differs from pull request most recent head 2b521c1. Consider uploading reports for the commit 2b521c1 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1180   +/-   ##
=======================================
  Coverage   69.76%   69.76%
=======================================
  Files          25       25
  Lines        3684     3684
=======================================
  Hits         2570     2570
  Misses        852      852
  Partials      262      262

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

/retest


Fixing ico not being exported

merging to fix issue on water-use

Coverage remained the same at 80.818% when pulling 67a17fff657e1737d15948d4e2fa5f6eb0021022 on jiwalker-usgs:cachebreaker into ccc3a0086efbba10fac1bc58adb076e6ad0659a2 on USGS-VIZLAB:master.


TST: adjust to vmImage dispatch in Azure

Try to fix the Azure image dispatch problem based on Microsoft feedback from @ericsciple

I may self-merge this in a few hours if no objections -- it is a typo fix that alleviates our failing Azure CI & confirmed by a few Microsoft engineers.


Update Intercept requests to localhost page

Raised from #837
Update example with --proxy-server option as it is required when using --proxy-bypass-list

--proxy-bypass-list=(<trailing_domain>|)[:][;...]
Tells Microsoft Edge to bypass any specified proxy for the specified semicolon-separated list of hosts. This flag must be used with --proxy-server.

Source: https://learn.microsoft.com/DeployEdge/edge-learnmore-cmdline-options-proxy-settings#command-line-options-for-proxy-settings
Add Firefox example

I can get Firefox to route through devproxy by setting the proxy server to <IP_ADDRESS>:8000 and, in about:config changing the network.proxy.allow_hijacking_localhost setting to true.

https://github.com/MicrosoftDocs/microsoft-cloud-pr/pull/332


Movement

export function setPlayerMovment(k, player) {
player.onUpdate(() => {
const directionVector = k.vec2(0, 0)
    if (k.isKeyDown("left")) {
        player.flipX = true;
        player.direction = "left"
        directionVector.x = -1
        playAnimIfNotPlaying(player, "player-side")
    }
    if (k.isKeyDown("right")) {
        player.flipX = false;
        player.direction = "right"
        directionVector.x = 1
        playAnimIfNotPlaying(player, "player-side")
    }
    if (k.isKeyDown("up")) {
        directionVector.y = -1
        player.direction = "up"
        playAnimIfNotPlaying(player, "player-up")
    }
    if (k.isKeyDown("down")) {
        directionVector.y = 1
        player.direction = "down"
        playAnimIfNotPlaying(player, "player-down")
    }

    const unitVector = directionVector.unit();
    player.move(unitVector.scale(player.speed));
})

k.onKeyRelease(() => {
    player.stop();
});

}

Can you try this instead? Does it work? If not what do error do you get?
export function setPlayerMovment(k, player) {
player.onUpdate(() => {
    const directionVector = k.vec2(0, 0)

    if (k.isKeyDown("left")) {
        player.flipX = true;
        player.direction = "left"
        directionVector.x = -1
    }
    if (k.isKeyDown("right")) {
        player.flipX = false;
        player.direction = "right"
        directionVector.x = 1
    }
    if (k.isKeyDown("up")) {
        directionVector.y = -1
        player.direction = "up"
    }
    if (k.isKeyDown("down")) {
        directionVector.y = 1
        player.direction = "down"
    }

    if (directionVector.x === 1 || directionVector.x === -1) {
        if (directionVector.y === -1) playAnimIfNotPlaying(player, "player-up");
        if (directionVectory.y === 1) playAnimIfNotPlaying(player, "player-down");

        playAnimIfNotPlaying(player, "player-side")
    }

    const unitVector = directionVector.unit();
    player.move(unitVector.scale(player.speed));
})

k.onKeyRelease(() => {
    player.stop();
});
}

the left and right keys work fine but the diagonal is still bugged and the up and down keys move the player, but the animation doesn't play.

How about this?
if (directionVector.x === 1 || directionVector.x === -1) {
        if (directionVector.y === -1) {
            playAnimIfNotPlaying(player, "player-up");
        } else if (directionVectory.y === 1) {
            playAnimIfNotPlaying(player, "player-down");
        } else {
            playAnimIfNotPlaying(player, "player-side");
        }
    }

now all the directions work exept for up and down(diagonals work as well) for up and down, the movement works, but the animation doesn't play.

This should probably work but is a repetitive. I'm sure you can find a way to make this concise. I don't have much time at the moment.
if (directionVector.x === 1 || directionVector.x === -1) {
        if (directionVector.y === -1) {
            playAnimIfNotPlaying(player, "player-up");
        } else if (directionVectory.y === 1) {
            playAnimIfNotPlaying(player, "player-down");
        } else {
            playAnimIfNotPlaying(player, "player-side");
        }
    } else {
     if (directionVector.y === -1) {
            playAnimIfNotPlaying(player, "player-up");
        } else {
            playAnimIfNotPlaying(player, "player-down");
        }
}

thanks! this works now! I just need to make the player stop when I'm not pressing anything as the code you gave me doesn't stop it.


2.1.3

Fix for no output files. Previous pilot version would fail with unknown internal problem.
Appending PandaID, PanDA_TaskID and PanDA_AttemptNr to payload command, needed to bring these env variables into container. Requested by EventIndex people

Travis test seems stuck on Github but has in fact pass on the Travis side. Closing this pull request and will create another pull request


v0.2 Roadmap

Goal: Load up basic chunks and send them to the client - requires other pre-setup packets too.

[x] Add Necessary Game Packets to be sent as a startup bundle
[ ] Create NBT library (or find one)
[x] Create a block/chunk system for ANVIL format
[x] Send chunk data as packets
[x] Add any other necessary formatting choices
[ ] Break and Place blocks!


feat: 🎸 Update latest SQ compatible with 5.3.0 chain

Description
Updates SQ version compatible with 5.3.0 testnet chain
Breaking Changes
NA
JIRA Link
DA-586
Checklist

[ ] Updated the Readme.md (if required) ?

To be merged after SQ change (https://github.com/PolymeshAssociation/polymesh-subquery/pull/135) gets merged and releases new version

:tada: This PR is included in version 5.9.0-alpha.2 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release
v5.9.0-alpha.2
v5.9.0-alpha.2

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 5.10.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release
v5.10.0
v5.10.0

Your semantic-release bot :package::rocket:


Update DS3231.cpp

Fixed problem with SAMD processors(no function call) by replacing all occurences of the term PM(which
is defined as a macro on SAMD controllers) by PM_time.

Sorry for the very late merge; wasn't watching this!


Fixed nav bar issues, have animations working

Don't add just yet. Just noticed something that broke


Exts: add database management extension

Add an extension for custom management of the persistent db file. In particular, this module should handle db backups.

Closing: not sure how persistence will work yet. Currently the app is ephemeral.


Fix escape down for collapsible section when collapsed

At the moment there's a logic that when the collapsible section is the very last node, pressing ArrowDown will append a paragraph below to escape the section. This only worked when the section is expanded, since the logic only applied on exiting the content node of the container, but when the section was collapsed, you exit from the title node, so the logic didn't kick in.
Extending the logic to check for either fixes the issue.
This was only an issue for the "Escape down", since "Escape up" is always going from the title, irrespective if expanded or collapsed.

right now, it runs in either case, collapsed or expanded. Didn't want to tie it to the node state, when it has a clean solution.


Bug: stuck "Invocation queue loop creating the instance" during sharding tests

worker stuck in:
invocation-loop: {"worker_id":"a52dd8e0-9e88-42b1-a580-b53f4c5e5381/sharding-test-1"}
waiting-for-permits: {"worker_id":"a52dd8e0-9e88-42b1-a580-b53f4c5e5381/sharding-test-1"}
grpc_request: {"account_id":"-1","api":"invoke_and_await_worker","calling_convention":0,"idempotency_key":"5edbc8cf-9733-469c-8c17-0fcc1f9d8a30","worker_id":"a52dd8e0-9e88-42b1-a580-b53f4c5e5381/sharding-test-1"}```

and log with this span after this, also the sharding test executor also got stuck on waiting

i rerun locally the sharding tests with 100 retries, they passed now, so hopefully this was only a temporary broken things, closing for now


Minecraft Servers OpBlocks and Penguin(pgg) not logging on

Minecraft Servers opblocks and penguin pgg say geyser is not working.  Screen shot is the same error message I get on both servers when trying to connect my Bedrock client to either server.  Discord staff of both servers say its a Geyser problem.  I use my laptop for connection for Minecraft and using the Minecraft launcher or 3rd party launcher results in the same message.

My bedrock account is using my laptop connection.

Duplicate of #4240


To provide automation script for CSPM onboarding

Describe the Issue
To onboarding/offboarding community to CSPM at scale, automation scripts to call DOME9 and Azure API are required.
Additional Context

Python Scripts to call  Dome9 API to create, update user/role using API Keys
Python Scripts to call Azure API using user access token to authenticate to add Azure AD user to Enterprise Application (OCIO - Check Point Infinity Portal)

Acceptance Criteria

[ ] Script to Add, Update, Delete Dome9 user/role
[ ] Script to add, remove Azure Ad user from the Enterprise Application

A set of pre-defined requirement that need to be met in order to mark the user story as “done”.  See below for team agreement details.

To expedite the process, the onboarding was done manually.   delegated api permission for MS graph api tested ok.  Will do the demo for the team in the documentation ticket.

Completed


Standardize which language the GLSL lsps get filtered by

I was recently doing some graphics programming and initially when I tried to install an LSP for GLSL I only saw glslls in Mason. After looking for alternatives I found glsl_analyzer which I then locally installed and was wondering why it wasn't included in mason but after checking I realised that it was just not when I filtered by GLSL.
I'm not sure which language would be more appropriate GLSL or OpenGL but I think it would cause less confusion if both were set to the same language.

Yeah, it's because they have different languages in the field of the package.yaml. I can understand why it's confusing so one language or both would be great but it's ultimately up to the maintainer


Give steps their own class

When using the library as of 0.9.x, we need to implement a lot of ugly switches.
Code would be much nicer if we could split the step logic into their own classes and the form would handle find the current step.
I got that interface which does the job:
public interface StepperFormStep {
    String getTitle();
    String getSubtitle();
    boolean isOptional();
    View createView();
    void onStepOpening(VerticalStepperFormLayout formLayout);
}

Then I have implementations such as:
public class StoreNameStep implements StepperFormStep {
    private Context context;
    private EditText name;

    public StoreNameStep(Context context) {
        this.context = context;
    }

    @Override
    public String getTitle() {
        return context.getString(R.string.wizard_store_step_title_name);
    }

    @Override
    public String getSubtitle() {
        return null;
    }

    @Override
    public boolean isOptional() {
        return false;
    }

    @Override
    public View createView() {
        name = new EditText(context);
        return name;
    }

    @Override
    public void onStepOpening(VerticalStepperFormLayout formLayout) {
        String name = this.name.getText().toString();

        if (Strings.isNullOrEmpty(name)) {
            String errorMessage = "You must indicate a store name";
            formLayout.setActiveStepAsUncompleted(errorMessage);
        } else {
            formLayout.setActiveStepAsCompleted();
        }
    }
}

My form looks like that:
public class EditStoreForm extends AbstractStepperForm {
    public EditStoreForm(Activity activity, VerticalStepperFormLayout stepperLayout) {
        super(activity, stepperLayout);
    }

    @NonNull
    @Override
    protected List<StepperFormStep> instantiateSteps() {
        return Lists.<StepperFormStep>newArrayList(
            new StoreNameStep(context)
        );
    }

    @Override
    public void sendData() {
    }
}

Setup in my fragment is a simple one-liner:
form = new EditStoreForm(getActivity(), formLayout);

And this class does the heavy lifting (that can be re-used for all my forms, and ideally it should be hidden by your library : users should just be providing forms and form steps):
public abstract class AbstractStepperForm implements VerticalStepperForm {

    protected Context context;
    protected VerticalStepperFormLayout formLayout;
    protected List<StepperFormStep> steps;

    public AbstractStepperForm(Activity activity, VerticalStepperFormLayout formLayout) {
        this.context = activity;
        this.formLayout = formLayout;
        this.steps = Lists.newArrayList(instantiateSteps());

        VerticalStepperFormLayout.Builder builder = VerticalStepperFormLayout.Builder.newInstance(formLayout, getStepTitles(), this, activity);
        initializeBuilder(builder);
        builder.init();
    }

    @NonNull
    protected abstract List<StepperFormStep> instantiateSteps();

    protected void initializeBuilder(VerticalStepperFormLayout.Builder builder) {
        builder
                .primaryColor(ContextCompat.getColor(context, R.color.colorPrimary))
                .primaryDarkColor(ContextCompat.getColor(context, R.color.colorPrimaryDark))
                .stepsSubtitles(getStepSubtitles());
    }

    @Override
    public View createStepContentView(int stepNumber) {
        StepperFormStep step = steps.get(stepNumber);
        return step.createView();
    }

    @Override
    public void onStepOpening(int stepNumber) {
        StepperFormStep step = steps.get(stepNumber);
        step.onStepOpening(formLayout);
    }

    @NonNull
    private String[] getStepTitles() {
        Collection<String> titles = Collections2.transform(steps, new Function<StepperFormStep, String>() {
            @Nullable
            @Override
            public String apply(StepperFormStep step) {
                return step.getTitle();
            }
        });

        return titles.toArray(new String[titles.size()]);
    }

    @NonNull
    private String[] getStepSubtitles() {
        Collection<String> subtitles = Collections2.transform(steps, new Function<StepperFormStep, String>() {
            @Nullable
            @Override
            public String apply(StepperFormStep step) {
                return step.getSubtitle();
            }
        });

        return subtitles.toArray(new String[subtitles.size()]);
    }
}

Then the only callback method needed is the sendData() callback.

I like it. I'll probably adapt something like this.

First of all, sorry for the immense delay in my response; after a long period of abandonment it is only recently that I have found the time to work on this library again.
Regarding the issue, your suggested approach is quite reasonable, so I have implemented something similar in the new version of the library. Thanks for your collaboration!


Does not inject files generated with file-loader

So using HtmlWebpackPlugin, I'm able to inject the hashed js file into my template, however I also use the file-loader to generate my css file.
What do I need to do to use the plugin to inject this file into my template?
my output looks like this
Version: webpack 1.12.13
Time: 15900ms
                               Asset       Size  Chunks             Chunk Names
                          bundle.css    86.4 kB          [emitted]
             9474e9321064005fa388.js     421 kB       0  [emitted]  app
         9474e9321064005fa388.js.map  296 bytes       0  [emitted]  app
../../templates/layouts/default.html    1.87 kB          [emitted]
   [0] multi app 28 bytes {0} [built]
    + 198 hidden modules
Child html-webpack-plugin for "/var/templates/layouts/default.html":
        + 1 hidden modules

Are you sure it's not injected?
It should inject all emitted files named *.css.

Yes I'm certain,
vagrant@vagrant:/var/www/web$ cat templates/layouts/default.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title></title>
</head>
<body>
</body>
</html>
vagrant@vagrant:/var/www/web$ ./node_modules/.bin/webpack
Hash: 9474e9321064005fa388
Version: webpack 1.12.13
Time: 17665ms
                               Asset       Size  Chunks             Chunk Names
                          bundle.css    86.4 kB          [emitted]
             9474e9321064005fa388.js     421 kB       0  [emitted]  app
         9474e9321064005fa388.js.map  296 bytes       0  [emitted]  app
../../templates/layouts/default.html  173 bytes          [emitted]
   [0] multi app 28 bytes {0} [built]
    + 198 hidden modules
Child html-webpack-plugin for "/var/templates/layouts/default.html":
        + 1 hidden modules
vagrant@vagrant:/var/www/web$ cat templates/layouts/default.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title></title>
</head>
<body>
<script src="/bundle/9474e9321064005fa388.js"></script></body>
</html>

Hmm could you please add the plugin configuration?

const path = require("path");
const webpack = require("webpack");
const HtmlWebpackPlugin = require("html-webpack-plugin");

module.exports = {
    cache: true,
    entry: {
        app: [
            path.join(__dirname, "assets/js/app.js"),
        ],
    },
    output: {
        path: path.join(__dirname, "public/bundle"),
        publicPath: "/bundle/",
        filename: "[hash].js",
        sourceMapFilename: "[file].map",
    },
    module: {
        loaders: [
            {
                test: /\.html$/,
                loaders: ["raw"]
            },
            {
                test: /\.jsx?$/,
                exclude: /(node_modules)/,
                loader: "babel",
                query: {
                    plugins: ["transform-runtime"],
                    presets: ["es2015-loose"],
                },
            },
            {
                test: /\.s?css$/,
                loaders: ["file?name=bundle.css", "extract", "raw", "sass"]
            }
        ],
    },
    resolve: {
        modulesDirectories: [
            "node_modules",
            "assets/js",
        ]
    },
    plugins: [
        // Run webpack in production mode.
        // figure out the conflicts between this and dev server later.
        new webpack.DefinePlugin({
            "process.env": {
                // This has effect on the react lib size
                "NODE_ENV": JSON.stringify("production")
            }
        }),
        new webpack.optimize.DedupePlugin(),
        new webpack.optimize.UglifyJsPlugin({
            compress: {
                warnings: false
            }
        }),
        new HtmlWebpackPlugin({
            template: path.join(__dirname, "templates/layouts/default.html"),
            filename: path.join(__dirname, "templates/layouts/default.html"),
            inject: "body",
        }),
    ],
};

The filtering is done here:
https://github.com/ampedandwired/html-webpack-plugin/blob/bfed3a29cc8fc10564c90015751ff0714f6924cb/index.js#L418-L426
As you see it picks up all files called .css from all chunks.
Maybe you could start debugging there

It looks like the css file I'm outputting isn't in the chunkFiles.

Sorry I am not so deep into the file loader and the chunk files are generated directly by webpack itself.

It looks as though the file-loader does not generate a new chunk for the files it generates

You have full access to the current compilation in your templates - just disable inject and add the js/css files by your own logic


Update gen-admin-api-def.sh

Summary

Full changelog

[Implement ...]
[Add related tests]
...

Issue reference

Fix #[issue number]

Hello, this does not seems to be a valid PR. Please feel free to reopen with updates if this is a mistake.


Need a supported load balancer for workloads for TCE on vSphere (and Docker?)

Feature Request
Since we deferred the option of using NSX ALB for user workloads until after MVP for both inward-facing (no commitment to help support the community use) and outward-facing reasons (too complex for developers, too resource-intensive for small clusters) reasons, we should include a blessed approach to L4 load balancing, preferably in the MVP if possible.
Describe alternatives you've considered
The Contour team has already demonstrated interoperability with MetalLB. However, there have been complaints about stagnation and lack of support in the MetalLB community since its founder left the project over a year ago. Kube-vip and haproxy are also options that are known to be used by members of the upstream k8s-on-VMware user group.
Alex Xu reports that if we opt for MetalLB, the Contour team would package and help test, and if either of the others, we should package but they would help test, all in time for 0.8.0.

@qnetter any update on this? Something we still want to get in the roadmap or potentially prioritize?

I think we do want to do this, and there is momentum internally and externally toward kube-vip.


NTR rereplication phenotype

NTR: abnormal negative regulation of mitotic DNA replication initiation resulting in complete rereplication and mitotic cell cycle arrest
synonym: abnormal mitotic cell cycle arrest with re-replicated DNA
synonym: abnormal mitotic arrest with 4C DNA content
A regulation phenotype observed in the vegetative growth phase of the life cycle in which negative regulation of the initiation of mitotic DNA replication is abnormal, resulting in one or more rounds of rereplication of the entire genome and mitotic cell cycle arrest  (in G2?/ before mitosis?).
descendant  of
FYPO:0001425
abnormal negative regulation of mitotic DNA replication initiation resulting in complete rereplication
A regulation phenotype observed in the vegetative growth phase of the life cycle in which negative regulation of the initiation of mitotic DNA replication is abnormal, resulting in one or more rounds of rereplication of the entire genome.
descendant of
FYPO:0000444	abnormal mitotic cell cycle arrest with replicated DNA
A cellular process phenotype in which progression through the mitotic cell cycle is arrested when cells have replicated genomic DNA, under conditions where arrest does not normally occur.
PMID:7774573

@jvhayles

abnormal negative regulation of DNA replication initiation resulting in complete rereplication, with mitotic cell cycle arrest FYPO:0005710
edit file: b195a9d2b07819052614b9be249a6853a8888f7e
release: 99819ea12811962b4b71c6793722c4377f141fde

Should I see this as a descendant of FYPO:0000444 placeholder term?

oops, forgot the has_part. will add next time I'm editing ...

link added
edit file: cbe103231a15beb10dd459ec1fe7cc3be47463b7
release: b00a4204edc68986290523b3a5695954b3d19b46


Would you mind fact-checking my inclusion of rauthy in my comparision table?

I developed obligator with many of the same design goals as Rauthy. Honestly if I had known about Rauthy at the time (it was brought to my attention recently by @erlend-sh) it might have met my needs better than making my own from scratch. Rauthy is much more complete. Anyway, I was wondering if you might help me fill out/fix any errors I may have made concerning Rauthy in the comparison table here. You can suggest edits to the source spreadsheet here. Thanks!

Hey, yes sure.

if I had known about Rauthy at the time

Yes that is often a problem, that there might be interesting projects that are rather hard to find.
Tbh, I am not a guy that goes out into social media and stuff for announcements and so on, so it might even be a bit harder to find rauthy.
I am moving away from google, so I would post my answer here instead of adding to the spreadsheet.
Multi-domain auth
I read in your readme, what you mean with that. And this would be more a client than a server thing, so nothing Rauthy does or would implement. This could be done easily on a clients callback site, or something like an "auth proxy", which just has a list of redirects.
However, depending on the client, you would be logged in directly, because you will get a Session for Rauthy itself once you logged in successfully. This means if I log in to Client A now and in 10 min, I want to log in to Client B, I will see the login page from Rauthy for a short moment, but will be logged in without interaction. This depends however on the setup and config, and if then client forces a fresh login with the prompt param.
HTTP API
Yes, there is a pretty big one with dedicated API Keys and fine grained access control. It is fully documented with OpenAPI / Swagger as well:

You can created fine-grained API keys with optional key expiry for almost all of the actions, apart from a few ones that were excluded explicizly, like creating new API keys (prevent privilege escalation if a key has been leaked somewhere) and edit Upstream Auth Providers:

Trusted Header Auth
Is not implemented, but would be a nice feature though. You could already use Rauthy for instance with the Traefik ForwardAuth middleware and it would basically only require Rauthy to add some headers there.
I guess I will just add this feature to the TODO list, since it would be implemented pretty quickly anyway.
SAML + LDAP
No support here and it never will. Even though these are used in big enterprise still a lot, I think OIDC is simply better in every way.
Standalone reverse proxy
What exactly do you mean with that? Like pass through each single request through Rauthy, make sure authentication is valid, and forward to the final destination? Because afaik there is really only oauth2 proxy (which you listed as well), which does this. The others support things like the above mentioned ForwardAuth but don't really proxy the request itself, but lease let me know if I am wrong.
Btw a really cool tool for collecting code lines is tokei -> cargo install tokei

With #339 you can consider the trusted auth headers to be implemented, just not released yet. :)

just not released yet

... by now it has been released with v0.22.1

I just found rauthy too, and it looks awesome!
My question would be regarding this:

SAML + LDAP
No support here and it never will. Even though these are used in big enterprise still a lot, I think OIDC is simply better in every way.

Does this refer to LDAP for authentication? OIDC definitely is much better, but LDAP is more than only authentication. LDAP is a directory of users, groups, and - in our case - quite a bit more data.
Authentication is only one part, another is authorization and on- and offboarding. With rauthy managing users and groups, how do these end up in the applications? Are there plans to implement SCIM? How would users be removed/disabled in applications when removed in rauthy?
Would adding LDAP as a backend for what users and groups exist (not passwords etc.) an option? Other applications would sync with LDAP, but authentication would happen with OIDC and rauthy, including MFA, passkeys, etc.

I just found rauthy too, and it looks awesome!

Thanks! ;)

LDAP is a directory of users, groups, and - in our case - quite a bit more data.

Rauthy can handle all of that too.
You manage everything, all of your mappings, roles, groups, whatever, from inside the Admin UI (or via API key, if you like).
Rauthy can map:

Roles
Groups
Scopes
Fully custom values (all of these are considered as strings for most compatibility)

For instance, I can create roles and groups (in addition to the OIDC default ones) and then map them to a user like this:

You can then create fully customized values for each user as well, for instance

Which can then be set as a key / value entry for each single user

Then you can create custom scopes, so that only the clients you allow will actually get access to all of these values and mappings and you can decide, which of these custom attributes you want to link / map to a scope, if any at all (maybe your downstream app only cares about the scope itself being present)

... and finally map / allow scopes and mappings for each client independently

How would users be removed/disabled in applications when removed in rauthy?

Depends on your downstream app, but usually fully automatically. On login, a user gets redirected to Rauthys Login screen and if it does not exist anymore or has been disabled, you would simply not get a token for this user.
Additionally, when your downstream app checks the tokens after login for updates on roles / groups (like it always should), all mappings will be forwarded and distributed right away.
All information about the user would be included in the signed id_token after a successful login, so you don't need any other external sync's and updates at all. You only care about the settings on Rauthy's side and that's it.

Are there plans to implement SCIM?

No, you simply don't need it with OIDC, because all of the information an application needs (and is actually allowed to see) will be included in the id_token upon a successful login.

Would adding LDAP as a backend for what users and groups exist (not passwords etc.) an option?

I never thought about this. I never had a use case or question regarding this so far.
I just don't see a reason why someone would want that, if you can do all the mapping and setup with OIDC directly without external dependencies, but I am no LDAP prof and I have never really used it at a bigger scale.

Would adding LDAP as a backend for what users and groups exist be an option?

If LDAP is a hard requirement (e.g. in order to accommodate a legacy system) you’re probably better off with Kanidm by @Firstyear as it has an LDAP interface to support that exact use case.

Thanks for the attribute mapping guide. I tried that before, and the plus button of the scopes didn't appear, but now it did work.
That isn't exactly the kind of data I meant, though. My LDAP contains users accounts, groups, (and other records) that, for example, are used by our self-hosted mail servers too (postfix/dovecot). User records have many email addresses beside a primary address, groups can have addresses and work like email aliases. It is a shared hierarchical database, a directory service, accessed by different tools, not just a password store. Groups can be hierarchical, include other groups, and be mapping into a flat memberOf list on users using ppolicy overlays.
You could even store UNIX users in LDAP, with UID/GID, home directory, storage quotas, so that they can log in on all machines. Usually, the actual login would be through Kerberos. This is still common in many universities or high-performance computing labs.
So, LDAP stores a lot more than users and passwords. To keep all that data in one place, it would be nice to have LDAP as a backend, not for authorization, but for users, groups, and attributes. It would be wonderful to map LDAP attributes to user and IAM attributes, e.g. to use them in OIDC profiles (could be text config file). Still being able to sync password changes back to LDAP for legacy applications would be a plus.

If LDAP is a hard requirement (e.g. in order to accommodate a legacy system) you’re probably better off with Kanidm by @Firstyear as it has an LDAP interface to support that exact use case.

This is an LDAP interface for authentication, based on the users and groups stored in kanim. That the other way then asked and can be covered with OIDC.
The question still is, how do you expect user on- and offboarding with rauthy to happen? Deleting a user in rauthy will not remove them from any application where they signed in. If you add a user to a group, how will that be synced to an application using that group?

All information about the user would be included in the signed id_token after a successful login, so you don't need any other external sync's and updates at all. You only care about the settings on Rauthy's side and that's it.

That is the primary problem with SSO systems. Until a user signed in to a system, they will not be present there. You wouldn't be able to assign them a ticket before they signed in to the ticket system.
Worse, they can never be offboarded from a system through a login. Accounts in these applications would stay active forever. If you run a GitLab with SSO, even after removing the user from rauthy, their GitLab account would stay active, and they could access repos through SSH keys. If you have any tool with per-seat payment, you would have to continue to pay for "removed" users. Any application that sends notification emails (like comments on a ticket) would continue sending that to "removed" users.
If you change groups for a user, these won't change in any application until the user logs in again. Sometimes that could be elevated with short session timeouts in applications, but it might not. Removing a user from groups (restricting access) won't actually restrict access until the user logs in again. If they are still signed on to the application, they can still access all resources.
You will need either LDAP or SCIM to actively propagate changes. With LDAP, applications regularly sync users and group mappings. This is a more legacy approach that can be tricky with not syncing users to applications they actually have no access to. As far as I know, SCIM was invented by OIDC IAM providers, such as Okta, to push events to downstream applications. The moment you remove a user from a group, the IAM will send HTTP requests to each downstream application so that they can create, update, or remove the user. That would be independent of whenever the user would log in again. And it would be for user creation and deletion. For example, GitLab would deactivate the user, block SSH keys, and stop sending notification emails.
Please excuse that long message, but rauthy would be a wonderful solution for all my home, club, and some business use cases, if it could do that. Other solutions I've found either can't do user lifecycle or read data from LDAP, or are huge JVM applications with limited OIDC support and no WebAuthn/passkeys.

Okay, I got what your problem is.
There are ways to solve a few of the problems with just OIDC (offline tokens for instance to sync user data), but the downstream application has to support this.
I never used SCIM so far, so I have no idea how much work it would be to implement it, but I don't think that Rauthy will support LDAP sync as long as not someone else provides all the PRs for this. I simply don't know enough about LDAP and I have no use case for this, so for me all these many hours of work (probably?) would not have any benefit.
I thought about providing an LDAP interface in the very beginning, only to find out that there simply were no crates that supported it (back then, haven't checked in a while), so I decided to simply leave it out, because it would be too much work and I personally don't use / need it at all.
SCIM though sounds reasonable, but I would need to check this and dig deeper into the topic to give a definite answer. This would be quite a bit of additional work while I don't even use / need it right now (maybe in the future).

Thanks for the attribute mapping guide. I tried that before, and the plus button of the scopes didn't appear, but now it did work.

This only appears for custom scopes and when custom attributes exist, so you can't mess up OIDC default scopes by accident.

Since this is off-topic and a bit more, we should continue in the discussions and not this issue though :)
https://github.com/sebadob/rauthy/discussions/349

Thanks @sebadob; very helpful!

I am moving away from google, so I would post my answer here instead of adding to the spreadsheet.

Fully support that

Multi-domain auth
I read in your readme, what you mean with that. And this would be more a client than a server thing, so nothing Rauthy does or would implement. This could be done easily on a clients callback site, or something like an "auth proxy", which just has a list of redirects. However, depending on the client, you would be logged in directly, because you will get a Session for Rauthy itself once you logged in successfully. This means if I log in to Client A now and in 10 min, I want to log in to Client B, I will see the login page from Rauthy for a short moment, but will be logged in without interaction. This depends however on the setup and config, and if then client forces a fresh login with the prompt param.

I'm not sure how you would do what I need purely on the client side. Say I have app1.domain1.com and app2.domain2.com. My auth server is running at auth.domain1.com. Once I have a session there, it sets a cookie for *.domain1.com and uses trusted header auth, so any apps on that domain are implicitly authenticated without needing to do any OIDC flow explicitly. Those apps don't even need to implement OIDC, which is the beauty of trusted header auth.
However, app2.domain2.com and any other apps on *.domain2.com would each have to implement OIDC and perform individual flows.
What I mean by multi domain auth is that I would also expose the same auth server instance at auth.domain2.com, and perform a redirect whenever the user authenticates at auth.domain1.com with a special token that would automatically create a session on auth.domain2.com as well. That way you can host apps on as many domains as you want and none of them have to implement OIDC directly. Note that this still isn't implemented in obligator yet.

Standalone reverse proxy
What exactly do you mean with that? Like pass through each single request through Rauthy, make sure authentication is valid, and forward to the final destination? Because afaik there is really only oauth2 proxy (which you listed as well), which does this. The others support things like the above mentioned ForwardAuth but don't really proxy the request itself, but lease let me know if I am wrong.

Yeah that's basically the idea, and I'm pretty sure several of the options in the table support it. If not there's a lot of incorrect information in the table.

Btw a really cool tool for collecting code lines is tokei -> cargo install tokei

Thanks I'll check it out

Also, passwordless email login in the way of "sending a magic link and be logged in without anything else" is not supported by Rauthy.

Aw good to know. Can you clarify what is meant by "True passwordless accounts with E-Mail + Magic Link + Passkey" in the feature list? Maybe you can get a magic link but have to associate it with a passkey immediately?

Would adding LDAP as a backend for what users and groups exist be an option?

If LDAP is a hard requirement (e.g. in order to accommodate a legacy system) you’re probably better off with Kanidm by @Firstyear as it has an LDAP interface to support that exact use case.

@erlend-sh since you mentioned Kanidm, any chance you could take a look at it's new entry in the spreadsheet and let me know if I have anything wrong?

@sebadob one other question: are you interested in more attention for Rauthy? I think if it was more widely known it would likely become quite popular. Personally I've learned I don't always want more attention for my open source projects haha.

What I mean by multi domain auth is that I would also expose the same auth server instance at auth.domain2.com, and perform a redirect whenever the user authenticates at auth.domain1.com with a special token that would automatically create a session on auth.domain2.com as well. That way you can host apps on as many domains as you want and none of them have to implement OIDC directly.

Okay, I got the idea. What is unclear to me though is how you would want to perform the redirect between auth1 and auth2, because you cannot use cookies. Or would you do something like store some value for auth2 on the client side,  basically cache some id or whatever, redirect to a special endpoint on auth1 and basically lookup that value? Since it is the exact same backend just listening to another pub url, it would find the value inside the cache?
Yes that could work, but you would still have redirects.
When you want to access your apps without them being able to use OIDC, then there is only trusted headers, right.
I already thought about developing something like rauthy-proxy when I am close to v1.0. The idea was that you could run it as a sidecar (I am running all workloads on Kubernetes) which does kind of the same thing as oauth2 proxy, but natively with Rauthy without much config and setup, and using the most secure defaults.
When you do it like that, as well as with trusted headers, you have one big problem though that a lot of people forget:
You don't have a really good CSRF protection, because it technically can not work, when the downstream app does not implement some mechanism for this.
If you can be sure that all users have at least browsers that support Lax and Strict cookies(hopefully by now) and all your applications are 100% safe regarding XSS attacks, than only setting a cookie is safe these days.
When your downstream apps support OIDC, it is the safer option for sure.
A redirect for another login to a new site to the same IdP is not a big thing though. Rauthy can re-use the clients active session and issue a token without user interaction and I think this is not too bad of a UX. It depends on how locked down your config is though, of course.
There is another way to log in to multiple apps, but this is annoying as hell. :D
You can request permission from the browser to set a cookie for a different domain, but this is very weird from a UX. Microsoft does this for instance and it is such a bad UX.
If you want to log in to multiple downstream apps with a "single login", I would create a tiny proxy app. This could be the callback for all your apps and it just has a list of apps it should log in to. When finished with the first flow, just redirect to app2 login, finish, then app3, and so on.
This would work and you would get a few redirects, but all of these at least without user interaction. The "bad" thing about this is that this proxy would be the MITM and if this would be vulnerable, all of your apps would have security issues.

Yeah that's basically the idea, and I'm pretty sure several of the options in the table support it. If not there's a lot of incorrect information in the table.

I needed this in the past for a kubernetes storage dashboard which did not implemtn any auth at all and the only option I found was oauth2 proxy, which was specifically built to serve this purpose. We were using Keycloak back then as well and it was not able to do this for sure.
Even if possible, I would not do it tbh. The problem is, that if any of your apps behind your auth reverse proxy is vulnerable for XSS attacks, they could actually steal your maybe even Strict cookie, because they would run on the same origin as your auth provider. This is actually the only situation where Lax and Strict cookies are not safe - via another app on the same domain that is vulnerable.
Because of this attack vector and you never know how people host / build their apps, Rauthy Admin UI for instance cannot be accessed with just a token. The UI uses a session which is managed in the backend with an additional CSRF token from the local storage.

Aw good to know. Can you clarify what is meant by "True passwordless accounts with E-Mail + Magic Link + Passkey" in the feature list? Maybe you can get a magic link but have to associate it with a passkey immediately?

What I mean with this is that you can have a fully working 2FA / MFA account without ever setting a password at all. Upon registration, you will get an E-Mail with a Link to set your initial "Password". Using this link will validate your email at the same time. But you can then choose between a password and a passkey. If you choose passkey, you never need any password at all.
Some providers create problems with this Email, for instance Microsoft. They scan their customers emails and even use(!) the links inside to "protect" them from bad Links. But what Rauthy does by default is, that it sets an encrypted binding cookie once that link has been used. From that moment on, even if no data has been submitted via the form, the link can only accessed by this very browser and none else. The reason is that only having a secure URL with random, non-guessable values is not really secure. URL are always in plain text and they are even logged in plain text in a lot of places.
With this binding cookie, Rauthy makes sure that even if someone would pick such a URL out of a log, use it and would just be faster than the original user (or automated...), that it cannot happen. For an initial account sign up, this is not the end of the world, but for a password reset it is for sure.
Rauthy tries to be as secure as possible by default, but companies like Microsoft kill it. This is the only reason why you can opt-out of this binding cookie feature.

@sebadob one other question: are you interested in more attention for Rauthy? I think if it was more widely known it would likely become quite popular. Personally I've learned I don't always want more attention for my open source projects haha, so figured I'd ask before I start recommending it to people.

I am absolutely on your side here. Especially in the beginning it just costs more time than anything else, because you have so much on your TODO anyway but you have to answer issues and stuff. This is the reason why I did not do too much about it.
But, Rauthy is very close to v1.0 release now. The next phase will be about some benchmarking, code clean up, some leftover TODOs and optimization, but feature wise we are almost there. Apart from that the experimental FedCM will be implemented as well, but this does not keep us from releasing stable.
So, yes sure it would actually be nice to have some more people testing before the first v1.0.0 release. :)

This is an LDAP interface for authentication, based on the users and groups stored in kanim. That the other way then asked and can be covered with OIDC.

The Kanidm LDAP interface does a lot more than you describe here :)

Okay, I got the idea. What is unclear to me though is how you would want to perform the redirect between auth1 and auth2, because you cannot use cookies. Or would you do something like store some value for auth2 on the client side, basically cache some id or whatever, redirect to a special endpoint on auth1 and basically lookup that value? Since it is the exact same backend just listening to another pub url, it would find the value inside the cache? Yes that could work, but you would still have redirects.

Yep that's basically the idea. You still have redirects, but the difference is they happen all at once when you log in to your auth server, ie no redirects for individual apps.

When you want to access your apps without them being able to use OIDC, then there is only trusted headers, right. I already thought about developing something like rauthy-proxy when I am close to v1.0. The idea was that you could run it as a sidecar (I am running all workloads on Kubernetes) which does kind of the same thing as oauth2 proxy, but natively with Rauthy without much config and setup, and using the most secure defaults.

I'm curious what your motivation for making rauthy-proxy would be, as opposed to just using an off-the-shelf reverse proxy?

When you do it like that, as well as with trusted headers, you have one big problem though that a lot of people forget: You don't have a really good CSRF protection, because it technically can not work, when the downstream app does not implement some mechanism for this. If you can be sure that all users have at least browsers that support Lax and Strict cookies(hopefully by now) and all your applications are 100% safe regarding XSS attacks, than only setting a cookie is safe these days. When your downstream apps support OIDC, it is the safer option for sure.
A redirect for another login to a new site to the same IdP is not a big thing though. Rauthy can re-use the clients active session and issue a token without user interaction and I think this is not too bad of a UX. It depends on how locked down your config is though, of course.

This is a valid point. It sucks because not needing to implement authentication makes apps significantly easier to implement. I suppose that can be mitigated with nice OIDC libraries though. And most multi-user apps still need some authorization implementation, which is more work than authentication anyway, so maybe you're not really saving that much.
That said, have you thought at all about possible ways to mitigate these CSRF concerns? Maybe some sort of system-wide Referer shenanigans built into rauthy-proxy? Probably not doable...

If you want to log in to multiple downstream apps with a "single login", I would create a tiny proxy app. This could be the callback for all your apps and it just has a list of apps it should log in to. When finished with the first flow, just redirect to app2 login, finish, then app3, and so on. This would work and you would get a few redirects, but all of these at least without user interaction. The "bad" thing about this is that this proxy would be the MITM and if this would be vulnerable, all of your apps would have security issues.

This would mitigate the UX issues somewhat, but I agree doing an individual OIDC flow per app isn't actually that big of a deal. And honestly it's probably good training for the user to develop a slightly clearer mental model of the security they're operating under.

I needed this in the past for a kubernetes storage dashboard which did not implement any auth at all and the only option I found was oauth2 proxy, which was specifically built to serve this purpose. We were using Keycloak back then as well and it was not able to do this for sure. Even if possible, I would not do it tbh. The problem is, that if any of your apps behind your auth reverse proxy is vulnerable for XSS attacks, they could actually steal your maybe even Strict cookie, because they would run on the same origin as your auth provider. This is actually the only situation where Lax and Strict cookies are not safe - via another app on the same domain that is vulnerable. Because of this attack vector and you never know how people host / build their apps, Rauthy Admin UI for instance cannot be accessed with just a token. The UI uses a session which is managed in the backend with an additional CSRF token from the local storage.

This vulnerability only exists if you're running on the exact same domain, with apps at different paths, right? ie as long as you're using separate subdomains for apps you should be safe? IMO trying to get multiple apps running on the same domain is a bad idea, for this and other reasons.

What I mean with this is that you can have a fully working 2FA / MFA account without ever setting a password at all. Upon registration, you will get an E-Mail with a Link to set your initial "Password". Using this link will validate your email at the same time. But you can then choose between a password and a passkey. If you choose passkey, you never need any password at all. Some providers create problems with this Email, for instance Microsoft. They scan their customers emails and even use(!) the links inside to "protect" them from bad Links. But what Rauthy does by default is, that it sets an encrypted binding cookie once that link has been used. From that moment on, even if no data has been submitted via the form, the link can only accessed by this very browser and none else. The reason is that only having a secure URL with random, non-guessable values is not really secure. URL are always in plain text and they are even logged in plain text in a lot of places. With this binding cookie, Rauthy makes sure that even if someone would pick such a URL out of a log, use it and would just be faster than the original user (or automated...), that it cannot happen. For an initial account sign up, this is not the end of the world, but for a password reset it is for sure. Rauthy tries to be as secure as possible by default, but companies like Microsoft kill it. This is the only reason why you can opt-out of this binding cookie feature.

Generally the way I've seen this scanning behavior mitigated is by introducing an additional click in the process. So the link goes to a page that can be loaded as many times as you want, and that page contains only a single button/link that says "Click to continue logging in", and that link can only be followed once. Would that solve your problem with Microsoft or am I missing something?
I'm not sure I understand your concern with the magic link going into logs though. Are you referring to email provider logs? They already have complete control over your email, they can steal your accounts whenever they want. Same for a compromised browser. Or are you referring to a different vector? Generally I support defense-in-depth, but this seems fairly likely to result in UX issues if the user starts the flow on one devise and opens the email on another.

I am absolutely on your side here. Especially in the beginning it just costs more time than anything else, because you have so much on your TODO anyway but you have to answer issues and stuff. This is the reason why I did not do too much about it. But, Rauthy is very close to v1.0 release now. The next phase will be about some benchmarking, code clean up, some leftover TODOs and optimization, but feature wise we are almost there. Apart from that the experimental FedCM will be implemented as well, but this does not keep us from releasing stable. So, yes sure it would actually be nice to have some more people testing before the first v1.0.0 release. :)

Right on, well I for one am going to start evangelizing. This is an excellent project.
BTW, would love to hear your thoughts on FedCM, especially as compared to Mozilla Persona. We could have that conversation over on my IndieBits forums, or here or somewhere else if you prefer. I know @erlend-sh would likely be interested in joining as well.

Originally I wanted to find a solution for exactly the CSRF issue with a rauthy-proxy, and I have thought a lot about it. But to make it really secure, you cannot treat security as an addon, it must be an integral part instead. Oauth2 proxy is using double submit cookies and you can do stuff like:

You mean oauth2-proxy is using double submit cookies for the auth page, or it's somehow injecting them into the traffic from the apps?

This is why I created the rauthy-client. When someone has to set up OIDC without knowing anything about it, this can be pretty terrifying at first. Typical OIDC libraries allow you to configure anything and everything, but most of the stuff you usually don't need. What you need are safe defaults and that's it, as long as you don't have a very special use case. So, the rauthy-client does exactly that and not more. It doesn't let you choose encryption algorithms and stuff, it does all of that for you in the most secure way possible, and I think the examples are pretty easy to follow.

This is a laudable effort. In my experience trying to create very simple OAuth2 clients, it's the redirects that are tricky to abstract away in a super clean way. The end result isn't too bad, but it does require developers to essentially understand the flow, which is pretty confusing at first IMO.

Generally the way I've seen this scanning behavior mitigated is by introducing an additional click in the process. So the link goes to a page that can be loaded as many times as you want, and that page contains only a single button/link that says "Click to continue logging in", and that link can only be followed once. Would that solve your problem with Microsoft or am I missing something?

That would work if I was allowed to use JS inside an HTML Email. :D No the idea behind this is that all the bits in pieces in between on your requests's journey through the internet are the problem. The URL is always clear text and is logged in so many places. Every router or host your request is traveling through, every firewall. And when you take a look at any app, any log, any whatever, a URL is never considered to be sensitive information. For instance, even if I am just a normal user on a linux host, I can see logs of sshd without issues. If I could fetch a "secure" URL from such a place easily, there is nothing that could stop me from using it.

Not sure how JS is involved here. What I'm trying to describe only involves links.
Also what do you mean by the URL being clear text? Are you referring to HTTP connections? HTTPS definitely hides the URL from everything until the terminating proxy, which is pretty much always controlled by a trusted entity. Only the SNI is sent in plaintext. You could certainly argue that logs can be accidentally leaked by those entities, but it depends on your thread model.

This is being implemented into Rauthy behind an experimental feature flag pretty soon. We have @sjud working on this and he already has a working example set up. The integration into Rauthy should be pretty easy, since most of the structures do exist already.
This is a really cool and interesting draft, but only when they implement BYOIDP properly. And, more important, all the other browser vendors need to follow along. If this will stay only a Google thing, it will not make it into Rauthy stable. I already hate it so much when websites are built against chromium engines only and start falling apart on other browsers.
When this is properly implemented, this is the best solution from a UX perspective. It only has one issue, which is the Cookie problem when you have multiple apps on the same origin and one of them is vulnerable. Because especially with something like FedCM, you probably want pretty long lived cookies to provide a good UX. The cookie issue can be solved when people know what they are doing. Then it would be absolutely secure as long as your main app is, but this is most often not the case.

Right on, thanks for the perspective.

You mean oauth2-proxy is using double submit cookies for the auth page, or it's somehow injecting them into the traffic from the apps?

Yes so what is does is basically pretty simple:

intercept all traffic to the app
look for the oauth2 proxy cookies
if they exist, validate them
if not, redirect to the SSO provider, get a token, validate it and set the cookies

So it enables applications that do not have authentication to actually work with SSO in a pretty convenient way.
For instance, the pretty popular Longhorn storage provider for K8s has a nice dashboard for managing your volumes, but without any authentication at all. I am running OAuth2 Proxy in front of it which then secures the whole application for me. As I said I have not taken at look at it since some time, because by now all my stuff does work with OIDC natively, but I think I does CSRF protection with the double submit cookie pattern as well, yes.
Since it is running as a reverse proxy to the application, it can inject and validate cookies.
If your app supports it, a native OIDC implementation should be favored, but if not, the oauth2 proxy is really nice to have.

Also what do you mean by the URL being clear text? Are you referring to HTTP connections? HTTPS definitely hides the URL from everything until the terminating proxy, which is pretty much always controlled by a trusted entity. Only the SNI is sent in plaintext. You could certainly argue that logs can be accidentally leaked by those entities, but it depends on your thread model.

Yes, you usually only see the domain name in the Client Hello. TLS 1.3 can encrypt that too, but browser support is still limited.
I am coming from the enterprise world where you usually have quite a few proxies in between and they often do deep packet inspection and such things. So they decrypt the traffic and re-encrypt it afterwards again (and often not, which is really bad). The current implementation seems a bit paranoid I must confess and I already talked with quite a few people about it. My plan is to reverse the logic for these security addons and make them opt-in in favor of a better UX, because most people will never come in the situation with decrypting proxies in between. Also, the additional E-Mail verification that currently exists on the password reset form will be removed in favor of a nicer UX.

Btw, one thing that came to my mind is the confusion often between same site and same origin. I don't know if I said it correctly.
So the problem with Cookies is, that we don't have a same-origin attribute, only same-site. This is nice, but for instance auth.example.com and evilapp.example.com are not the same origin, but they are the same site. This means when you have set a cookie for auth.example.com and access evilapp.example.com  with that cookie, it can read it.
So, different domains is fine, but a different sub-domain will always receive the cookie.

BTW, would love to hear your thoughts on FedCM, especially as compared to Mozilla Persona. We could have that conversation over on my IndieBits forums, or here or somewhere else if you prefer. I know @erlend-sh would likely be interested in joining as well.

Reading the FedCM MDN docs, it appears as though it's up to relying parties to add the IDP to any page for authentication. So rather than being a distributed "byo account" service, it looks a lot more like a way that you can just add "auth with google" to your service.

@Firstyear that is what it is at risk of becoming, but there is an open call for indies to implement true BYOIDP: https://github.com/sebadob/rauthy/discussions/145#discussioncomment-8831943

@erlend-sh Given the current browser situation, and what's already happened to webauthn at the hands of ms/google, I wouldn't be optimistic. :(

and what's already happened to webauthn at the hands of ms/google

@Firstyear what exactly do you mean with that?
They often simply "destroy" things to make more business. So many annoying things, instead of "creating a better internet". This is simply not their goal, even if they sell it as their tag line.
I stopped self-hosting email because Microsoft just decides to block mail from your server randomly, when you are too small. You can contact them and make sure to them you are not a spammer, but they don't care, simply ignore you. -.-

Btw, one thing that came to my mind is the confusion often between same site and same origin. I don't know if I said it correctly.
So the problem with Cookies is, that we don't have a same-origin attribute, only same-site. This is nice, but for instance auth.example.com and evilapp.example.com are not the same origin, but they are the same site. This means when you have set a cookie for auth.example.com and access evilapp.example.com with that cookie, it can read it. So, different domains is fine, but a different sub-domain will always receive the cookie.

@sebadob I've been thinking more about the subdomain cookie/trusted header problem, and have an idea. Here's the problem as I understand it:
With trusted header auth you sign in to auth.example.com, and it sets a cookie with the Domain attribute set to .example.com. Then whenever a request is checked at the forward auth endpoint, if it has a valid cookie it sets some sort of UserID header, which is then trusted implicitly by any upstream apps, say app1.example.com and app2.example.com.
The problem is if app2.example.com is evil, or has an XSS vulnerability, it can send AJAX requests directly to app1.example.com, and the auth cookie will be sent even it SameSite is strict or lax, because they are on the same same, enabling app2.example.com to perform whatever actions it wants on app1.example.com.
Could this possibly be solved by having an additional cookie per domain? So basically when you navigate to app1.example.com for the first time, it sets a simple cookie with no Domain attribute, which means it won't be shared with other subdomains. Then when making requests, the auth server only sets the UserID header if the request contains both an auth cookie and a valid cookie for the exact domain as the host.
I'm not a security expert and very likely missing something obvious. Thoughts?

I'm not a security expert and very likely missing something obvious. Thoughts?

No, this is correct. This is the reason why Rauthy does not set a domain attribute on its cookies, only the path to /auth.
The problem is, that not all user agents treat cookies without a domain attribute correctly and just set it by default to "the current one" if it's omitted. If have not tested this in a while, if at least by now all major browser do this correctly, but this is my latest information on this.
For instance, when I check Rauthy's cookies no inside my browser, it shows the domain attribute, even if I do not set this on purpose. But I am not sure, if they treat it correctly and maybe just show it for convenience.
I would need to do some tests to check all current browsers, how they handle this.
The problem then is, if someone is using IE8 or something :D . But you cannot protect these people imho and if you try to do so, security is like 95% of your code even for a todo app. When I do stuff like this, I do not use bleeding edge technology, but I consider everything stable that at least all major browsers can do for like half a year. Within that time, people should have upgrade at least.
It would be interesting to know what the current state is on modern browsers regarding cross domain sharding with explicitly not set domain attribute.

Setting the Domain attribute to the full subdomain should solve that, right?

Setting the Domain attribute to the full subdomain should solve that, right?

That's how my cookies currently look like, when I omit the domain attribute. I guess this should fix the issue with the sub domains. But I can't guarantee that, I have not tested it yet.

Well, this would solve the issue until you don't have sub-sub-domains. :p
Let's say I have my app running at iam.example.com and the domain is set to exactly that, then evil.example.com would not get the cookie, but sub.iam.example.com would get it again afaik. But yes, would need to set up some test env.

@mholt if you have a few cycles to look at this, I'd appreciate your take. I think it's a fairly serious security concern and might be worth handling in Caddy's forward_auth implementation.
For brevity you should be able to jump in just a few messages back, starting with https://github.com/sebadob/rauthy/issues/337#issuecomment-2075449429

Usually you can strip headers without issues and you even must strip certain headers when doing reverse-proxying. I don't know a reverse proxy that can't do it.
I just had a second look and when I omit the domain with Rauthy's cookies, the browser does set the domain to "the current one" but also set's the HostOnly flag. I am not sure though if that means sub-sub-domains can see it or not.

Oh duh. For some reason I was thinking you would need to strip the headers in the forward auth configuration itself, but it should work fine to just strip them as you normally would in the reverse proxy.

@anderspitman That's a lot to read for context :sweat_smile: (even with the summary comment). If the crux of the matter is this:

Not sure if any of them allow you to strip headers when using forward auth, since the main point is to append headers.

Then no worries as you discovered (probably). Caddy lets you strip headers by prefixing the header field name with -.

@mholt no worries. Thanks for checking in!

@anderspitman It should work as expected in all current browser. Taken from the MDN docs:

If you omit the domain completely, not even sub-sub-domains could access it from these docs.

Oh duh. For some reason I was thinking you would need to strip the headers in the forward auth configuration itself, but it should work fine to just strip them as you normally would in the reverse proxy.

Proxies often need to strip certain outgoing headers  to prevent leakage of internal data, and they need to adjust hops and so on, but they should care this without you needing to think about this.

Seems like the double cookie thing might work then.

Could this possibly be solved by having an additional cookie per domain? So basically when you navigate to app1.example.com for the first time, it sets a simple cookie with no Domain attribute, which means it won't be shared with other subdomains. Then when making requests, the auth server only sets the UserID header if the request contains both an auth cookie and a valid cookie for the exact domain as the host.

Yes I mean, that can be solved, but then we are at native OIDC per app again. Basically have separate clients and signed tokens, for which you validate the aud / azp, so you can use a token for client a only there and client b would reject it.
Trusted headers should only be used if you are sure that there are no hostile hosts / apps.
I mean you could start setting different headers for different hosts, which they look for and keep this "secret" somehow. But tbh, then implementing OIDC is far easier at some point. :D
All other idea's how you could solve a "be authenticated immediately everywhere" would involve oauth / oidc and client side app modification. But when you need to modify your client app anyway, you could just implement OIDC directly and have it way more secure.

I think you're describing something a lot more complicated than what I have in mind. Again, I'm likely missing something.
The goal is to determine if a request is cross-origin between two apps, and only include trusted headers for requests that are not cross-origin. If you can do that, you should be protected against evil/exploited apps.
My proposal for detecting this is to set a simple cookie for every app. Doesn't need to have any signing or randomness. Just needs to exist. Those cookies will not have the domain set, so they should never be sent cross origin. When the auth server is asked to validate a request, it considers it valid as long as it has a .example.com auth cookie, but it only sets the trusted auth headers if it has both a .example.com cookie and an app1.example.com cookie, which indicates the request is not cross-origin.
Note that cross origin requests are still considered valid, but the trusted auth headers will not be set, so the upstream app will reject the requests unless they contain proper authorization tokens of some sort. But that falls under the normal security of the app, not a systemic problem like what I'm trying to address.

Ahh now I got what you mean.
Yes I guess this would be a solution, if you need to host your apps under the same sub domain. That's actually pretty clever.

Actually I think I'm overcomplicating it. I think checking the Origin header should be sufficient.

https://wiki.mozilla.org/Security/Origin

Outside the browser all bets are off. My cookie plan would also be easily thwarted in that case. But CSRF isn't really a concern outside the browser.
You shouldn't need to worry about evil app backends making requests. They won't even have access to the .example.com auth cookies because those will be stripped by the reverse proxy.

Hmm right. As long as the other evil apps never get your cookies, then origin header checking would be fine.
Then you only need to worry about really making sure to strip headers correctly.
But then I am asking myself again, wouldn't it be easier to just use OIDC everywhere? Then you would not need to worry about all of this.

The point is that the apps don't have to worry about this. You implement it once in your auth server and every app gets authentication for free, needing only to trust the headers provided.


[ISSUE #2180] Start namesrv failed using obsolete jvm options

What is the purpose of the change
#2180
Brief changelog
choose diffrent java gc options base on java version
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Thanks, there are still many improvements for G1 config. I will merge this PR first and continue to polish here. @RongtongJin @duhengforever

linux 安装openjdk version "11.0.8" 2020-07-14 LTS
nohup sh bin/mqnamesrv & 无法启动
报错：
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Unrecognized VM option 'UseCMSCompactAtFullCollection'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.


Improved user feedback

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Fat finger syndrome..