Possible problem on timed_transitions

Consider these transitions:
fsm.add_timed_transition(&state_waitfortriggerack, &state_waitfortriggerack, 3000, NULL);
fsm.add_transition(&state_waitfortriggerack, &state_rpinotworking, TIMEOUT, NULL);

and then I am doing a:
fsm.trigger(TIMEOUT);

I was expecting the loop of the 3000ms to stop, but it keeps calling the entry function of state_waitfortriggerack. The fsm.trigger(TIMEOUT) is actually changing the state and doing its thing, but the timed transition keeps repeating.

Ping, anyone has managed to use timed transactions to same state?

Hi, this is what I used and it seemed to work:
/* ----------------------------------------------------- */
#include "Fsm.h"
#include "Button2.h";
/* ----------------------------------------------------- */
#define TIMEOUT 1000
/* ----------------------------------------------------- */
void waitfortriggerack() {
  Serial.println("waitfortriggerack");
}

void rpinotworking() {
  Serial.println("rpinotworking");
}
/* ----------------------------------------------------- */
State state_waitfortriggerack("wait", &waitfortriggerack);
State state_rpinotworking("rpinotworking", &rpinotworking);

Fsm fsm(&state_waitfortriggerack);
Button2 btn = Button2(D0);
/* ----------------------------------------------------- */
void btn_released(Button2& btn) {
  Serial.println("CLICK");
  fsm.trigger(TIMEOUT);
}
/* ----------------------------------------------------- */
void setup() {
  Serial.begin(9600);
  delay(50);

  fsm.add_timed_transition(&state_waitfortriggerack, &state_waitfortriggerack, 3000, NULL);
  fsm.add_transition(&state_waitfortriggerack, &state_rpinotworking, TIMEOUT, NULL);

  btn.setReleasedHandler(btn_released);
}
/* ----------------------------------------------------- */
void loop() {
  btn.loop();
  fsm.run_machine();
}
/* ----------------------------------------------------- */

This is the output - I pressed the button after a little while:
waitfortriggerack
waitfortriggerack
waitfortriggerack
waitfortriggerack
waitfortriggerack
CLICK
rpinotworking

After the button press, the machine keeps the RPI state.

In case anyone runs into this, there's a limitation that isn't super obvious in the FSM code as it is implemented. The state isn't recorded as being changed until after the on_enter is called on the new state. As a result, if you use trigger to send an event from the start of the new state, it won't find a transition from that new state to the next state.
i.e., if you have three states "one", "two" and "three", and you have a timed transition from two back to itself, it'll get called forever if you try to do a trigger(three) in the start method for two.
I fixed it in a local copy of the library by moving the m_current_state = transition->state_to; line in make_transition above the transition->state_to->on_enter() call.
Its a behavioral change, but because there's no real documentation for the library its unclear if the prior behavior is as-intended or not.


Fix typo

Description
Fixes a typo in contributing page

Testing

Checklist
Things need to be done before this Pull Request can be merged.

[ ] Code is well-formatted and adheres to project style guidelines: cargo fmt
[ ] Code does not produce any clippy warnings cargo clippy

Thanks @Alvin-HZ


Process exits when notify-send not installed

On Linux, if the notify-send package isn't installed, using native-notifier causes the parent process (in my case brunch) to crash/exit.  Adding an error handler to the spawned process solves the problem.

Ok duh, you've fixed this in master.  The version of native-notifier used by the latest brunch doesn't have this fix though.


[Sema] NFC: Assert that pattern binding entry has type during validation

Slava okayed this change via the forum.

@swift-ci please smoke test and merge


Cannot create shell with custom certificates

Describe the bug
I'm using devenv on a work machine, where custom certificates are injected with Zscaler. I've set security.pki.certificates in my nixos config, which has generally addressed the issue (I can run general nix commands without issue). But trying to create a shell with devenv does not work:
> RUST_BACKTRACE=full devenv shell
• Building shell ...
thread 'main' panicked at devenv/src/command.rs:284:26:
Failed to get cache: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("cachix.org")), port: None, path: "/api/v1/cache/devenv", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 2, error: "unable to get issuer certificate" })) }
stack backtrace:
   0:     0xaaaabf90ba54 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h75dbf07dc864c0bd
   1:     0xaaaabf9472f4 - core::fmt::write::h0c7806eb64ac0461
   2:     0xaaaabf8f8f48 - std::io::Write::write_fmt::ha53e993f45f43425
   3:     0xaaaabf90b87c - std::sys_common::backtrace::print::hb1055ea5fa282e57
   4:     0xaaaabf90c38c - std::panicking::default_hook::{{closure}}::h9bc2adbd2b89efe9
   5:     0xaaaabf90c09c - std::panicking::default_hook::h55811f2d21d601c1
   6:     0xaaaabf90c944 - std::panicking::rust_panic_with_hook::h976eed390f682de4
   7:     0xaaaabf90c7fc - std::panicking::begin_panic_handler::{{closure}}::h98a1c1b557857ce3
   8:     0xaaaabf90bca4 - std::sys_common::backtrace::__rust_end_short_backtrace::h70cf8b31899cdba4
   9:     0xaaaabf90c59c - rust_begin_unwind
  10:     0xaaaabf5f6734 - core::panicking::panic_fmt::h0971f4e715269fcb
  11:     0xaaaabf5f6aa8 - core::result::unwrap_failed::h73b70ef0c4d97d03
  12:     0xaaaabf5fefd0 - devenv::command::<impl devenv::App>::prepare_command::hf8c09b348137dbe1
  13:     0xaaaabf5fbb64 - devenv::command::<impl devenv::App>::run_nix::hcd20f469f09515a0
  14:     0xaaaabf60c2a4 - devenv::App::get_dev_environment::h4848001b2f84f86b
  15:     0xaaaabf608704 - devenv::App::prepare_shell::h83197b9cd4e9edf5
  16:     0xaaaabf602b4c - devenv::main::h41255c1257229c5d
  17:     0xaaaabf669af0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h14f1e4a0403543e0
  18:     0xaaaabf6583d8 - std::rt::lang_start::{{closure}}::hd0e66ad970d06b52
  19:     0xaaaabf90c450 - std::panicking::try::h1ea181fc1a5f2df1
  20:     0xaaaabf914f00 - std::rt::lang_start_internal::h739eda15a5578387
  21:     0xaaaabf619254 - main
  22:     0xffff83e2c500 - __libc_start_call_main
  23:     0xffff83e2c5d8 - __libc_start_main@@GLIBC_2.34
  24:     0xaaaabf5f6e30 - _start
  25:                0x0 - <unknown>
✔ Building shell in 0.5s.

To reproduce
Not sure how to provide generally reproducible info, b/c it's a custom certificate
Version
devenv 1.0.5 (aarch64-linux)

@ptonner, could you please give this branch a try: https://github.com/cachix/devenv/pull/1238

thanks for the fast turnaround! unfortunately it didn't work for me
> nix shell github:cachix/devenv/use-native-cert-store
> which devenv
/nix/store/063n1qkghq66y6rj1aciq27b4sm8x1rk-devenv-1.0.5/bin/devenv
> RUST_BACKTRACE=full devenv shell
• Building shell ...
thread 'main' panicked at devenv/src/command.rs:286:51:
Failed to get cache: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("cachix.org")), port: None, path: "/api/v1/cache/devenv", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 2, error: "unable to get issuer certificate" })) }
stack backtrace:
   0:     0xaaaad417f0c4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hcea0e942556235da
   1:     0xaaaad41a3914 - core::fmt::write::hc71024ef34e28312
   2:     0xaaaad415b348 - std::io::Write::write_fmt::h41d92926d93a7bb9
   3:     0xaaaad417eeec - std::sys_common::backtrace::print::h7819a8115109f212
   4:     0xaaaad4180218 - std::panicking::default_hook::{{closure}}::hb0ffd4171483a401
   5:     0xaaaad417ff44 - std::panicking::default_hook::hb5291fb36b5b8917
   6:     0xaaaad41807d8 - std::panicking::rust_panic_with_hook::hc474b468a8118958
   7:     0xaaaad4180694 - std::panicking::begin_panic_handler::{{closure}}::hf357bf09ed76c036
   8:     0xaaaad417f314 - std::sys_common::backtrace::__rust_end_short_backtrace::h904e6ea50bec8cae
   9:     0xaaaad4180434 - rust_begin_unwind
  10:     0xaaaad3dc275c - core::panicking::panic_fmt::h278e3cddb1b6e4ac
  11:     0xaaaad3dc25e0 - core::result::unwrap_failed::h82776741042be38e
  12:     0xaaaad3dca4f8 - devenv::command::<impl devenv::App>::prepare_command::h417cc773b3aa99b8
  13:     0xaaaad3dc7dd0 - devenv::command::<impl devenv::App>::run_nix::h698e48742579b7e5
  14:     0xaaaad3dd75e0 - devenv::App::get_dev_environment::hd8f2e09a53fb53c1
  15:     0xaaaad3dd3a10 - devenv::App::prepare_shell::h0546863ef40deaf0
  16:     0xaaaad3dcdd10 - devenv::main::h614e3978473ac29d
  17:     0xaaaad3e20808 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf1ffc8aafce73f8f
  18:     0xaaaad3e3ca64 - std::rt::lang_start::{{closure}}::h2236a5465481f2f6
  19:     0xaaaad41802dc - std::panicking::try::hb14e0e162f35f3ef
  20:     0xaaaad417b328 - std::rt::lang_start_internal::h6fa7342305762cae
  21:     0xaaaad3de44a8 - main
  22:     0xffffb30fc500 - __libc_start_call_main
  23:     0xffffb30fc5d8 - __libc_start_main@@GLIBC_2.34
  24:     0xaaaad3dc2bb0 - _start
  25:                0x0 - <unknown>
✔ Building shell in 0.5s.

Just FYI, this may have been an error on my part - I hadn't set up all the certificates. Not sure why base nix was able to work around this, but things seem to be working for me now. Will close unless something else shows up


feat(source): support JSON schema addtionalProperties (map)

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

#17110  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @xxchan and the rest of your teammates on  Graphite

kindly ping @xiangjinwu
Although our support for JSON schema looks fragile, I feel this one-line change is no-brainer. 🥶


attempt to concatenate local 'cb_server_name' (a nil value)

Hello,
I followed all steps in order to make it to work and everything was just fine.
After I added the configuration text to my default.conf file I'm getting the following error when I visit the page in my browser.
error.log
017/04/10 15:36:12 [error] 26805#0: *1 lua entry thread aborted: runtime error: /usr/local/openresty/nginx/conf/access.lua:18: attempt to concatenate local 'cb_server_name' (a nil value)
stack traceback:
coroutine 0:
	/usr/local/openresty/nginx/conf/access.lua: in function </usr/local/openresty/nginx/conf/access.lua:1>, client: <IP_ADDRESS>, server: _, request: "GET / HTTP/1.1", host: "<IP_ADDRESS>"

Here's the whole configuration file:
server {
    # Listen on port 80.
    listen 80 default_server;
    listen [::]:80 default_server;

    # The document root.
    root /usr/local/openresty/nginx/html/default;

    # Add index.php if you are using PHP.
    index index.html index.htm;

    # The server name, which isn't relevant in this case, because we only have one.
    server_name _;

    ### CUSTOM: Nginx Google Auth
    set $ngo_client_id      "-removed-";
    set $ngo_client_secret  "-removed-";
    set $ngo_token_secret   “-removed-”;
    set $ngo_secure_cookies "true";

    access_by_lua_file "/usr/local/openresty/nginx/conf/access.lua";
    ### CUSTOM: Nginx Google Auth

    # When we try to access this site...
    location / {
        # ... first attempt to serve request as file, then as a directory,
        # then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }

   # Reverse proxy for Jenkins URI.
    location /jenkins {
        proxy_pass  http://<IP_ADDRESS>:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

I would appreciate any help, thanks.

As far as I can tell by reading access.lua, this bit from the README is wrong:

$ngo_callback_host The host for the callback, defaults to first entry in the server_name list (e.g supersecret.net).

I don't see any mechanism for it to default to anything.  I recommend setting it in your config file.

Hello @exowaucka
You are right. In fact I managed to solve this by just adding the "or" option to the cb_server name:
local cb_server_name = ngx.var.ngo_callback_host or server_name


first run of Parquet schema read much slower than BlobClient?

First run this took 40 seconds +
#slow
with abfs.open('example-container/test.parquet') as f:
    schema = pq.read_schema(f)

This which I'm pretty sure downloads the entire blob takes under half a second.
#fast
import io
with io.BytesIO() as byte_stream:
    myBlobServiceClient.get_blob_client(
                    'example-container',
                    'test.parquet').download_blob().readinto(byte_stream)
    schema = pq.read_schema(byte_stream)

but then if I run the abfs line again (even with a diferent file) it's fast! 101ms
#now it's fast
with abfs.open('example-container/test2.parquet') as f:
    schema = pq.read_schema(f)

Am I doing something wrong? Is there some odd initialize or scan that's making the initial use of abfs so slow?

You will experience this delay only if your container contains a lot of blobs. The reason behind this is unnecessary caching of blob details from the whole container presented in
https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L751-L759
This should probably be fixed as it affects both .open() and .info() methods if the container is big. One can retrieve the file contents with .cat() which is not affected as it's not calling .info() but it's not ideal.

This is related to #248 because the info method is called for .open() and would be fixed by PR #321.

Well, I cloned and installed from the repo and I am now running adlfs-2022.10.0+4.gf15c37a . My original example of :
with abfs.open('example-container/test2.parquet') as f:
    schema = pq.read_schema(f)

still takes 40 seconds on first run when simply reading in the entire blob with get_blob_client takes 1-2 seconds.
abfs.info('example-container/test2.parquet')

Similarly takes 40 seconds on first run. (then fast after that.)
The code in #360 certainly seems like it'd improve info performance but I'm specifying the full path and it's still slow first run. F eels like it's scanning an entire container.

@astrowonk -- Can you share a reproducer?  I've tried this with single file from a partitioned parquet dataset and I'm getting < 0.5 seconds for the open operation, and <0.001 second for the info() operation.

@astrowonk -- Can you share a reproducer?  I've tried this with single file from a partitioned parquet dataset and I'm getting < 0.5 seconds for the open operation, and <0.001 second for the info() operation.

I'll try; I need a public (?) container with lots of files. This is happening with a container with probably thousands of files in the container. How many other blobs are in your test container?

Hi @hayesgb. It's not really possible to make a simple test because you need a lot of files in the container to see performance problem. However if you can get the debugger to print DEBUG level statements (I have had no luck so I had to switch line 920  to a print) you'll see that it's calling ls on the entire container for any .info. The original fix in #321 would have re-ordered the calls such that this line:
out = await self._ls(path, invalidate_cache=invalidate_cache, **kwargs)
would run first. But the actual merged code in #360 still does this first:
        out = await self._ls(
            self._parent(fullpath), invalidate_cache=invalidate_cache, **kwargs
        )

self.parent(fullpath) in the parent container, so it is now calling ls on the entire container which will be extremely slow if there are lots of files. The original re-ordering of #321 is preferred, where the first _ls call is to the requested path without wrapping in self._parent
Honestly if I'm calling info on a specific path, I don't quite understand why we'd ever want to ls the entire container in response.. even if the full path was missing. I'd just raise a path not found error.

Adding more details here @hayesgb. The ordering from https://github.com/fsspec/adlfs/pull/321 is preferred over https://github.com/fsspec/adlfs/pull/360 as @astrowonk mentioned but there's another issue even with the corrected ordering.
It's when you call AzureBlobFileSystem.info() for cached blobs
    def info(self, path, refresh=False, **kwargs):
        try:
            fetch_from_azure = (path and self._ls_from_cache(path) is None) or refresh
        except Exception:
            fetch_from_azure = True
        if fetch_from_azure:
            return sync(
                self.loop,
                self._info,
                path,
                refresh,
                version_id=kwargs.get("version_id"),
            )
        return super().info(path)

this will call AbstractFileSystem.info() which contains the doomed ls call on (possibly) large container and we're doing ls on collection for every single blob again 😞
https://github.com/fsspec/filesystem_spec/blob/master/fsspec/spec.py#L614

My use case of
with abfs.open('example-container/test2.parquet') as f:
    schema = pq.read_schema(f)

is much faster now. Thank you!
My bigger picture use case which is something like:
import pyarrow.dataset as ds
dataset = ds.dataset(myblobs,filesystem=abfs,format='parquet',schema=schema,exclude_invalid_files=False)
conn = duckdb.connect(":memory:")

and then run some aggregation with a duckdb query is still very slow on the first run. I specify the paths to 60 blobs and first run is 1 minute, second run is 10 seconds. So something about dataset or duckdb is causing (I think) yet another crawl through the entire container again. When I can figure out a minimal reproduction, I'll make a new issue.


[Feature] add name_loader in eval metric

Fixes #20  and add dataset name in the eval metrics.

Hey @huguesva ,
the wandb table logging bug, log.eval_epoch_freq is missing from this change. Can you please add them?
Updated the line 254 in base.py and LogConfig init in config.py:
if (self.epoch%self.config.log.eval_epoch_freq == 0) or self.config.log.eval_each_epoch:

line 467 of stable_ssl.base.py needs to be this, i.e., the 'index' col needs to be removed:
table = wandb.Table(columns=["epoch", name])


[ACM-12773] Fix flaky test

CI upgrade to go 1.21 fails due to unsorted list from a map: https://github.com/openshift/release/pull/54478
This PR fixes this.

/cherrypick release-2.11


You tube Video

"Watch this video to see how to use the Expense Report sample."
Video is marked as private.  Unable to view even after signing it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 379862e0-89e8-8e6b-b686-77d6836f0434
Version Independent ID: 29fc7f25-b6b5-26d2-69f3-eada7fff64b3
Content: Install and configure the Expense Report sample for canvas apps - PowerApps
Content Source: powerapps-docs/maker/canvas-apps/expense-report-install.md
Service: powerapps
GitHub Login: @yijw2017
Microsoft Alias: yijw

YouTube link is updated. :)


Don't indent module block contents

Reading through @chriselrod's evaluation of which existing julia-mode issues are resolved by julia-ts-mode (thanks @chriselrod!), I thought I'd file issues for the few problems that you might have some control over.
It appears from the testing of https://github.com/JuliaEditorSupport/julia-emacs/issues/12, that julia-ts-mode tries to indent the contents of a module which is not in keeping with the style guide from the Julia manual.
EDIT: I thought there were a couple of other things I saw, but re-reading the comment this is the only one that jumps out to me. :shrug:

Nice catch! Thanks @non-Jedi! It should be correct now :)


OpenPBS job queue hang

When running jobs in OpenPBS 20.0.1 we have found what looks to be partly related to #1301 & #1473
Currently our PBS implementation uses job history output, when running jobs in PBS with reframe the job will error out due to this:
[CMD] 'qstat -f 1015.sch1 1016.sch1 1017.sch1 1018.sch1'
[  FAILED  ] Ran 0/2 test case(s) from 3 check(s) (0 failure(s))
[==========] Finished on Wed Apr 14 16:32:55 2021
/usr/bin/reframe: run session stopped: job scheduler error: qstat failed with exit code 35 (standard error follows):
qstat: 1015.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1016.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1017.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1018.sch1 Job has finished, use -x or -H to obtain historical job information

if we disable the job history support (not ideal), the job just hangs infinitely regardless of if the PBS jobs fail or run successfully
Reframe version:
reframe --version 3.5.0
SLE 15 SP2
lmod
OpenPBS 20.0.1

Hi @cblackworth I suspect where the problem comes. ReFrame uses qstat -f job1 job2 ... to query the status of pending jobs. Do you see messages such as Job state not found (job info follows): in the log file? (or simply rerun ReFrame with -vvv) Could you copy those messages here?

here's the output with logging set to debug (running with -vvv didn't output any more details)
[  FAILED  ] Ran 0/2 test case(s) from 3 check(s) (0 failure(s))
[==========] Finished on Wed Apr 21 20:04:02 2021
/usr/bin/reframe: run session stopped: job scheduler error: qstat failed with exit code 35 (standard error follows):
qstat: 1037.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1038.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1039.sch1 Job has finished, use -x or -H to obtain historical job information
qstat: 1040.sch1 Job has finished, use -x or -H to obtain historical job information

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/reframe/frontend/cli.py", line 968, in main
    runner.runall(testcases, restored_cases)
  File "/usr/lib64/python3.6/site-packages/reframe/frontend/executors/__init__.py", line 410, in runall
    self._runall(testcases)
  File "/usr/lib64/python3.6/site-packages/reframe/frontend/executors/__init__.py", line 473, in _runall
    self._policy.runcase(t)
  File "/usr/lib64/python3.6/site-packages/reframe/frontend/executors/policies.py", line 365, in runcase
    self._poll_tasks()
  File "/usr/lib64/python3.6/site-packages/reframe/frontend/executors/policies.py", line 409, in _poll_tasks
    part.scheduler.poll(*part_jobs)
  File "/usr/lib64/python3.6/site-packages/reframe/core/schedulers/pbs.py", line 204, in poll
    f'qstat failed with exit code {completed.returncode} '
reframe.core.exceptions.JobSchedulerError: qstat failed with exit code 35 (standard error follows):
qstat: 1037.sch1 Job has finished, use -x or -H to obtain historical job information

If we turn off job history which is needed for other uses at the site I get this error when the job hangs and never closes (reframe will just run indefinitely until it times out) basically hanging at "Entering stage: run_complete"
Reached concurrency limit for partition 'testbed:login': 8 job(s)
Polling 8 task(s) in 'testbed:login'
[S] local: pid 66859 already dead or assigned elsewhere
[S] local: pid 67056 already dead or assigned elsewhere
[S] local: pid 67170 already dead or assigned elsewhere
[S] local: pid 67773 already dead or assigned elsewhere
[S] local: pid 68068 already dead or assigned elsewhere
[S] local: pid 68180 already dead or assigned elsewhere
[S] local: pid 68765 already dead or assigned elsewhere
Entering stage: run_complete
Entering stage: run_wait
Removing task from the running list: ('HelloMultiLangTest_c', 'testbed:login', 'builtin')
Entering stage: run_complete
Entering stage: run_wait
Removing task from the running list: ('HelloMultiLangTest_c', 'testbed:login', 'cray')
Entering stage: run_complete
Entering stage: run_wait
Removing task from the running list: ('HelloMultiLangTest_cpp', 'testbed:login', 'gnu')
Entering stage: run_complete
Entering stage: run_wait
Removing task from the running list: ('HelloTest', 'testbed:login', 'builtin')
Polling 4 task(s) in 'testbed:pbs'
[CMD] 'qstat -f 1047.sch1 1048.sch1 1049.sch1 1050.sch1'
[S] pbs: Return code is 153: jobids not known by scheduler, assuming all jobs completed
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Scheduling test case ('HelloTest', 'testbed:login', 'cray') for running
Entering stage: compile
Copying /home/user/reframe_testbed/hello/src to stage directory
Symlinking files: []
[CMD] '/usr/share/lmod/lmod/libexec/lmod python show cray-pals'
[CMD] '/usr/share/lmod/lmod/libexec/lmod python load cray-pals'
[CMD] '/home/user/reframe_testbed/stage/testbed/login/cray/HelloTest/rfm_HelloTest_build.sh'

==> timings: setup: 0.006s compile: 1.287s run: 5.723s sanity: 0.000s performance: 0.000s total: 7.024s
Entering stage: cleanup
Copying test files to output directory
Removing stage directory
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete
Entering stage: run_complete

I think it is a bug of our PBS scheduler backend in the way we set the job final state. I'll propose a fix in a PR.

I pulled down your branch and put it on our test system and using the same config, same system that worked for 3.5.0 is throwing this error on tests for scheduler items:
/usr/bin/reframe: no modules system is set: module 'cray-pals' will not be loaded: check the 'modules_system' configuration parameter for your system

Hi @cblackworth, this should be just a warning... It simply says that you haven't set a modules system, so any attempt to load cray-pals will be ignored. You should set the modules_system configuration parameter to lmod or to whichever modules systems you are using. But for the scope of this issue, it should not play any role.

And you get this warning, because my branch that you pulled is beyond 3.5.3 :-)

This actually is set (the error wasn't there before the update). It causes the otherwise able to run jobs to fail to run as it can no longer compile my tests now.
to be clear, the only change to anything has been to update the version of reframe, everything else is unchanged from before when this test worked.
Reverting the version of reframe allows this test to work but fails to run on PBS.
So I am unable to verify if this patch works.
[  FAILED  ] Ran 3/3 test case(s) from 3 check(s) (1 failure(s), 0 skipped)
[==========] Finished on Thu Apr 22 17:14:36 2021

==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for HelloMultiLangTest_cpp
  * Test Description: HelloMultiLangTest_cpp
  * System partition: generic:default
  * Environment: builtin
  * Stage directory: /shared_home/user/reframe/stage/generic/default/builtin/HelloMultiLangTest_cpp
  * Node list:
  * Job type: local (id=None)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: compile
  * Rerun with '-n HelloMultiLangTest_cpp -p builtin --system generic:default -r'
  * Reason: build system error: I do not know how to compile a C++ program

Hi @cblackworth, regarding the error of not recognising your system after you pulled in my branch, it looks like a regression in the framework (see #1948), but meanwhile you can try to explicitly pass --system=testbed in order to test this PR.

That doesn't seem to work?
cblackworth@sc01:/shared_home/cblackworth/reframe_testbed> reframe -C ./settings.py -c ./hello/ -L -vvv --show-config=systems/0/name
Loading user configuration
Loading configuration file: './settings.py'
Detecting system
Looking for a matching configuration entry for system 'sc01'
Configuration found: picking system 'generic'
Selecting subconfig for 'generic'
Initializing runtime
Selecting subconfig for 'generic:default'
Initializing system partition 'default'
Selecting subconfig for 'generic'
Initializing system 'generic'
Initializing modules system 'nomod'
"generic"
cblackworth@sc01:/shared_home/cblackworth/reframe_testbed> reframe -C ./settings.py -r -C ./settings.py --system=testbed
/usr/bin/reframe: failed to load configuration: could not find a configuration entry for the requested system: 'testbed'
/usr/bin/reframe: Log file(s) saved in: '/tmp/rfm-rhrq61ed.log'
cblackworth@sc01:/shared_home/cblackworth/reframe_test>

That last command should be "reframe -C ./settings.py -c ./hello/ --system=testbed -r"

Hmm, your case doesn't seem to be the same as in #1948. May I ask how do you install ReFrame? One strange thing that I see and can't understand is that your settings file does not define the generic system and I'm wondering how ReFrame picks it up? It shouldn't be the case...

It looks like there was a lingering module in python somewhere. I re ran this using just the bootstrap script and it worked.
For the github releases, is there a "correct" install that doesn't use bootstrap.sh? (this system isn't able to access the internet)

I am now running in to this odd behavior and I'm wondering if its related? This error only shows up on this one job and only when used with PBS, the local only job of the same test runs fine:
(both the gnu and cray tests fail, hence the changing test name, same error though)
caught builtins.FileNotFoundError: [Errno 2] No such file or directory: '/home/cblackworth/stage/testbed/pbs/cray/HelloTest/rfm_HelloTest_job.err'

cblackworth@sch1:~> ls -al /home/cblackworth/stage/testbed/pbs/cray/HelloTest/rfm_HelloTest_job.err
-rw------- 1 cblackworth users 0 Apr 28 14:12 /home/cblackworth/stage/testbed/pbs/cray/HelloTest/rfm_HelloTest_job.err

Further details:
[     FAIL ] ( 4/15) HelloTest on testbed:pbs using gnu [compile: 0.752s run: 0.818s total: 1.605s]
==> test failed during 'sanity': test staged in '/home/cblackworth/stage/testbed/pbs/gnu/HelloTest'

------------------------------------------------------------------------------
FAILURE INFO for HelloTest
  * Test Description: HelloTest
  * System partition: testbed:pbs
  * Environment: cray
  * Stage directory: /home/cblackworth/stage/testbed/pbs/cray/HelloTest
  * Node list: x1000
  * Job type: batch job (id=1116.sch1.hsn0.cm.testbed.imi.alcf.anl.gov)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: cleanup
  * Rerun with '-n HelloTest -p cray --system testbed:pbs -r'
  * Reason: file not found error: [Errno 2] No such file or directory: '/home/cblackworth/stage/testbed/pbs/cray/HelloTest/rfm_HelloTest_job.err'
Traceback (most recent call last):
  File "/reframe/reframe/frontend/executors/__init__.py", line 269, in _safe_call
    return fn(*args, **kwargs)
  File "/reframe/reframe/core/hooks.py", line 36, in _fn
    func(obj, *args, **kwargs)
  File "/reframe/reframe/core/pipeline.py", line 97, in _wrapped
    return fn(*args, **kwargs)
  File "/reframe/reframe/core/pipeline.py", line 1665, in cleanup
    self._copy_to_outputdir()
  File "/reframe/reframe/core/pipeline.py", line 1618, in _copy_to_outputdir
    self._copy_job_files(self._job, self.outputdir)
  File "/reframe/reframe/core/pipeline.py", line 1612, in _copy_job_files
    shutil.copy(stderr, dst)
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/cblackworth/stage/testbed/pbs/cray/HelloTest/rfm_HelloTest_job.err'

Hi @cblackworth is this failure persistent? It really looks like #1394. There, we were on a Lustre filesystem and I'm not sure if there was somehow glitch, but we haven't seen that error since a while ago.

Hi @cblackworth, I'll close this issue now. Feel free to reopen it, if it persists.


[POC] MenuButton - Option 1 implementation

DO NOT MERGE
This is a playground for evaluating the API of a MenuButton component.
It implements Option 1 from https://github.com/mui/material-ui/issues/32088#issuecomment-1216305769
Playground: https://codesandbox.io/s/base-cra-typescript-forked-byxysy?file=/src/App.tsx

I'd like to focus on implementing other options first so we can evaluate them before going deeper into any of them. Ultimately, after we choose a winning option, I'll create both a component and a hook.


removing not needed annotations

I take it these annotations are used for helm management. But it is best to let helm add them in case of helm changing them and using different ones (which I believe it has already done). Also, best to let helm add the namespaces. Who knows, maybe they have the options for multi-namespaces releases in the future

i did not get the motivation to remove those things

These are things added by helm, so we are hard-coding them. It can be left as is, but helm adds app.kubernetes.io/managed-by: Helm as well as the namespace. I can close the PR, and it can be changed if needed (for example, if helm changes the annotation)


Error spawning language server

When I open an .cs file it shows this error: /usr/share/nvim/runtime/lua/vim/lsp/rpc.lua:800: Spawning language server with cmd: `{ "-z", "--hostPID", "46227", "DotNet:enablePackageRestore=false", "--encoding", "utf-8", "--languageserver" , "FormattingOptions:EnableEditorConfigSupport=true", "Sdk:IncludePrereleases=true" }` failed. The language server is either not installed, missing from PATH, or not executable.
Another question, where do I put the configurations mentioned on the docs?
this part:
{
    lsp = {
        -- Sets if you want to use omnisharp as your LSP
        omnisharp = {
        -- When set to false, csharp.nvim won't launch omnisharp automatically.
            enable = true,
            -- When set, csharp.nvim won't install omnisharp automatically. Instead, the omnisharp instance in the cmd_path will be used.
            cmd_path = nil,
            -- The default timeout when communicating with omnisharp
            default_timeout = 1000,
            -- Settings that'll be passed to the omnisharp server
            enable_editor_config_support = true,
            organize_imports = true,
            load_projects_on_demand = false,
            enable_analyzers_support = true,
            enable_import_completion = true,
...

Did you work this out or do you still need help


Setting socket options immediately after connecting may fail

I'm seeing this issue on OS X, but I believe it could happen on Linux as well.  I'm creating a TCP socket through mio::tcp::TcpStream::connect, and then immediately calling TcpStream::set_nodelay.  About 20% of the time, the set_nodelay call fails with error 22 (EINVAL).  dtruss reveals that in these cases, the initial connect system call failed with error 36 (EINPROGRESS), but mio is swallowing these erorrs.
I've worked around the issue for the moment by only setting socket options after the first non-error ready notification, but it would be nice if there were a way to set the option closer to the socket setup.  net2::TcpBuilder has no set_nodelay option.

At the very least, this is a documentation issue. How to create a custom TCP socket & setting nodelay should be documented.
I am less sure about the EINPROGRESS situation. On one hand, it could be nice to return that info, on the other, the odds are very strong that the connection will be in progress and the way to wait for the connection to become established is to wait for a readiness notification. My guess is that this is also a documentation issue.

Tracking this as a documentation issue.


[BUGFIX] modify the log average loss

Description
Modify the log average loss during training and validation proposed in #1102 .
Fixes #1102.
The log message looks like:
2020-01-08 08:53:50,968 - root - [Epoch 0 Batch 100/1043] loss=6.4215, ppl=614.9522, gnorm=0.7030, throughput=39.15K wps, wc=606.57K
2020-01-08 08:54:04,325 - root - [Epoch 0 Batch 200/1043] loss=5.8612, ppl=351.1320, gnorm=0.3394, throughput=43.74K wps, wc=584.18K
2020-01-08 08:54:17,698 - root - [Epoch 0 Batch 300/1043] loss=5.4339, ppl=229.0387, gnorm=0.3442, throughput=44.33K wps, wc=592.78K
2020-01-08 08:54:30,365 - root - [Epoch 0 Batch 400/1043] loss=5.0884, ppl=162.1247, gnorm=0.3193, throughput=45.10K wps, wc=571.26K
2020-01-08 08:54:42,646 - root - [Epoch 0 Batch 500/1043] loss=4.8152, ppl=123.3674, gnorm=0.3304, throughput=45.15K wps, wc=554.48K
2020-01-08 08:54:54,788 - root - [Epoch 0 Batch 600/1043] loss=4.6290, ppl=102.4127, gnorm=0.3167, throughput=44.99K wps, wc=546.26K
2020-01-08 08:55:07,879 - root - [Epoch 0 Batch 700/1043] loss=4.4968, ppl=89.7277, gnorm=0.3128, throughput=43.31K wps, wc=566.96K
2020-01-08 08:55:20,536 - root - [Epoch 0 Batch 800/1043] loss=4.3731, ppl=79.2917, gnorm=0.3120, throughput=43.57K wps, wc=551.40K
2020-01-08 08:55:33,160 - root - [Epoch 0 Batch 900/1043] loss=4.2842, ppl=72.5451, gnorm=0.3177, throughput=45.76K wps, wc=577.61K
2020-01-08 08:55:46,234 - root - [Epoch 0 Batch 1000/1043] loss=4.2068, ppl=67.1437, gnorm=0.3055, throughput=44.99K wps, wc=588.16K
2020-01-08 08:56:07,177 - root - [Epoch 0] valid Loss=4.0706, valid ppl=58.5939, valid bleu=3.06
2020-01-08 08:56:19,967 - root - [Epoch 0] test Loss=4.1586, test ppl=63.9789, test bleu=2.72
2020-01-08 08:56:19,973 - root - Save best parameters to gnmt_en_vi_l2_h512_beam10/valid_best.params
2020-01-08 08:56:33,670 - root - [Epoch 1 Batch 100/1043] loss=4.0797, ppl=59.1269, gnorm=0.3241, throughput=44.73K wps, wc=593.66K
2020-01-08 08:56:47,342 - root - [Epoch 1 Batch 200/1043] loss=4.0627, ppl=58.1332, gnorm=0.3102, throughput=46.20K wps, wc=631.62K
2020-01-08 08:57:00,902 - root - [Epoch 1 Batch 300/1043] loss=3.9731, ppl=53.1482, gnorm=0.3028, throughput=44.22K wps, wc=599.65K
2020-01-08 08:57:14,253 - root - [Epoch 1 Batch 400/1043] loss=3.8960, ppl=49.2063, gnorm=0.3091, throughput=43.87K wps, wc=585.70K
2020-01-08 08:57:27,026 - root - [Epoch 1 Batch 500/1043] loss=3.8288, ppl=46.0072, gnorm=0.3117, throughput=44.40K wps, wc=567.14K
2020-01-08 08:57:38,951 - root - [Epoch 1 Batch 600/1043] loss=3.7593, ppl=42.9191, gnorm=0.3261, throughput=43.74K wps, wc=521.63K
2020-01-08 08:57:52,448 - root - [Epoch 1 Batch 700/1043] loss=3.8015, ppl=44.7673, gnorm=0.3136, throughput=43.86K wps, wc=591.94K
2020-01-08 08:58:05,002 - root - [Epoch 1 Batch 800/1043] loss=3.7148, ppl=41.0497, gnorm=0.3299, throughput=44.91K wps, wc=563.74K
2020-01-08 08:58:18,252 - root - [Epoch 1 Batch 900/1043] loss=3.6045, ppl=36.7644, gnorm=0.3157, throughput=43.63K wps, wc=578.06K
2020-01-08 08:58:30,346 - root - [Epoch 1 Batch 1000/1043] loss=3.5059, ppl=33.3121, gnorm=0.3349, throughput=43.29K wps, wc=523.53K
2020-01-08 08:58:48,447 - root - [Epoch 1] valid Loss=3.4634, valid ppl=31.9260, valid bleu=7.37
2020-01-08 08:59:00,361 - root - [Epoch 1] test Loss=3.4913, test ppl=32.8275, test bleu=7.86
2020-01-08 08:59:00,367 - root - Save best parameters to gnmt_en_vi_l2_h512_beam10/valid_best.params
2020-01-08 08:59:13,555 - root - [Epoch 2 Batch 100/1043] loss=3.3069, ppl=27.3007, gnorm=0.3481, throughput=42.90K wps, wc=542.87K

The log message prior to this fix is:
2020-01-08 07:23:10,514 - root - [Epoch 0 Batch 100/1043] loss=6.2948, ppl=541.7368, gnorm=0.7030, throughput=38.01K wps, wc=606.57K
2020-01-08 07:23:23,880 - root - [Epoch 0 Batch 200/1043] loss=5.7232, ppl=305.8789, gnorm=0.3394, throughput=43.71K wps, wc=584.18K
2020-01-08 07:23:36,590 - root - [Epoch 0 Batch 300/1043] loss=5.2613, ppl=192.7247, gnorm=0.3442, throughput=46.64K wps, wc=592.78K
2020-01-08 07:23:49,343 - root - [Epoch 0 Batch 400/1043] loss=4.9213, ppl=137.1854, gnorm=0.3193, throughput=44.79K wps, wc=571.26K
2020-01-08 07:24:01,503 - root - [Epoch 0 Batch 500/1043] loss=4.6531, ppl=104.9109, gnorm=0.3304, throughput=45.60K wps, wc=554.48K
2020-01-08 07:24:14,218 - root - [Epoch 0 Batch 600/1043] loss=4.4185, ppl=82.9729, gnorm=0.3167, throughput=42.96K wps, wc=546.26K
2020-01-08 07:24:26,804 - root - [Epoch 0 Batch 700/1043] loss=4.3236, ppl=75.4571, gnorm=0.3128, throughput=45.05K wps, wc=566.96K
2020-01-08 07:24:39,098 - root - [Epoch 0 Batch 800/1043] loss=4.2010, ppl=66.7544, gnorm=0.3120, throughput=44.86K wps, wc=551.40K
2020-01-08 07:24:51,588 - root - [Epoch 0 Batch 900/1043] loss=4.0903, ppl=59.7581, gnorm=0.3177, throughput=46.25K wps, wc=577.61K
2020-01-08 07:25:04,737 - root - [Epoch 0 Batch 1000/1043] loss=4.0345, ppl=56.5169, gnorm=0.3055, throughput=44.73K wps, wc=588.16K
2020-01-08 07:25:25,812 - root - [Epoch 0] valid Loss=3.0219, valid ppl=20.5296, valid bleu=3.06
2020-01-08 07:25:38,293 - root - [Epoch 0] test Loss=3.1398, test ppl=23.0996, test bleu=2.72
2020-01-08 07:25:38,299 - root - Save best parameters to gnmt_en_vi_l2_h512_beam10/valid_best.params
2020-01-08 07:25:51,729 - root - [Epoch 1 Batch 100/1043] loss=3.8406, ppl=46.5551, gnorm=0.3241, throughput=46.13K wps, wc=593.66K
2020-01-08 07:26:05,829 - root - [Epoch 1 Batch 200/1043] loss=3.8650, ppl=47.7020, gnorm=0.3102, throughput=44.80K wps, wc=631.62K
2020-01-08 07:26:19,786 - root - [Epoch 1 Batch 300/1043] loss=3.8157, ppl=45.4108, gnorm=0.3028, throughput=42.97K wps, wc=599.65K
2020-01-08 07:26:32,657 - root - [Epoch 1 Batch 400/1043] loss=3.7100, ppl=40.8531, gnorm=0.3091, throughput=45.51K wps, wc=585.70K
2020-01-08 07:26:45,399 - root - [Epoch 1 Batch 500/1043] loss=3.6674, ppl=39.1519, gnorm=0.3117, throughput=44.51K wps, wc=567.14K
2020-01-08 07:26:57,125 - root - [Epoch 1 Batch 600/1043] loss=3.5434, ppl=34.5833, gnorm=0.3261, throughput=44.49K wps, wc=521.63K
2020-01-08 07:27:10,752 - root - [Epoch 1 Batch 700/1043] loss=3.5904, ppl=36.2497, gnorm=0.3136, throughput=43.44K wps, wc=591.94K
2020-01-08 07:27:23,379 - root - [Epoch 1 Batch 800/1043] loss=3.4860, ppl=32.6562, gnorm=0.3299, throughput=44.65K wps, wc=563.74K
2020-01-08 07:27:36,107 - root - [Epoch 1 Batch 900/1043] loss=3.4471, ppl=31.4081, gnorm=0.3157, throughput=45.42K wps, wc=578.06K
2020-01-08 07:27:48,146 - root - [Epoch 1 Batch 1000/1043] loss=3.3270, ppl=27.8557, gnorm=0.3349, throughput=43.49K wps, wc=523.53K
2020-01-08 07:28:06,191 - root - [Epoch 1] valid Loss=2.5888, valid ppl=13.3142, valid bleu=7.37
2020-01-08 07:28:18,089 - root - [Epoch 1] test Loss=2.6505, test ppl=14.1612, test bleu=7.86

Checklist
Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[x] Code is well-documented

@szhengac Does this change make sense to you?

Job PR-1103/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1103/1/index.html

See my comment on #1102.

Also, both scripts average losses by number of iterations in a single interval not the number of tokens.

I think it should not be labeled as BUGFIX.

The difference is whether to average over all the tokens or first average over the tokens and then average over the samples. I think we can obtain the "average over token" approach as the new standard.

Also, both scripts average losses by number of iterations rather than not the total number of tokens in a single interval.

No, train_transformer average over all the tokens. And I don't think the log metric is related to multi-gpu training.

Also, both scripts average losses by number of iterations rather than not the total number of tokens in a single interval.

I double checked the script, you are right. But how about line 176
avg_loss_denom += (tgt_seq.shape[1] - 1)

I think it should use
avg_loss_denom += (tgt_valid_length - 1).sum().asscalar()

Otherwise, validation loss will be much smaller than it should be.

The difference is whether to average over all the tokens or first average over the tokens and then average over the samples. I think we can obtain the "average over token" approach as the new standard.

I experiment with both options, it appears that the "average over token" approach has slight higher loss than the other option. From this perspective, I think current option looks better (it displays lower loss).
BTW, my inspiration for current PR comes from the usage of mxnet EvalMetric. It accumulates both the self.num_inst and self.sum_metric, it gets the current metric by dividing self.sum_metric / self.num_inst.

Another point we need to consider is that the number of tokens could be very large. There will be some potential numerical problems.
The original Line 176 is correct because it used ‘mean’ rather than sum to get the loss.

Another point we need to consider is that the number of tokens could be very large. There will be some potential numerical problems.
The original Line 176 is correct because it used ‘mean’ rather than sum to get the loss.

For line176, my point is why using tgt_seq.shape[1] instead of tgt_valid_length.mean()? During training, denom is computed from the tgt_valid_length.

@liuzh91 The initial implementation should be correct. Adding tgt_seq.shape[1] is the correct behavior because we are dividing by tgt_seq.shape[1]


Inconvenience due to changing visibility of Repository to Public

Hey there,
I am very sorry for the inconvenience that I have caused to you.
Yesterday, I have made this repository public due to which all the created fork of this repository isn't link to this repository. Due to which,  you won't be able to create PR from the fork repository to base repository.
I would like to request all of my @Contributor to create a fork of this repository and work on the new forked repository.
For those who have already open PR, I am sorry to trouble you guys as you would have to do what you have did from the new forked repository.
And for those who haven't created PR but working on their forked repository, create a fork as mentioned in above message and just copy the changes you have made to old command repository to new forked repository.
I am so sorry for the inconvenience that I have created.
Thank you!
Best regards,
Susheel Thapa

If you have any question related to this topic you can just reply to this.

@suravshrestha @SuprimDevkota @Sandesh-Pyakurel @Sandukkk @SauravKumarMahato @PRASUN-SITAULA
@Rajkumarpaneru18
Please create a new fork from this repository before contributing and delete the previous forked but make sure you have keep the changes you have made in the previous forked repository so that you can populated the change in new forked repository.


Update vulnerability matching to be more consistent across sources and types

Currently, packages are matched against completely different vulnerability sources with their own identifiers, severities, descriptions, and affected-version semantics. This can be confusing and cause duplicate matches for packages depending on which dbs they appear in (e.g. GitHub and NVD).
To resolve this, all matches for vulnerabilities should be against CVEs unless there is a DB that reports vulnerabilities that will not have CVEs issued (e.g. GHSA but no linked CVE) where the advisory is the only record of the vulnerability. All vendor-specific advisories or vulnerability dbs should be treated as more specific information about an NVD record on a specific set of packages.
This should be visible to users in the following ways:

Vulnerability matches should always be against a CVE unless there are other unique identifiers from other dbs that are not linked to a CVE at all.
Vulnerabilities matched from a vendor advisory should include the advisory reference and link and all metadata for a vulnerability shown to a user should be from the most accurate and specific source available for a given package (e.g. use RHEL data for RHEL packages). This is mostly the case now, but can be made more consistent.

Is this something that could potential fall into 0.8.0?


Multiple frameworks UI change

Would something like this be welcomed?

Yes please :heart:

Modification

[ ] Emit extra html for tabs and static CSS for styling
[ ] Fallback to original rendering if CSS is missing (offline HTML w/o CSS)

Sample data
Preview


Average frame time in results

Please add the average frame time and/or more statistics to the results.json.
Other things to add might be:

average actions sent
average actions*time/frame
time for first frame

Stuff like "total minerals gathered" "build order" etc would be neat to see too

@spacekitteh total mineral gathered is available to the bot through the score interface. https://github.com/Blizzard/s2client-proto/blob/9906df71d6909511907d8419b33acc1a3bd51ec0/s2clientprotocol/score.proto#L31
The build order could also be logged by the bot itself.
I think the LM should only log things that are not available to the bot. Otherwise the result gets cluttered with lots of information that nobody really needs, that are just 'neat to see'. Just imagine the result json after 20 or 100 games if the build order is always included for both bots.

@Archiatrus I believe this idea is inspired by Battle-net, where you can see gathering rate/count and actual build order. I doubt many bot authors would care about such things as BO logging especially because it could be considered as private strategy and requires additional efforts to implement such logging.
Another con for this is that we don't provide bot logs to third person, only to the author. And as you've said above such logs could be huge enough with hundreds of logged games.

True, I keep forgetting people might not want to expose everything


Npm gulp

Commit messages in npm-gulp branch explain the changes.

Hey, thanks for contributing, I have to update the readme, but there is a new front end located at src/main/resources/static This version has the gulp dependency.


[KTO]: Fix nan losses and crashing job

fixes #1447

use nanmean() instead of mean() for losses to avoid nan losses
remove obsolete accelerator.gather for metrics as the metrics are all collected to cpu anyway and averaged later when logging them.

thanks @claralp

btw. I also tested it on multiple GPUs now. It is running and both GPUs show usage, but the distribution does not seem to be effective as there is no speedup compared to 1 GPU.
However, this is the same behavior as before this PR. Might look into that later

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@kashif @lewtun: is there anything more I should validate or can you approve this for now?
I am looking further into the multi GPU case atm, but would prefer to keep this small and open another PR if I find something.

@claralp would you be able to test out #1476 ?

@kashif for me https://github.com/huggingface/trl/pull/1476 has a few issues, (see comments there).
Except from this:

removing the interleaving of datasets works for me and is a useful change
using e.g. metrics[f"{prefix}rewards/chosen"] = self.accelerator.gather(mean_chosen_reward).nanmean().item()
instead of metrics[f"{prefix}rewards/chosen"] = chosen_rewards.detach().nanmean().cpu() makes no difference for balanced batches. Both move the mean value to CPU at the end (one as tensor and one as scalar tho).
However, self.accelerator.gather(mean_chosen_reward) leads to issues when mean_chosen_reward is nan. This happened if there are no 'chosen' samples in the batch

thanks @claralp with regard to moving the loss to the self.args.device I thought that accelerate handles all that for us?

@kashif for the loss accelerate handles that for us, yes.
But this is just for the rewards/logp that are stored as metrics to be logged. In my understanding, that's why the metrics are collected across all GPUs and sent to CPU to calculate the mean value there.
E.g. in dpo_trainer, you use metrics[f"{prefix}rewards/chosen"] = chosen_rewards.mean().cpu()

@kashif, I applied the suggested changes from the code style pipeline. Should be green again now

we're facing a problem that possibly will be solved by this PR, thanks for this @claralp ! we're going to test it and give the feedback.

hey! I just tested the code in branch claralp:fix_nans, and it's hanging during evaluation.

@johncordeiro could you try the version here and see if you're sill experiencing hanging? https://github.com/kawine/trl if so, more context would be helpful
@claralp thank you for all the fixes! seems like this version is only using metrics from the main process' microbatch, by getting rid of the gathers? i have a pr in #1499 that includes most of your changes here but still supports batch-wise gathering if you want to take a look

@johncordeiro could you try the version here and see if you're sill experiencing hanging? https://github.com/kawine/trl if so, more context would be helpful
@claralp thank you for all the fixes! seems like this version is only using metrics from the main process' microbatch, by getting rid of the gathers? i have a pr in #1499 that includes most of your changes here but still supports batch-wise gathering if you want to take a look

@kawine as I understand and experienced it, this version does not use only the main process' micro batch.
It sends the micro batch of every GPU directly to the CPU to calculate metrics there.
Using accelerator.gather would only gather them all in the main process first before sending them to CPU. This works, but is not necessary, as the metrics are collected on CPU later anyway.
For DPO it is implemented the same without gathering: https://github.com/huggingface/trl/blob/0ee349dcd43b0f4b3169449f16751c38ac4a609f/trl/trainer/dpo_trainer.py#L1055

It sends the micro batch of every GPU directly to the CPU to calculate metrics there.

thanks @claralp ! maybe it's just me, but I'm not seeing the CPU collection happen? if there is no explicit gather, then the logging only reflects the statistics from the microbatch on the main process only.
the HF documentation for Trainer, which is subclassed here, also says for log_metrics that "Under distributed environment this is done only for a process with rank 0."

@johncordeiro do you have prediction_loss_only enabled in your evaluation step? The logits were not propagated, so this part was not working in the previous version.
If this is not the reason behind the hanging issue in evaluation, can you provide the line where it hangs?

@kawine it uses .tolist() for all metrics, which moves them to CPU. The log_metrics of HF Trainer is not used by KTO Trainer, it uses its own subclassed log function

@claralp if i add the line metrics[f"device"] = torch.Tensor([float(str(self.args.device)[-1])]).cpu() to get_batch_loss_metrics, i can see in wandb that the value is always 0 (i.e., the main process), suggesting that only metrics from one device are tracked without the explicit gather

@claralp if i add the line metrics[f"device"] = torch.Tensor([float(str(self.args.device)[-1])]).cpu() to get_batch_loss_metrics, i can see in wandb that the value is always 0 (i.e., the main process), suggesting that only metrics from one device are tracked without the explicit gather

@kawine you need to read self.accelerator.process_index to get the current index when using Accelerator.
And make sure you use the latest version, in a previous version store_metrics was really just called from the main process, which was wrong

still getting the same thing with metrics[f"device"] = torch.Tensor([float(str(self.accelerator.process_index)[-1])]).cpu() and the latest version of accelerate (0.28.0)

@kawine, I just tested this with per_device_batch_size=2 on 2 GPUs. When printing the metrics in store_metrics, I get:
{'rewards/chosen': [-0.23836135864257812, -0.9353666305541992], 'rewards/rejected': [], 'kl': 1.9421863555908203, 'logps/chosen': [-33.11909484863281, -32.70828628540039], 'logps/rejected': [], 'device': 1}
{'rewards/chosen': [-0.018026351928710938], 'rewards/rejected': [-0.438812255859375], 'kl': 1.9421863555908203, 'logps/chosen': [-24.179973602294922], 'logps/rejected': [-138.9338836669922], 'device': 0}

I print them in store_metrics, because the logging to e.g. wandb prints an average of 0.5 for 2 devices instead of showing the metrics per device.
Which accelerate config do you use?

@kashif or @lewtun can you confirm if the current changes work for you as well?

sure @claralp let me see!

@claralp i've tried this with deepspeed and the regular data parallel (examples/accelerate_configs/multi_gpu.yaml) and it still only reports stats from the main process.
if you are printing the stats in store_metrics, it will print them from each process, yes, so you will have one line with 'device': 1 and another with 'device': 0. But if you look at the chart for train/device in wandb, it will show only 0, because the two processes' stats are never gathered (at least for me)

if you look at PPOTrainer, the metrics are explicitly being gathered across all machines at each step with an all_reduce: https://github.com/huggingface/trl/blob/ab0d11d81550618546cd1d8807627a594a58f029/trl/trainer/ppo_trainer.py#L897

@kawine reading your plot -> is it possible that you train on multiple different machines with one GPU each?
It reads "copper-paper" and "noble-pyramid". I think thats names coming from k8s?

closing this due to #1514


deps: use assert

Replaces if ... throw with assert. Using unassertify these statements can be stripped during compilation, resulting in a smaller bundle size for production.
Thanks!

:zap:


fix(errors): change how config errors are displayed

Change how config errors are displayed.
Rather than rejecting the promise, which results in an Atom toast error message, treat the error like a lint error and display it in the footer.
This becomes far less annoying when opening projects that don't include an .eslintrc.json file.
Fixes #166

Great thanks @bsyk 👍


Relax Record class for now

As discussed here: https://github.com/facebook/immutable-js/pull/805#issuecomment-210793138 the new Record libdef broke old code.
I've relaxed it again to any. We will loose comfort but it will not error.
Ideas to improve exist, can't do them yet though.
Also fixed errors I've introduced with the last PR. (minor)

cc @bestander This should probably also fix https://circleci.com/gh/facebook/react-native/5344

I can check on Monday if you release 3.8.1

Thank you so much for the fix!


Disable submodule-build of QtKeychain

Pull request checklist:

[x] CHANGELOG.md was updated, if applicable

Description
The original reason for using the fork was to support our qmake build method. The changes we had made specifically were:

Updated the .pri file to point to the right mac/apple-specific file (since added upstream)
Added #undef singals in libsecret.cpp to fix compilation. (no longer necessary since we build with cmake now)

However, it's not as easy as just swapping to upstream - the way the library is built makes it difficult to include as part of our build process. Because of this, we now use FetchContent to ensure the build happens completely separately, and it also provides us with proper build/source directories we can use to find the include files.
~The PR currently fetches the project from the internet, however once the build is stable the submodule will be updated and we'll just point to the local directory instead.~

Not very feasible atm, closing


Update for python3

This commit updates mdtable2csv for python3. This includes changes
to the print calls and adjustments to the strings (no encoding
needed anymore). The script header now refers to /bin/python3 to
be more explicit about the environment called.

Thanks for you help.


Request: To show skin on discord chat and compatible with fabrictailor

As said in the title.
To show discord bot the skin that we are using and be compatible with fabrictailor.

Just saw this; perhaps have a look at https://github.com/QuiltServerTools/BlockBot/issues/26 and perhaps https://github.com/QuiltServerTools/BlockBot/pull/39/commits/b4db31003508518dbc2cf2e7659f9e4ad6951ddb

You can even use builtin 1.9.0 method in TailoredPlayer ;)


Update README.md

gcloud components update is what fixed the empty results issue for me

Thanks for contributing.


net/haproxy: add support for client certificate authentication

Via forum post.
Examples:
https://www.haproxy.com/blog/ssl-client-certificate-management-at-application-level/
http://www.loadbalancer.org/blog/client-certificate-authentication-with-haproxy/

... we are really interested in this feature, too. :-)

here, too!

Count me in, how can I help?

@mahescho Currently nobody is working on this. If you are able to help developing this feature, please submit a pull-request; I'd gladly review your work :)

As I will not use the load balancing feature any way I will have a look at NginX. As I really like HA proxy I will have a look at the source when I find some time. Very basic support should not be to hard to implement.

will there be an approx. timeframe for implementation?
thanks!

@bluesource, in case you missed it:

Currently nobody is working on this.

No ETA :)

Would someone be willing/motivated to add this feature for some beer or money? :)

Due to popular demand this feature will be available in the upcoming os-haproxy release 2.10:

If someone is willing to test it, please apply the current development code:
$ opnsense-patch -c plugins 813dfc2b

I did a quick test and it seems to work as expected:
$ openssl s_client -connect opnsense.example.com:8443 -cert clientcert1.crt -key clientcert1.key


[REF] Switch experimental auth to auth0

Closes #379

Changes proposed in this pull request:

Replace Google ID with Auth0 (we have our own project there now)
Exclude the auth0-react library (equivalent of the google library) from vite optimization. For reasons I don't yet fully understand, not doing so results in all component tests breaking with:

[vite] Internal server error: Failed to parse source for import analysis because the content contains invalid JS syntax. If you are using JSX, make sure to name the file with the .jsx or .tsx extension.
  Plugin: vite:import-analysis
  File: /home/runner/work/query-tool/query-tool/node_modules/.vite/deps/@auth0_auth0-react.js?v=7ecd79c7:828:352189

Checklist
This section is for the PR reviewer

[x] PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see our Contributing Guidelines for more info)
[ ] PR has a label for the release changelog or skip-release (to be applied by maintainers only)
[ ] PR links to GitHub issue with mention Closes #XXXX
[ ] Tests pass
[ ] Checks pass
[ ] If the PR changes the participant-level and/or the dataset-level result file, the query-tool-results files in the  neurobagel_examples repo have been regenerated

For new features:

[ ] Tests have been added

For bug fixes:

[ ] There is at least one test that would fail under the original bug conditions.

Summary by Sourcery
Enhancements:

Replace Google authentication with Auth0 for improved authentication management.

@surchs
Is this expected?

@rmanaem possibly yes. The callback URL is currently http://localhost:5173, so if you are running something different (e.g. different port), it wouldn't work. Let me add you to the Auth0 group


09_Flutter_Navigation: Multi-Screen Apps Using Routes and Navigation

Description
Description
Description
To Explain about Navigation in Flutter
Domain
Android Development in Flutter
Type of Contribution
Video
Domain
Android Dev (Flutter)
Type of Contribution
Video
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign


add pairbenchmark

Initial commit for pairbenchmark.
Start server mode in a node, and start a client mode in another node. Specify some handy parameters and start performance testing.
Server side:
./bin/pairbenchmark
2022-04-02T00:03:39.908-0700    INFO    Starting pairbenchmark server
2022-04-02T00:03:39.909-0700    INFO    Started pairbenchmark server on port 9736

Client side:
./bin/pairbenchmark -e <IP_ADDRESS> -m client -d 5 -c 10
2022-04-02T00:00:41.255-0700    INFO    [worker 9] test job started
2022-04-02T00:00:41.255-0700    INFO    [worker 0] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 1] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 3] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 2] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 4] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 7] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 8] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 6] test job started
2022-04-02T00:00:41.256-0700    INFO    [worker 5] test job started

... after a while ...

2022-04-02T00:00:46.276-0700    INFO    [worker 1] test job finished
2022-04-02T00:00:46.277-0700    INFO    [worker 5] test job finished
2022-04-02T00:00:46.277-0700    INFO    [worker 9] test job finished
2022-04-02T00:00:46.283-0700    INFO    [worker 3] test job finished
2022-04-02T00:00:46.283-0700    INFO    [worker 2] test job finished
2022-04-02T00:00:46.283-0700    INFO    [worker 0] test job finished
2022-04-02T00:00:46.294-0700    INFO    [worker 8] test job finished
2022-04-02T00:00:46.295-0700    INFO    [worker 7] test job finished
2022-04-02T00:00:46.295-0700    INFO    [worker 6] test job finished
2022-04-02T00:00:46.303-0700    INFO    [worker 4] test job finished
2022-04-02T00:00:46.327-0700    INFO    The total throughput is 140040

2022-04-02T00:00:46.327-0700    INFO    The TPS is 28008

2022-04-02T00:00:46.327-0700    INFO    The P50 latency is 6.5µs

2022-04-02T00:00:46.327-0700    INFO    The P90 latency is 21.166µs

2022-04-02T00:00:46.327-0700    INFO    The P99 latency is 10.543584ms

Initial performance results looked too good to be true. Debugged a bit and modified buffer handling. Now looks much more realistic.
./bin/pairbenchmark -e <IP_ADDRESS> -m client -d 5 -c 10
2022-04-02T00:53:41.697-0700    INFO    [worker 2] test job started
2022-04-02T00:53:41.698-0700    INFO    [worker 8] test job started
2022-04-02T00:53:41.698-0700    INFO    [worker 6] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 5] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 3] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 4] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 9] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 1] test job started
2022-04-02T00:53:41.697-0700    INFO    [worker 7] test job started
2022-04-02T00:53:41.698-0700    INFO    [worker 0] test job started
2022-04-02T00:53:46.697-0700    INFO    [worker 6] test job finished
2022-04-02T00:53:46.699-0700    INFO    [worker 7] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 4] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 1] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 5] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 0] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 8] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 3] test job finished
2022-04-02T00:53:46.703-0700    INFO    [worker 2] test job finished
2022-04-02T00:53:46.704-0700    INFO    [worker 9] test job finished
2022-04-02T00:53:46.710-0700    INFO    The total throughput is 15038

2022-04-02T00:53:46.710-0700    INFO    The TPS is 3007

2022-04-02T00:53:46.710-0700    INFO    The P50 latency is 47.667µs

2022-04-02T00:53:46.710-0700    INFO    The P90 latency is 10.483041ms

2022-04-02T00:53:46.710-0700    INFO    The P99 latency is 15.842ms

@vodkaslime Nice work

Now added relay mode. Server side could now work as a relay server, while the client side sending test packet according to protocol.
Server side
./bin/pairbench
2022-04-02T21:04:31.467-0700    INFO    Starting pairbench relay server at port 9736

Client side (from local machine)
./bin/pairbench -r client -d 5 -c 10
2022-04-02T21:04:49.288-0700    INFO    [worker 9] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 4] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 2] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 8] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 5] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 7] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 1] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 3] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 0] test job started
2022-04-02T21:04:49.288-0700    INFO    [worker 6] test job started
2022-04-02T21:04:49.289-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.289-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.289-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.289-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:49.290-0700    INFO    Relay client noise protocol handshake is finished
2022-04-02T21:04:54.290-0700    INFO    [worker 0] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 3] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 2] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 6] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 5] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 1] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 8] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 7] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 4] test job finished
2022-04-02T21:04:54.290-0700    INFO    [worker 9] test job finished
2022-04-02T21:04:54.376-0700    INFO    The total throughput is 890639

2022-04-02T21:04:54.376-0700    INFO    The TPS is 178127

2022-04-02T21:04:54.376-0700    INFO    The P50 latency is 41.125µs

2022-04-02T21:04:54.376-0700    INFO    The P90 latency is 79.292µs

2022-04-02T21:04:54.376-0700    INFO    The P99 latency is 221µs


feat: Added a thank you pagerefactor: checkout button shifted to right

Whenever the user clicks the checkout button inside cart section the user is redirected to thank you page.


Allow NettyPerfClient to connect multiple frontends

log response based on response code range rather than single response
code value.
allow perfClient to connect multiple frontends.

Codecov Report

Merging #1217 into master will decrease coverage by 0.01%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master    #1217      +/-   ##
============================================
- Coverage     69.67%   69.66%   -0.02%
- Complexity     5476     5478       +2
============================================
  Files           431      431
  Lines         33554    33573      +19
  Branches       4258     4262       +4
============================================
+ Hits          23380    23387       +7
- Misses         9000     9017      +17
+ Partials       1174     1169       -5

Impacted Files
Coverage Δ
Complexity Δ

....github.ambry/tools/perf/rest/NettyPerfClient.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

.../main/java/com.github.ambry.router/EncryptJob.java
92.1% <0%> (-5.27%)
4% <0%> (-1%)

...b.ambry.network/BlockingChannelConnectionPool.java
70.42% <0%> (-1.88%)
8% <0%> (ø)

...va/com.github.ambry.replication/ReplicaThread.java
74.62% <0%> (-0.19%)
66% <0%> (-1%)

...in/java/com.github.ambry.store/BlobStoreStats.java
71.13% <0%> (ø)
103% <0%> (-1%)
:arrow_down:

...java/com.github.ambry.network/SSLTransmission.java
69.74% <0%> (+0.31%)
69% <0%> (+1%)
:arrow_up:

...ain/java/com.github.ambry.router/PutOperation.java
90.6% <0%> (+0.35%)
110% <0%> (+2%)
:arrow_up:

....github.ambry.clustermap/StaticClusterManager.java
74.04% <0%> (+1.7%)
83% <0%> (+1%)
:arrow_up:

...github.ambry.rest/AsyncRequestResponseHandler.java
90.87% <0%> (+2.28%)
23% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 79b311b...2084972. Read the comment docs.


Maintenance of project?

Hey @peritus can you give an update about your plans about this project?
It appears that @c4urself has an active fork with pypi releases (as bump2version).
Perhaps a transition can be worked out?

@ionelmc
I should probably add a sunsetting notice to the top of the README, saying this is not actively maintained and people should switch to one of the forked versions.
The other project I'm aware of is @vortec's versionbump (https://pypi.python.org/pypi/versionbump / https://github.com/vortec/versionbump) that looks like a nice replacement.

@peritus yes but this project is more complete. Perhaps consider conceding maintenance to someone?

@peritus So I take that's a no. Not even giving pypi upload permissions?

@peritus Yes, please! The only reason a bump2version exists is to be able to upload it to PyPI under a different name.

Any updates on transferring the maintenance of this project to someone else?

any updates? ..

Any updates?

@deeplook https://github.com/c4urself/bump2version

What a pity, that also the Debian package of this helpful piece of software stalls due to the uncertainty :(
I just filed a wishlist bug asking the maintainer of the Debian package for considering c4urself`s fork for future package updates.
@peritus: it would be so nice and reduce the pain and additional work so much, if you could manage to let this great project continue to blossom in the hands of fellow (co-)maintainers.

This could get resolved pretty soon. See: https://github.com/c4urself/bump2version/issues/86#issuecomment-530106531.


Tokens not auto-detected on Polygon

Describe the bug!
Tokens on Polygon does not show up automatically even with a positive balance.
Tokens are missing icons that will differentiate them from the Mainnet.
Steps to reproduce (REQUIRED)
Steps to reproduce the behavior

Enable Polygon Network
Swap Matic (on Polygon) to DAI (on Polygon)
DAI does not show up

Expect
6a. Auto-display tokens with positive balance on Polygon
Display token icons
YFI
0xDA537104D6A5edd53c6fBba9A898708E465260b6
https://zpl.io/bW85XGj

WRX
0x72d6066F486bd0052eefB9114B66ae40e0A6031a
https://zpl.io/VQNz1yx

WMATIC
0x0d500B1d8E8eF31E21C99d1Db9A6444d3ADf1270
https://zpl.io/bJDB1dJ

WBTC
0x1BFD67037B42Cf73acF2047067bd4F2C47D9BfD6
https://zpl.io/VOvr13L

UNI
0xb33EaAd8d922B1083446DC23f610c2567fB5180f
https://zpl.io/a3EG9GN

USDC
0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174
https://zpl.io/VDg81ke

USDT
0xc2132D05D31c914a87C6611C10748AEb04B58e8F
https://zpl.io/be3mWdD

UBT
0x7FBc10850caE055B27039aF31bD258430e714c62
https://zpl.io/2j3OWrO

TEL
0xdF7837DE1F2Fa4631D716CF2502f8b230F1dcc32
https://zpl.io/bz3pLB3

SWAP
0x3809dcDd5dDe24B37AbE64A5a339784c3323c44F
https://zpl.io/2v3QPz7

QUICK
0x831753DD7087CaC61aB5644b308642cc1c33Dc13
https://zpl.io/bPER1km

LINK
0x53E0bca35eC356BD5ddDFebbD1Fc0fD03FaBad39
https://zpl.io/VYqDXjw
LEND
0xD16937941BD94Bd00050B21E2C1308DB6B89B485
HEX
0x23D29D30e35C5e8D321e1dc9A8a61BFD846D4C5C
https://zpl.io/2j3O7jW

COMP
0x8505b9d2254A7Ae468c0E9dd10Ccea3A837aef5c
https://zpl.io/bl3B6Gz

DAI
0x8f3Cf7ad23Cd3CaDbD9735AFf958023239c6A063
https://zpl.io/am3JWp9

MATIC
0x0000000000000000000000000000000000001010
https://zpl.io/aNxW109

Screenshots

Do I have to refresh, reinstall, reimport to see this?

Do I have to refresh, reinstall, reimport to see this?

@colourfreak I just released the title of the issue only reflects 1 of the problem. Re-opening:

[x] Update Polygon Token Icons — they should just appear with the new version without having the reinstall. Maybe restart will help.
[ ] Tokens on Polygon not appearing — will check.

@hboon You can check out my wallet and tokens that are displayed and not.
https://explorer-mainnet.maticvigil.com/address/0x80cc263cb3fA1bE2AeC748b3811261c1E2a1ba8d/tokens

It wouldn’t because there was no way to fetch transaction history (and hence detect tokens) currently. Coming soon.


Storage Values/Solar Energy per Timeframe

Hey @chpro,
sorry to bother you. I have two additional questions.

How is it possible there is a higher value for "from Storage" than "to Storage". Am I making a mistake here? I am exactly using you json for the Dashboard and I also saw this on your readme example.

compared to my HomeAssistant what is exactly like the SolarWeb Application says.

How can I make an additional panel for "Solar Energy produced" for the Dashboard timeframe? I tried to figure out via expression and math etc. but I cant make it work. May I ask you for your help? For my understanding it must be something like
Home Usage - From Grid - From Storage + To Grid + To Storage. Do you have a code snippet for that by a chance :) ?

Thank you. :)

Form Storage and To Storage are currently calculated from the power values via an integral function which is only an approximation of the energy used. I have not had time to document this and did not find the value which represents the energy as for local energy production.  Currently 1 measurement a minute is in influx, I assume a lower deviation when the data is collected more frequently.
I was also struggling with that that's why currently the Local Energy Production incorporates both from battery and solar panels (which is the provided value by the API) and it is also related to the inaccuracies of the From/To storage calculation. When the From/To Storage values would reflect the real values more accurately the calculation can easily be done within the transformation tab in Grafana. You would then have 5 queries which return the above mentioned values and do the calculation in transformation tab.

I see, I've changed every interval (except solar forecast ofc) to 10s yesterday. Just trying it out.
OK I'll try to figure out further on that at the Weekend.

Thanks for your fast response

Thanks for the praise and input. I will try to improve the accuracy and investigate how HA does the calculation. Have some time in summer to investigate that further. When there is an improvement I will post an update.

@robotnikz Had some time to take a look at the queries for From Storage and To Storage there is a Group By clause which may have a negative effect on the precision if you still have the data available you may want to remove GROUP BY time(1s) fill(0) and change select clause to integral("P_Akku", 1h) in these queries and check if the situation improves. Thank you for your help.
Queries would look like for from storage
SELECT integral("P_Akku", 1h) FROM "autogen"."powerflow" WHERE ("P_Akku"::field >= 0) AND $timeFilter tz('${tz:raw}')
and for to storage
SELECT abs(integral("P_Akku", 1h)) FROM "autogen"."powerflow" WHERE ("P_Akku"::field <= 0) AND $timeFilter tz('${tz:raw}')

@chpro
Thanks! Will try that out and give feedback.

This is the situation at 4p.m. today
Grafana:

Home Assistant: (same as mentioned in Fronius SolarWeb)

From/To Grid is great.

Hey again @chpro,
I tried to figure out how to solve the gap between Home Assistant and the Grafana queries but at the end I came back to your last proposed solution since I was unable to find a more accurate way to read all the data.
Update from yesterday.
Home Assistant:

Grafana:

Think that wont help you here, just as a last update here from my side. Will report back if I found a better solution.
Thanks for your work again. Enjoy the sun! :)


Add progress bar, colorized terminal output?

A progress bar would be a nice feature for users, especially on longer running tasks.

closed with #32


[Scenario] New Mansions of Madness Scenario

Description of Problem or Question
I would like my first scenario to be added to valkyrie's listings.
Files can be found at:
https://github.com/DarkVanf/valkyrieseahide
Valkyrie Version
Attach Save, Quest or Scenario
Log File:
%LOCALAPPDATA%Low\NA\Valkyrie\output_log.txt
Saves:
%appdata%/Valkyrie/MoM/Save
%appdata%/Valkyrie/D2E/Save
Quest:
%appdata%/Valkyrie/D2E/Editor
Scenario:
%appdata%/Valkyrie/MoM/Editor

Added to the manifest. It would be available within one hour.
@antontimmermans, could you add it to Valkyrie's statistics?

added to Statistics sheets


Add .mp4 as a valid asset extension

Thank you for the PR

@satya164 This was merged into extra-assets branch, but not master. I didn't realize at the time that you had already squash merged extra-assets into master, or I would have set the target of this PR to be master. Sorry about that!

Although this was added, I get:
./src/features/.../dog.mp4
Module parse failed: Unexpected character '' (1:0)
You may need an appropriate loader to handle this file type.
(Source code omitted for this binary file)
@ ./src/features/.../Component.native.js 14:9-36
@ ./src/features/.../index.js

Should I report this in an issue?

Mmm, actually my version didn't have this change, adding the extension by hand fixed.

Ok so the issue is that this didn't reach any version and the last one published (1.0.0) is from June 2017....


Solve #138131

          #138131

Originally posted by @A111one in https://github.com/flutter/flutter/pull/147464#pullrequestreview-2026970539

Please review https://github.com/flutter/flutter/blob/master/CONTRIBUTING.md. This issue is not useful because it doesn't follow any of the templates.

@exts @LuxuryImportGlow


Publish the style guide to a static site

Overwrite styleguide.reactioncommerce.com since that is no longer correct.
Generally probably the same process as for the new docs site. The command is yarn run styleguide:build but this hasn't been tested in quite a while.

@ticean @impactmass - Let's use Netlify for this  🎉 I believe https://styleguide.reactioncommerce.com/ is the last site on the old infrastructure.

@ticean @impactmass @aldeed - Since the NPM packages are auto-released w/ semantic release, should we also tie a docs release w/ every semantic release? Makes the most sense to me.
We can first start by testing it on a URL like https://staging-styleguide.reactioncommerce.com/ before the real deal, https://styleguide.reactioncommerce.com/.

@machikoyasuda I think it can be triggered on CI at the same time as semantic-release (on every push to master), but it should publish the static site every time (whereas semantic-release may or may not actually publish a new version). Once you're comfortable with the process, it should build straight to the prod URL. There is no need for managing multiple versions or anything like that, so it should be safe and simple.

Master is now being deployed to: https://stoic-hodgkin-c0179e.netlify.com/


Feedback for "Manage Authorizations and Roles"

https://help.sap.com/docs/SAP Business Application Studio/9d1db9835307451daa8c930fbd9ab264/01e69c53003c4b0a8a64310a3f08867d.html
It may be helpful to line out, that using role collection administrator is not enough to manage Business Application Studio.
Also Developer is needed.

Hi,
Can you please tell me as an admin, for which operation you needed the developer role?
Thanks!

Hi,
Problem was, that accessing a newly created BAS was not possible after adding the admin role to the creating user.
We would have expected, that an admin role has more rights than a developer, not a purly distinct role and using Postman and similar for actions to be tanken.
So access to BAS is only possible for a User with developer role assigned, not for admin role.

I added a note at the beginning on the topic stating that you need both roles.
It will be visible after our next release.
Thank you for your help in making our product better!


Getting rid of default.py

moved all the content of default.py to config_spec_and_defaults.conf
and to default arg values in features_crawler.
got rid of this feature: contianer_id to namespace maps.

Signed-off-by: Ricardo Koller<EMAIL_ADDRESS>

Current coverage is 86.37% (diff: 92.00%)

Merging #172 into master will increase coverage by <.01%

@@             master       #172   diff @@
==========================================
  Files            31         30     -1
  Lines          2889       2869    -20
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           2495       2478    -17
+ Misses          394        391     -3
  Partials          0          0

Powered by Codecov. Last update ac897af...c0b75c7

Need some clarifications (see review), please correct me if I'm wrong!

okay, most are interim changes.


New version: BlueBubbles.BlueBubbles version 1.12.0

- [X] Have you signed the Contributor License Agreement?

[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.5 schema?

Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

No errors to post.

(Automated response - build 679.)

Manual Validation ended with:

Automatic Validation ended with:

Installation failed with exit code -1978334972

(Automated response - build 683.)

@wingetbot waivers Add Validation-Installation-Error

Hi @Arch-Frost,

This PR's version number 1.12.0 has 3 parameters (sets of numbers between dots - major, minor, etc), which is less than the current manifest's version <IP_ADDRESS>, which has 4 parameters.

Is this intentional?
(Automated response - build 684)

Hi @Arch-Frost,

This PR's version number 1.12.0 has 3 parameters (sets of numbers between dots - major, minor, etc), which is less than the current manifest's version <IP_ADDRESS>, which has 4 parameters.

Is this intentional?
(Automated response - build 684)

No. This was not intentional. I added version with the version number that was uploaded on the author's download page.

So does the version number need to change, or is this PR OK as is?

So does the version number need to change, or is this PR OK as is?

The PR is OK as it is

Thanks! I'll assume the same for the other 4 referenced above.


model: add deserialization tracing

Add tracing to the custom deserialization impls of models. This will step through the deserializer and won't repeat information that would be given from error returns.
Closes #251.

I tried to add it, though it will make some lines that are rather strange to read for example

We should figure out why it does that

Ready for review

Rebased and added tracing dependency


Upstream rebase to 2.12.1

Create a rebase tree using rebasebot:

rebasebot --source https://github.com/kedacore/keda:release/v2.12 --dest openshift/keda:main \
          --rebase joelsmith/keda:rebase --tag-policy=strict --update-go-modules --dry-run \
          --github-user-token <(yaml2json ~/.config/hub | jq -r '."github.com"[0].oauth_token')

cd .rebase
Update .ci-operator.yaml and squash into existing carry commit
Clean up carry commits, update OWNERS,  squash where appropriate
Add verify history drop commit to use upstream's release/v2.12 branch as the merge base
Push to own fork, open PR

Instruction for reviewer(s):
Compare with upstream (in reverse order): https://github.com/kedacore/keda/commits/release/v2.12
First 90 commits here (dc76ca7..107947c) are the top 90 there. Beyond verifying that this is the case, no review is required of these commits (already reviewed upstream). Verifying only the commit hash of the most recent commit in both places is sufficient to verify all (as the hash would be different if anything had changed).
Compare the merge commit e160f8d4 to its upstream parent 517ed30 to verify they are identical, showing that the rest of our prior changes are reverted by this merge commit and that we're back to a clean slate, relative to upstream.  Here's a link to the diff: https://github.com/openshift/kedacore-keda/compare/dc76ca7...e160f8d4 (Files changed: 0)
or you can check out this branch and run git diff dc76ca7 e160f8d4 and see that there are no changes.
Verify that remaining commits (prefixed with UPSTREAM:) are all the carry patches re-applied post-rebase and that drop patches are not.
Compare with currently-carried patches (in reverse order): https://github.com/openshift/kedacore-keda/commits/main

Let's see if clientset-verify will succeed now that John's https://github.com/openshift/release/pull/49456 has merged.
/retest

/retest

/retest

Why aren't the tests running?
/retest

/retest

/retest

/retest

Strange... without the fix for TestCpuScaler, all the other tests pass, but with the fix, TestEvents failed twice in a row and TestScaledObjectGeneral failed once. Are they flaky or fully broken?
/retest

looks good!
/lgtm

Upstream e2e test fix merged.
/hold cancel


feat(helm): publish OCI helm charts

publish OCI helm-charts see helm docs
fixes #1429

@open-telemetry/helm-maintainers please review

Will merge on Friday unless there is dissent.


Investigate addition of median home value threshold to the higher ed formula

The higher ed threshold may exclude some census tracts that would otherwise be considered an EJ community. Investigate the impact of including median home value in the formula - something like:
Higher ed enrollment is at or above 20% AND median home value is greater than X
X should be determined based on this analysis

Starting this off with some comments -- this is not yet ready for review.
I'm hesitant to use MHV in this way. I did a quick analysis, looking at if we restricted median home value to be less than every quantile (so in the 10th or lower, 20th or lower and so on). To get a population that seems (based on the comparison tool) to have a similar profile to Def M, we have to cut this threshold quite low (10ish or 20ish). This adds back very few tracts, and maybe not the tracts I'd have thought of. In other words, I don't think this looks great (but could be convinced otherwise).  I can clean up this analysis, but this conversation might be easier in person @BethMattern @lucasmbrown-usds
I'm wondering if we could do something like: 20% of students, and no neighbors are DACs. This is similar in how we might think about donut hole DACs -- areas with lots of DACs become difficult to resolve for the purposes of allocating benefits AND reflect systemic vulnerability -- but different in the sense that we are allowing there to be clusters of non-DACs. The idea would be -- yes, there are students, but there's also widespread environmental impacts.
This would include a tract like: https://screeningtool.geoplatform.gov/en/cejst#15.17/33.750517/-84.41107, but not a tract like: https://screeningtool.geoplatform.gov/en/cejst#15.53/37.877445/-122.260258. I plan to do some more analysis here tomorrow.
Here, there's some nuance additionally -- we would have to calculate this once simultaneously for all tracts, so if there are clusters of tracts that would have been included, the middle ones might get left out -- but I think it might make more sense than trying to play with another threshold.

Moved to review -- will discuss in meeting


authentication feature can be added

Once the application comes for login and sign-in, we can work on authentication. Please address this issue with relevant tags under GSSOC and Hacktoberfest.

have you started or not ???

@gupta-ritik can i take this issue.


Response headers filtering: multi-value header values are dropped

Response headers can be added multiple times by the server, in particular the Cookie and Vary header (see: https://www.rfc-editor.org/rfc/rfc7230#section-3.2.2)
Currently the application.rs part handling response's headers is only fetching the first value, thus discarding all other values.
This results in headers being dropped, even if there are no rules in place.
The issue rely on https://github.com/redirectionio/fastly-worker/blob/master/src/rio/application.rs#L207 where request.get_header(), is used instead of request.get_header_all().
The subsequent handling should be adapted too, as to push and append the original set of headers, including the multi-value ones.

Hello,
Do you want to give it a try and submit a PR?.if you can't, this is perfectly fine, I'll do it 👍🏼

@thePanz so it looks like the issue comes from fastly?

collecting (only) the first value for each header

response.get_header_names() returns only the first header?

IIRC response.get_header_names() get the names, but the values can be multiple (for Vary and Cookie for example).

The issue rely on https://github.com/redirectionio/fastly-worker/blob/master/src/rio/application.rs#L207 where request.get_header(), is used instead of request.get_header_all().
The subsequent handling should be adapted too, as to push and append the original set of headers, including the multi-value ones.

fixed in https://github.com/redirectionio/fastly-worker/commit/d445ff7d4d3cb80f0835941974612fb7bd878efd


auto import suggestion is not working in version 7.5.6

Describe the bug
other libraries auto import works but only rxjs doesn't support auto import / suggestion in visual studio code.

Expected behavior
auto import should be showed on IDE

Reproduction code
No response
Reproduction URL
No response
Version
7.5.6
Environment
No response
Additional context
No response

I am not sure why this is related with rxjs core. I just created bare empty repo and vscode suggests this:


Apply EXCLUDE_FROM_ALL to effcee

Saves us a few targets to build by default

There are two very fast-running effcee tests that normally run.  But now they fail because the underlying code was not built ?
I would still like the SPIRV-Tools public bots to run those tests, to make sure our test infra is working at least in basic ways.   Can you reframe this?

Hmm. Works fine for me on MacOS. Didn't check windows previously. I'll abandon this PR then since it saves just a small amount of time.