drivers/modem helper libraries refactoring

Introduction
The drivers/modem contains libraries which are used for generic
functionality including uart interface, modem sockets and
command handling. These libraries work as expected, yet, they
lack documentation and APIs.
Purpose
This pull request adds well documented APIs to the libraries
and updates the in-tree drivers which depend on the libraries
to use the new APIs. The APIs are specifically designed to
prevent the need for directly accessing members of contexts,
which is the case for most of the initialization and use
of the libraries currently.
Added feature (auto assigning socket id)
The refactoring does not change the functionality except for
one added feature: A configuration option has been added to
the modem_socket library, which determines if the socket id
for a given socket is automatically assigned by the library,
which is the case for the quectel and simcom modems, or if
the id must be assigned later, as is the case with the ublox
sara modem.
This new feature comes with three helpers for determining if
the id has been assigned, if the socket has been allocated,
and for safely setting the id. With this new feature, the
simcom and quectel drivers have been updated to use the id,
which now behaves correctly with the added config option,
instead of the sock_fd.
Status
I have verified that all of the in-tree modem drivers can
compile with the new changes, and have tested and verified
the quectel-bg9x modem driver and the PPP GSM driver with
a quectel-bg95. @warasilapm has verified the ublox-sara-r4
driver.
The simcom7070 is nearly identical to the bg9x driver, so
the changes are expected to work for both of them, no one
with the modem has been available to test the update.
Last words
Now is the time for adding features, cleaning up, adding
documentation etc. to the helper libraries, pull requests
will be much appreciated.

cc @bbilas

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching most of the files.

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching a bunch of the file.
As an aside, does anyone know what the plan is for getting net_buf et.al. compliant with the coding guidelines is? I looked around on the mailing list for stuff relating to that but didn't find anything.

Regarding the ublox-sara-r4, the order op operations would preferably be:

Test this PR to make sure ublox-sara-r4 is working (which i can't since i don't have that modem)
Merge this PR
Start work on fixes for ublox-sara-r4

I know nothing regarding net_buf

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching a bunch of the file.
As an aside, does anyone know what the plan is for getting net_buf et.al. compliant with the coding guidelines is? I looked around on the mailing list for stuff relating to that but didn't find anything.

Regarding the ublox-sara-r4, the order op operations would preferably be:

Test this PR to make sure ublox-sara-r4 is working (which i can't since i don't have that modem)
Merge this PR
Start work on fixes for ublox-sara-r4

I know nothing regarding net_buf

The only issue with this approach is that parts of the SARA-R4 driver were broken before. I can just rebase my fixes on that branch, but the SARA being "working" after this PR is a bit of a misnomer without further fixes that shouldn't be wrapped up in this single refactor commit.

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching a bunch of the file.
As an aside, does anyone know what the plan is for getting net_buf et.al. compliant with the coding guidelines is? I looked around on the mailing list for stuff relating to that but didn't find anything.

Regarding the ublox-sara-r4, the order op operations would preferably be:

Test this PR to make sure ublox-sara-r4 is working (which i can't since i don't have that modem)
Merge this PR
Start work on fixes for ublox-sara-r4

I know nothing regarding net_buf

The only issue with this approach is that parts of the SARA-R4 driver were broken before. I can just rebase my fixes on this branch, but the SARA being "working" after this PR is a bit of a misnomer without further fixes that shouldn't be wrapped up in this single refactor commit.

I see, so order of operations is

Merge this PR with main
Rebase SARA fixes on this PR (or main)
Merge SARA fixes with main

Or

Rebase SARA fixes on this PR
Merge SARA fixes to this PR
Merge this PR with main

If i understand correctly

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching a bunch of the file.
As an aside, does anyone know what the plan is for getting net_buf et.al. compliant with the coding guidelines is? I looked around on the mailing list for stuff relating to that but didn't find anything.

Regarding the ublox-sara-r4, the order op operations would preferably be:

Test this PR to make sure ublox-sara-r4 is working (which i can't since i don't have that modem)
Merge this PR
Start work on fixes for ublox-sara-r4

I know nothing regarding net_buf

The only issue with this approach is that parts of the SARA-R4 driver were broken before. I can just rebase my fixes on this branch, but the SARA being "working" after this PR is a bit of a misnomer without further fixes that shouldn't be wrapped up in this single refactor commit.

I see, so order of operations is

Merge this PR with main
Rebase SARA fixes on this PR (or main)
Merge SARA fixes with main

Or

Rebase SARA fixes on this PR
Merge SARA fixes to this PR
Merge this PR with main

If i understand correctly, and you prefer the first one

That's correct. It's not necessarily a problem to have them become part of this PR, but they shouldn't be squashed in with the existing refactor commit and it expands the scope of this PR so it may take longer to get this one merged in.

Since @ycsin brought up MISRA compliance, the ublox-sara-r4 file could probably use a once over by someone who has better ingested the new(ish) coding guidelines published in the docs. I'll be opening a PR for sara-r4 fixes in the near future and it would be good to get a feel for what following those guidelines entails and a good opportunity to make the changes while I'm touching a bunch of the file.
As an aside, does anyone know what the plan is for getting net_buf et.al. compliant with the coding guidelines is? I looked around on the mailing list for stuff relating to that but didn't find anything.

Regarding the ublox-sara-r4, the order op operations would preferably be:

Test this PR to make sure ublox-sara-r4 is working (which i can't since i don't have that modem)
Merge this PR
Start work on fixes for ublox-sara-r4

I know nothing regarding net_buf

The only issue with this approach is that parts of the SARA-R4 driver were broken before. I can just rebase my fixes on this branch, but the SARA being "working" after this PR is a bit of a misnomer without further fixes that shouldn't be wrapped up in this single refactor commit.

I see, so order of operations is

Merge this PR with main
Rebase SARA fixes on this PR (or main)
Merge SARA fixes with main

Or

Rebase SARA fixes on this PR
Merge SARA fixes to this PR
Merge this PR with main

If i understand correctly, and you prefer the first one

That's correct. It's not necessarily a problem to have them become part of this PR, but they shouldn't be squashed in with the existing refactor commit and it expands the scope of this PR so it may take longer to get this one merged in.

If you can confirm that the SARA modem driver is at least functional once this PR is merged, without the SARA fix PR, then we can go ahead and merge this PR so we can proceed with the SARA fix PR. I will test the ppp and bg9x driver to make sure they work as well, the simcom is a copy of the bg9x (or the other way around) so if i match the changes in the bg9x driver with the simcom, we should be good to go.

I have not had time to diagnose the issue, but the sara driver is at least partially broken with the current state of this PR.

I have not had time to diagnose the issue, but the sara driver is at least partially broken with the current state of this PR. I tested with the fixes required from my fork and UDP sockets are non-functional.

I can verify the BG9x and PPP drivers are working. The wifi/esp_at.c has had very minor changes to it, and the simcom7080 has had nearly identical changes made to it as the BG9x, since they are nearly identical drivers, I have high certainty that these drivers will work.
Once you have had time to diagnose the issue with the SARA-R4 changes in this PR, and it has been fixed, I will turn this from a draft to a PR so we can have it merged :)

Latest update to status:

quectel-bg9x tested and working
ublox-sara-r4 tested and working
gsm_ppp tested and working
simcom-sim7080 builds but not tested, changes are identical to quectel-bg9x, so expected to work
esp.c builds but not tested, only initialization of modem_cmd_handler and modem_iface_uart updated

The fix is now in a state where it can be merged with confidence.

@rerickson1 Now that we've passed the 3.2 release, is this something we can get merged in? I think it's ready to go.

@rerickson1 Now that we've passed the 3.2 release, is this something we can get merged in? I think it's ready to go.

Just need two reviews on this. I have never worked with this code so I welcome other reviewers on this to approve.

The following updates to this PR have been performed based on requests:
Changes

PR has been split into individual commits pr feature
modem_cmd_handler_init() has been updated to use a "designated initializer" as suggested by @mniestroj
modem_cmd_handler_init_cmds() has been merged with modem_cmd_handler_init() and removed, there is not only one init function for modem_cmd_handler.
Drivers have been updated to use the new modem_cmd_handler_init()
Whitespace changes from gsm_ppp.c have been removed.

Validation
gsm_ppp and quectel_bg9x have been tested with the new modem_cmd_handler_init(). All modem drivers can compile. Testing ublox-sara and esp_at drivers is encouraged.

@mniestroj Can you find time to re-review this PR? The following comment provides the updated status since you last requested changes :) https://github.com/zephyrproject-rtos/zephyr/pull/48530#issuecomment-1286774121

@ycsin Adressed all formatting issues you pointed out, please re review :)

@mniestroj Every commit now contains motivation for each change, and every commit is bisectable. The three commits update modem_cmd_handler, modem_iface_uart and modem_socket seperately. The commit descriptions should answer most of the questions you raised.
@ycsin All additions to the modem helper libraries and driver modifications should now be formatted correctly, formatting of existing code has not been altered. Fixing formatting of existing code is outside the scope of this PR.

@mniestroj ping for re-review, thanks!

@mniestroj can you please review?


I am not sure why this isn't running

Add these parameters up in the parameters list (Where Q_max is defined) and if it is up there you can leave off the "design." but be sure to add a comma after the end bracket

Is it Q_max or is it cute Qm_max? Also after I deleted the design i still got the same issue

Qm_max my bad (the m makes sure it is in Liters per second)


Update README.md

This is probably OS dependant - it may be worth addressing at an OS level set of instructions.
Windows: nw.exe
OSX: nw ./
Linux: ./nw

AFAIK, . in the beginning of the path means current directory on any system. So when I type ./nw shell will try to execute file named nw and located in current directory.
I would suggest just copying or referring official instructions in that case: https://github.com/nwjs/nw.js/wiki/How-to-run-apps

AFAIK, . in the beginning of the path means current directory on any system. So when I type ./nw shell will try to execute file named nw and located in current directory.
I would suggest just copying or referring official instructions in that case: https://github.com/nwjs/nw.js/wiki/How-to-run-apps


Change the Quote to replace "chuck norris" with a pokemon name, and display it letter by letter in a dialog box

The API says both "Chuck" and "Chuck Norris" so use both as key words to replace

replaced Chuck Norris with the pokemons name


Fix broken pipe error on docker layer creation

This reverts commit f12346d49359784617a62731501627dc032419d8.
This commit was an optimization which was using pipes to read the layer tarball.
It was causing a 141 exit code (means SIGPIPE for tar) while running Layer packing in the VM lauched for the runAsRoot command.
Packing layer...
[    3.755619] reboot: Power down
builder for '/nix/store/nzc2cpi2xh4whzq5h8dg7mgb5zns7993-docker-layer-mycontainer.drv' failed with exit code 141

This error happen consistently on multiple servers running Ubuntu 18.04 servers with Nix 20.09.
I've tested the patch and the error disappeared on all servers.
Motivation for this change
Fix this error by removing the optimization, so it won't happen again.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Could you share a minimal example? Another way to solve it would be to figure out why we're getting SIGPIPE, and fix that issue instead.
Likely a command fails and the pipe is closed prematurely.

Could you share a minimal example? Another way to solve it would be to figure out why we're getting SIGPIPE, and fix that issue instead.
Likely a command fails and the pipe is closed prematurely.

I agree but SIGPIPE is rather a major security breech as is and should not be taken lightly : https://vigilance.fr/vulnerability/QEMU-denial-of-service-via-NBD-SIGPIPE-Signal-23103

a major security breech

SIGPIPE is merely a normal way for unix systems to communicate that the process behind a pipe is gone. It is not security problem in and of itself. The issue you've linked is a flaw in QEMU, which is a distinct program that is not involved in this PR. Other than that, the Nix sandbox should not be used for running untrustworthy code because it has not been audited (as far as I know).

Could you share a minimal example? Another way to solve it would be to figure out why we're getting SIGPIPE, and fix that issue instead.
Likely a command fails and the pipe is closed prematurely.

In fact both commands are working as expected because when run sequentially they finishe properly.
The issue is in the timing here. It seems that on my servers, tarsum finishes before tar and thus broke the pipe. This is probably due to a delay while tar is syncing the file on the file system (which is a VM and might be slow...).
An interesting post which explains that well: https://stackoverflow.com/a/38549483/2165830
Maybe a better solution would be to ignore the broken pipe using the tee option -p as mentioned in that post.
I'll try that and change the PR if it works.

Here is the new patch which only add -p option to the tee command (no revert). I've tested  it on my servers and works as expected. No more exit 141 errors.

@utdemir looks like we need another approval before this can be merged.

When backporting changes, please follow https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#backporting-changes.
Namely, you should be doing git cherry-pick -x <rev> from a commit that has already landed in master. If the branches have diverged, you may alter the commit or add another commit to ensure that the package is able to still evaluate and build

Sorry @jonringer  I should have read the Contributing Guide before, thanks for the advice.
I've change the commit to what you've suggested and rebase my contribution on master. I'll make another PR for the backport into 20.09.

@mickours While figuring out what's going on with this PR, I noticed your commit does not have an author e-mail address that matches your GitHub account. You might want to fix that, either in https://github.com/settings/emails or in your git config and then git commit --amend --reset-author.
Other than that, it looks good, but to the person who will merge this, let's alter the branch name in the merge commit message to something sensible.

@mickours While figuring out what's going on with this PR, I noticed your commit does not have an author e-mail address that matches your GitHub account. You might want to fix that, either in https://github.com/settings/emails or in your git config and then git commit --amend --reset-author.
Other than that, it looks good, but to the person who will merge this, let's alter the branch name in the merge commit message to something sensible.

Done!

Thank you!


⚠ Bofrost.at has degraded performance

In 0d5afaa, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1229 ms

Resolved: Bofrost.at performance has improved in 8dc83a5 after 6 minutes.


Hot can I add playwright-ct-angular to new nx angular project?

Hello,
I'm trying to evaluate possibility of using playwright component testing.
Created fresh angular project using npx create-nx-workspace <my-project-name> and attempted to add playwright-ct-angular as described at https://github.com/jscutlery/devkit/tree/main/packages/playwright-ct-angular , here's my exact steps

npm init playwright@latest -- --ct
npm install -D @jscutlery/playwright-ct-angular @playwright/test
copied content of https://github.com/jscutlery/devkit/blob/main/tests/playwright-ct-angular-wide/playwright-ct.config.ts to  my playwright-ct.config.ts

When I run npm run test-ct I get following error:
~/pw-nx$ npm run test-ct

<EMAIL_ADDRESS>test-ct
playwright test -c playwright-ct.config.ts

Error: Package subpath './lib/mount' is not defined by "exports" in<EMAIL_ADDRESS>at new NodeError (node:internal/errors:387:5)
at throwExportsNotFound (node:internal/modules/esm/resolve:365:9)
at packageExportsResolve (node:internal/modules/esm/resolve:649:3)
at resolveExports (node:internal/modules/cjs/loader:554:36)
at Function.Module._findPath (node:internal/modules/cjs/loader:594:31)
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1012:27)
at Function.resolveFilename (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/transform/transform.js:191:36)
at Function.Module._load (node:internal/modules/cjs/loader:871:27)
at Module.require (node:internal/modules/cjs/loader:1098:19)
at require (node:internal/modules/cjs/helpers:108:18)
at Object. (/home/alex-aveva/pw-nx/node_modules/@jscutlery/playwright-ct-angular/index.cjs:6:13)
at Module._compile (node:internal/modules/cjs/loader:1196:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
at Object.i..ut._extensions. (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:1074:32)
at Function.Module._load (node:internal/modules/cjs/loader:909:12)
at Module.require (node:internal/modules/cjs/loader:1098:19)
at require (node:internal/modules/cjs/helpers:108:18)
at Object. (/home/alex-aveva/pw-nx/playwright-ct.config.ts:2:1)
at Module._compile (node:internal/modules/cjs/loader:1196:14)
at Module.f._compile (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/utilsBundleImpl.js:16:994)
at Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
at Object.i..ut._extensions. (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:1074:32)
at Function.Module._load (node:internal/modules/cjs/loader:909:12)
at Module.require (node:internal/modules/cjs/loader:1098:19)
at require (node:internal/modules/cjs/helpers:108:18)
at requireOrImport (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/transform/transform.js:172:20)
at requireOrImportDefaultObject (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/common/configLoader.js:83:53)
at ConfigLoader.loadConfigFile (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/common/configLoader.js:56:26)
at runTests (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/cli.js:120:55)
at qr. (/home/alex-aveva/pw-nx/node_modules/@playwright/test/lib/cli.js:40:7) {
code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}
Wonder what I'm missing

Hi @alex-aveva, thank you for raising this issue.
We'll check this out and keep you posted.

👋
Faced the same issue and it was due to a too high version of the @playwright/test package compared to the @jscutlery/playwright-ct-angular installed.
I went with the "@playwright/test": "~1.28.0" it solved the issue. (I'm on ng15 and "@jscutlery/playwright-ct-angular": "0.1.4")


[QUESTION] No teardown_appcontext?

Previously with Flask I wanted to remove database upon shutdown :
database.py
db_session = scoped_session(sessionmaker(.....)
app.py
from database import db_session

@app.teardown_appcontext
def shutdown_session(exception=None):
  db_session.remove()

But with FastAPI:
AttributeError: 'FastAPI' object has no attribute 'teardown_appcontext'
What to do please?

https://fastapi.tiangolo.com/tutorial/events/#shutdown-event

Hi @dmontagu how would one write a unit test for example:
@app.on_event("shutdown")
def shutdown_session():
    db_session.remove()

If you use a starlette TestClient via a context manager:
with TestClient(app) as client:
    client.get(...)

it will run the startup and shutdown events when entering/exiting the context. So that should help you run the events as part of your test suite.

Yeah, all what @dmontagu said. :point_up:
If that solves your problem @scheung38 , you can close the issue.

https://fastapi.tiangolo.com/tutorial/events/#shutdown-event

Above link is expired, here is the updated link https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown .


tls-roots doesn't work on android or ios devices

Bug Report
Flagging this because I spent a few hours working out that this is a problem and other people will probably hit it. rustls-native-certs doesnt support android or iOS so you'll need to setup https://github.com/rustls/rustls-platform-verifier or enable feature tls-webpki-roots.
Version
0.9.2
Platform

ios
android

Crates
tonic
Description
Connecting to an AWS LB returns error:  status: Unavailable, message: "error trying to connect: invalid peer certificate: UnknownIssuer"

Thanks for flagging this!! Saved me some time


MLFlow Logger Makes a New Run When Resuming from hpc Checkpoint

🐛 Bug
Currently the MLFlowLogger creates a new run when resuming from an hpc checkpoint, e.g., after preemption by slurm and requeuing. Runs are an MLFlow concept that groups things in their UI, so when resuming after requeue, it should really be reusing the run ID. I think this can be patched into the hpc checkpoint using the logger which I believe exposes the run ID. This can also be seen on the v_num on the progress bar which changes after preemption (in general that v_num probably shouldnt be changing in this case). I'm happy to attempt to PR this if the owners agree that it's a bug.
To Reproduce
Use MLFlowLogger on a slurm cluster and watch the mlflow UI when preemption happens, there will be a new run created.
Expected behavior
Runs are grouped neatly on the MLFlow UI
Environment

CUDA:
- GPU:
- available:         False
- version:           10.2
Packages:
- numpy:             1.20.1
- pyTorch_debug:     False
- pyTorch_version:   1.7.1
- pytorch-lightning: 1.2.0
- tqdm:              4.57.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.1
- version:           #1 SMP Thu Jan 21 16:15:07 EST 2021

cc @awaelchli @ananthsub @ninginthecloud @rohitgr7 @tchaton @akihironitta

Dear @Queuecumbe,
Yes, great catch ! We would be happy if you could contribute a PR to this issue :)
Best,
T.C

I think there's potentially a larger issue than MLFlow which is that the v_num shouldn't change on resumption which would effect every logger. Do you want my PR to include some logic around that as well?

The way I've implemented this is that the dump_checkpoint function adds the logger to the checkpoint in a key called "logger", then in restore_training_state I simply load that into self.trainer.logger. I'm still testing to see if this works, as it may require a callback to allow the loggers to boot up correctly,  but it would allow any logger to be restored properly during training. Does this design sound OK?

So it looks like this doesn't quite work because of the way _pre_training_routine in trainer is written (https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L538)
_pre_training_routine calls the on_pretrain_routine_start hook, which ModelCheckpoint registers a handler for (https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/callbacks/model_checkpoint.py#L211) . Inside of that handler, the ModelCheckpoint tries to compute the checkpoint directory using the logger version property (i.e., it reads that property). Then later _pre_training_routine calls checkpoint_connector.restore_weights() which reads the stored version string from the checkpoint and overwrites the version property on the logger if possible.
This procedure all works in my tests but there are two issues:

Since the ModelCheckpoint needs to run first to determine the checkpoint directory, it could potentially be using the wrong version string since the logger will generate a new one instead of reading it from the hpc checkpoint, so it would get changed later in this process without ModelCheckpoint knowing. It also potentially needs this version string in order to determine the correct directory to load the hpc_checkpoint from: a chicken/egg problem.
Any logger that creates assets, like mlflow does, when the version string is generated will end up having empty runs created. So for example, every time I do a slurm requeue, it makes a new run on my mlflow UI, then loads the correct one and continues, so you get a bunch of junk in the UI.

So I'd like to get some feedback from the maintainers on if there's a better way to implement this, it may be quite complex to break this up.

@tchaton do you have any input on this or do you know of anyone else on the team that might?

@PyTorchLightning/core-contributors

This issue has no repro. Is it still a problem in the latest version?


Basic usage issue

Hi,
I'm attempting to render a large table (such as the one in your 500k example), but I'm obviously doing something wrong.
I have an array like this:
var data = ["
fizz	buzz	bar
", ...]
My JS is:
var clusterize = new Clusterize({     rows: data,     scrollId: 'scrollArea',     contentId: 'contentArea' });
The HTML is also copy-pasted from your website (without the 'Loading data...' part).
I included the library like this:
<link href="https://cdnjs.cloudflare.com/ajax/libs/clusterize.js/0.17.6/clusterize.min.css" />

I do get some data rendered, but it ends with:

and if I continue scrolling, it's just a blank screen, although I am still inside the #contentArea.
Can you tell me what am I doing wrong?
Thanks!

Hi
Everything seems to be fine in your settings.
It hugely depends on content in your data array. Please provide reproducible example at codepen

Ok, here's the link: https://codepen.io/dinkom/pen/xqLGNb
It's the same even if I drastically reduce the size of data (from 1 million to 10k, for instance). Also, codepen thinks I have an infinite loop when I create a million entries, but I'm guessing it's just because it takes some time to run. If you try with 10k cap, the warning disappears, but it still doesn't work as expected.

You missed Clusterize's styles. That's it

Locally, I had styles included... but you still solved the problem for me because now I know that styles were the issue... so I took one more look at them - and I saw I didn't write rel="stylesheet" in the link... so I basically didn't include them 😭
Thanks man, I owe you a beer 🍺

you are welcome :)
\


[loc][query][VSCode] please provide more context

Hi, we have a question from our translator. Can you help please? Thanks!
Source:
skipped due to 'skipFiles'
skipped due to 'smartStep'
File Name:
out\node\nodeDebug.nls.resx
Resource ID:
".resx", 0; "source.skipFiles"
".resx", 0; "source.smartstep"
Query:
Can you please clarify what is skipped here?

These messages show up on stack frames that are grayed out while debugging. Usually when you 'step in' while debugging, the debugger will pause in the file that you are stepping into. But if you are using the skipFIles or smartStep attributes in your launch config, it may not pause in that file. So these messages explain why the file was skipped.


add health check queryParam for servers

follow up on https://github.com/apache/pinot/pull/9031
With the 2 new added APIs. we also want to make /health path return either liveness or readiness based on query parameter.
This way we can switch the default return health check from readiness to liveness in the future.

Codecov Report

Merging #9129 (1be2a07) into master (49c0e24) will decrease coverage by 41.48%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #9129       +/-   ##
=============================================
- Coverage     69.94%   28.45%   -41.49%
+ Complexity     4676       51     -4625
=============================================
  Files          1838     1826       -12
  Lines         97397    96999      -398
  Branches      14676    14634       -42
=============================================
- Hits          68120    27599    -40521
- Misses        24538    66788    +42250
+ Partials       4739     2612     -2127

Flag
Coverage Δ

integration1
26.32% <0.00%> (-0.06%)
:arrow_down:

integration2
24.83% <0.00%> (+0.01%)
:arrow_up:

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

 Take ten seconds to tell us how you rate us.


Upgrade to universal NFT support

This is the implementation used in Android to fully support NFT on every chain with Etherscan, Blockscout and 'Polygon/Matic' APIs.
(note: Etherscan, Blockscout and Matic do things in slightly different ways, all of which affect us)
You will already be familiar with the 'tokentx' route provided by Etherscan:
https://api-kovan.etherscan.io/api?module=account&action=tokentx&startBlock=1&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
You may be familiar with the equivalent route for NFT transfers:
https://api-rinkeby.etherscan.io/api?module=account&action=tokennfttx&startBlock=1&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
Note the action= 'tokennfttx' replaces 'tokentx' to get NFT transfers in Etherscan. Blockscout and Matic don't have the 'tokennfttx' route, they lump all NFT and FT transfers together. Fantom supports both the Etherscan routes, see below.
If you examine the dump from the above route you may see that each unit looks like this:
  "blockNumber": "8365144",
  "timeStamp": "1617710811",
  "hash": "0x06b9393c50186cbde6edc5f4e751145aca005ee7b14af6609b0c95885e438a9c",
  "nonce": "7",
  "blockHash": "0x7ddc3ed5d76d28fa94181279839d8d61d2ef27cb91f7753dfbd0d523ab24d962",
  "from": "0x0000000000000000000000000000000000000000",
  "contractAddress": "0xa567f5a165545fa2639bbda79991f105eadf8522",
  "to": "0xa20efc4b9537d27acfd052003e311f762620642d",
  "tokenID": "4",
  "tokenName": "Stormbird Art Contract",
  "tokenSymbol": "SART",
  "tokenDecimal": "0",
  "transactionIndex": "15",
  "gas": "432000",
  "gasPrice": "1000000000",
  "gasUsed": "147866",
  "cumulativeGasUsed": "2961783",
  "input": "deprecated",
  "confirmations": "279448"
}

Note that we have tokenId in the transfer description above. For Blockscout and Matic chains, to distinguish an NFT transfer from a normal FT transfer you see if there's a tokenId or a value respectively. (Note/TODO: find out what an ERC1155 transfer looks like; it should contain both tokenId and value.)
Providing we parse the transfers in ascending order, and knowing our wallet address we can detect transfer to and transfer from our account. By summing the transfer-to and transfer-from we can maintain a token balance.
Once you have the token balance; for any tokenId that doesn't yet have a metadata description (ie the name,image_url,description you normally get from Opensea), you need to fetch this:
    // call contract: function tokenURI(uint256 tokenId) public returns (string)
    private static Function getTokenURI(BigInteger tokenId)
    {
        return new Function("tokenURI",
                Arrays.asList(new Uint256(tokenId)),
                Arrays.<TypeReference<?>>asList(new TypeReference<Utf8String>() {}));
    }

    // call contract: function uri(uint256 tokenId) public returns (string)
    private static Function getTokenURI2(BigInteger tokenId)
    {
        return new Function("uri",
                Arrays.asList(new Uint256(tokenId)),
                Arrays.<TypeReference<?>>asList(new TypeReference<Utf8String>() {}));
    }

String responseValue = callSmartContractFunction(web3j, getTokenURI(tokenId), getAddress(), getWallet());
if (responseValue == null) responseValue = callSmartContractFunction(web3j, getTokenURI2(tokenId), getAddress(), getWallet());
        JSONObject metaData = loadMetaData(responseValue);

The result from the call to getTokenURI or uri (whichever works for the token) should be JSON and should look vaguely like this:
{
   "image":"http://james.lug.org.cn/tokenimage/fafa5a175353f72dccedb7e5ff6b5c92",
   "name":"Buena Vista Social Club",
   "description":"slightly overrated."
}

Note that some ERC721 follows the openZeppelin tokenURI standard, and some implement the, presumably legacy, uri. Since tokenURI seems to be a lot more common we try that first. the legacy version is used by a lot of artworks for sale on opensea, so we should support it.
For supporting blockscout chains, eg Eth Classic, xDai (which has a big NFT following) you would just use the normal tokentx route:
https://blockscout.com/poa/dai/api?module=account&action=tokentx&startBlock=2&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
For Polygon/Matic and Fantom Mainnet (Fantom testnet is not currently supported - it is only supported by covalent), the route looks like this:
Matic Testnet (Mumbai):
https://explorer-mumbai.maticvigil.com/api/v2/transactions?module=account&action=tokentx&startBlock=13123851&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
Polygon mainnet:
https://explorer-mainnet.maticvigil.com/api/v2/transactions?module=account&action=tokentx&startBlock=13123851&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
Fantom mainnet. Note: ftmscan is Etherscan compatible and supports both tokentx and tokennfttx, so treat it the same as the Etherscan routes WRT to the two routes:
https://api.ftmscan.com/api?module=account&action=tokennfttx&startBlock=1&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
https://api.ftmscan.com/api?module=account&action=tokentx&startBlock=1&address=0xa20efc4b9537d27acfd052003e311f762620642d&page=1&offset=100&sort=asc
Doing things this way means we can support discovery and keep current balance of NFT on almost every chain. Only Avalanche can't be supported like this, yet. There is a covalent route that gives you raw token balances which we can use for the main wallet page and to discover Tokens, but won't provide the receive event - we'd need a generic TokenScript or an event log scanner for that.
I will outline in a different issue how I got the full transaction from covalent (Fantom test, Avalanche and Fuji), together with some chicanery to get transaction and constructor events.

@JamesSmartCell do you have an example NFT on Polygon that implements tokenURI so I can verify? I tried with a few and they don't. The one we have on Rinkeby works.

Helps a lot. Thanks James!


Uploaded lock icon

Notes for Reviewers
@iArchitSharma , for your review.
This PR fixes #126
Signed commits

[x] Yes, I signed my commits.

We are not using svg anymore so closing the PR


Possible infinite loop in ZOOTR plugin

https://github.com/authorblues/bizhawk-shuffler-2/blob/409555a3048b17f8fd14ab54b883c5dd5f6be346/plugins/solo-zootr-multiworld.lua#L233-L236
A key having a nil value is the same as a key not existing in a Lua table. Inserting nil will not increment the table length, so this code should result in an infinite loop.

Thanks for catching this! This code path should hopefully happen so infrequently that I might never have caught the problem.


how to run on a new dataset?

As the title, I am not familiar with jap and tf. Looking forward to your advice. By the way, my experiments will be conducted on Linux and a single GPU using PyCharm.

Hi @YonghanLU, thanks for your question.
I understand that it may not be obvious as we are using tfds. Let me try to briefly explain how this would work, and the four existing datasets could potentially also be helpful for illustrations. Meanwhile, for JAX/Flax, I would recommend checking Flax examples which are great resources.
The main steps of adding a custom dataset with tfds for segmentation would be:

create a new folder in imgx_datasets/ https://github.com/mathpluscode/ImgX-DiffSeg/tree/main/imgx_datasets such as demo.
under imgx_datasets/custom_dataset create demo_dataset_builder.py, inside which we need to create a new class Builder, similar to male_pelvic_mr_dataset_builder.py, you can also read the TensorFlow documentation https://www.tensorflow.org/datasets/add_dataset. The implementation does not require much TensorFlow knowledge. The main requirement is that _generate_examples should yield uid, {UID: uid, IMAGE: image[..., None], LABEL: label} there the IMAGE should have (spatial_dimensions, input_channel).

Technically it is not a hard requirement as the JAX/FLAX framework is compatible with any data loaders. Together with #15, I understand that some toy examples/usages could potentially be more helpful. I will try my best to provide some examples in the coming weeks.

@mathpluscode Thanks. Another question also needs your help. In the reverse process, why only need 5 steps? I am not clear from your two papers and code. Is it randomly sampled 5 values from [T-1, 0] ? Can you give me a detailed pytorch-version source about this sampling method?

@mathpluscode Thanks. Another question also needs your help. In the reverse process, why only need 5 steps? I am not clear from your two papers and code. Is it randomly sampled 5 values from [T-1, 0] ? Can you give me a detailed pytorch-version source about this sampling method?

Dear @YonghanLU, assuming your "reverse process" is the denoising process (right to left in the figure below)

The corresponding code is at https://github.com/mathpluscode/ImgX-DiffSeg/blob/main/imgx/diffusion/gaussian/variance_schedule.py#L101. Although these codes are in JAX, we are using jax.numpy which has exactly the same signature as numpy. The relevant literature is https://arxiv.org/abs/2102.09672

In a nutshell, given 1001 steps, we sample 5 steps evenly, 1, 251, 501, 751, 1001, then uses alphas_cumprod or $\bar{\alpha}_{t}$ for $t = 1,251,501,751,1001$ to redefine the betas

Such a design can be considered as a hyper-parameter. You may be interested in the following literature which discusses more noising schedules

https://arxiv.org/abs/2301.10972
https://arxiv.org/abs/2305.08891

Please let me know if this is still unclear for you :)

Thanks for your detailed explanation! Btw, the training process and testing process both use the resampling strategy or only the testing process ?

Thanks for your detailed explanation! Btw, the training process and testing process both use the resampling strategy or only the testing process ?

Hi @YonghanLU, the resampling is certainly used for testing/inference for performance consideration. For training, it was used in our workshop paper https://arxiv.org/abs/2303.06040 as it showed improvements, however in our latest work https://www.melba-journal.org/papers/2023:016.html, we only observed the improvements in the muscle ultrasound data set but not in the abdominal CT data set. We thus did not apply the resampling for training in this work. This work only focuses on the recycling strategy itself.
However, this is indeed an interesting observation and this potentially may be linked to the noising schedule, noising method, application (network, data, etc.).


Require authentication and TCs acceptance for all the API

Note: this PR depends on jw35-add-referer which should be merged first.
This PR does the following:

Requires authentication and rate throttling for access to the Transport API endpoints
Requires TCs agreement before accessing any of the API
Adds a management command to set up API tokens for use by the tfc_web code, replacing the stand-alone script that just set TFC_WEB INTERNAL
Arrange to pass one of these tokens (TFC_WEB JS) into the smatrpanel code to authenticate access by widgets and by widget configuration to the transport API

Closes #212

Closes #179

Replaced by #226


addition of female population percentage of jordan and jamaica.

Hi, I want to add the female population percentage for jordan and jamaica.
Can I go ahead?

Hi, I want to go ahead with the issue of addition of female population percentage of jordan and jamaica.
Kindly allow me to do so

hii i want also want to explore this issue of addition . kindly allow me to do so


ARTEMIS-4314 Small Tweak: using executor directly if no delay

@gtully either merge it or close if you prefer not to do this please?
Thanks

tests are good.. I'm merging this


[BugFix] Fix runtime filter delivery

What type of PR is this：

[X] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #
Fixes daily test tpcds-1g query64
Problem Summary(Required) ：

This PR fixes:

different wait timeout on scan node(_scan_wait_timeout_ms) scan and other nodes(_wait_timeout_ms)
put global runtime filter into cache(runtime_filter_cache) if fragment instance is not allocated yet.
And during wait time, nodes on fragment instance will find runtime filter from cache(runtime_filter_cache)

Currently, which operator still run in non-pipeline mode?
Could we use simple and special test case to cover this bug?

Currently, which operator still run in non-pipeline mode?

Every node can run in pipeline mode. But some test cases are still use non-pipeline mode.

Could we use simple and special test case to cover this bug?

It's transient case. It happens when runtime filter is delivered to some node but fragment instance is not allocated yet. We can add some mechanism to make this case happens with higher probablity, but can not make it definitely.

run starrocks_admit_test

run starrocks_admit_test


Initiate payment request use case based

@martiBpl Greetings and thanks for raising a PR!
Would you mind sparing some time and putting down some explanations on the actual problem this PR attempts to resolve?
From what I can see right now - both InitiateBankPaymentRequest & InitiateCardPaymentRequest provide just a wrapper around InitiatePaymentRequest's bankPaymentMethod & cardPaymentMethod, and that hardly seems like the best potential solution for the problem
On top of that - overriding equals & hashCode does make sense based on field types, as long as you encounter comparing the InitiatePaymentRequest objects (was that the original reason for the PR?)
I must note, though, that solving this problem via static builder functions & keeping data class type, would have saved some lines of code as well as keeping the toString method

Hi @Mantas-Ragauskas,
The main point of this PR was to have a non nullable fields (mostly bankPaymentMethod) where we know exactly that we're initiating a bank payment. Based on that concrete model we wouldn't have check again if the bankPaymentMethod is null and provide dummy default values if its not - and we are sure that it's not.
Given this the builder pattern wouldn't help as it would still be consisting nullable fields.
To be honest the equals and hashCode methods are autogenerated by the IDE, those where by default provided by the kotlin data class which I needed to get rid of, because of the inheritance.
One other solution would be to create a separate model class (non inherited) which would have a method mapToInitiatePaymentRequest.
Side question: from the docs I've read that when creating a card payment you have to always provide a bankPaymentMethod

Initiate payment by calling the /pis/payment endpoint with both bankPaymentMethod and cardPaymentMethod objects, and set paymentMethodPreferred to card

Does this mean that the bankPaymentMethod is required at all times?

Does this mean that the bankPaymentMethod is required at all times?

It should be. It would be way easier to just mark it as mandatory, even if that is only a business rule that is not implemented on the API yet.


Improve Travis Config to Build & Test All Configurations

I think this pull request is now complete. Wren now successfully builds in 32 and 64 bit on OS X and Linux on Travis. I've also added in builds which disable NaN tagging which run on the same platforms.
munificent/wren/#167

Ok. I'm pretty happy with this now. There's still a couple of issues though:

Gcc on Linux seems to fail to compile libuv when WREN_NAN_TAGGING is disabled for some reason.
A few of the tests fail when run with a 32 bit Wren:

There's a literal that's too big in example/syntax.wren:133
The test/language/deeply_nested_gc.wren test takes a very long time to complete on 32 bit builds. I'm guessing that this is to do with poor allocation or gc performance but I've not been able to investigate it well enough yet.

GC slowdown seems to be in the GC, not in reallocation:

Looks like there's a sign overflow or underflow in the logic for choosing the next collection point on 32 bit. The 64 bit build makes it through the run with 2 gcs whereas the 32 bit one seems to collect at each allocation.
64:
GC 104857645 before, 104831820 after (25825 collected), next at 157247730. Took 0.033s.
-- gc --
GC 118473276 before, 118473484 after<PHONE_NUMBER>3709551408 collected), next at 177710226. Took 0.046s.
done

32:
-- gc --
GC 104857785 before, 104835472 after (22313 collected), next at 28404189. Took 0.050s.
-- gc --
GC 104835500 before, 104835728 after<PHONE_NUMBER> collected), next at 28404573. Took 0.041s.
-- gc --
GC 104835984 before, 104835756 after (228 collected), next at 28404615. Took 0.039s.
-- gc --
GC 104835784 before, 104836012 after<PHONE_NUMBER> collected), next at 28404999. Took 0.047s.
-- gc --
GC 104836268 before, 104836040 after (228 collected), next at 28405041. Took 0.040s.
-- gc --
GC 104836068 before, 104836296 after<PHONE_NUMBER> collected), next at 28405425. Took 0.031s.
-- gc --
GC 104836552 before, 104836324 after (228 collected), next at 28405467. Took 0.038s.
-- gc --
GC 104836352 before, 104836580 after<PHONE_NUMBER> collected), next at 28405851. Took 0.040s.
-- gc --
GC 104836836 before, 104836608 after (228 collected), next at 28405893. Took 0.045s.
-- gc --
GC 104836636 before, 104836864 after<PHONE_NUMBER> collected), next at 28406277. Took 0.040s.
...

Seems to be lots of test failures on travis..

Yes. There's two main groups of failures. The first is lots of failures seem to be due to precisision issues when NAN_TAGGING is disabled on 32 bit builds. The second is that libuv doesn't compile when NAN_TAGGING is disabeld on linux for 32bit gcc builds.
I'm looking into what can be done about the precision issues, but i'm stumped as to why that one linux configuration fails to build libuv, especially seen as it builds fine when we don't disable WREN_NAN_TAGGING.

Why it happens, it's because there is an assert that compare a size_t to a ssize_t (but why only gcc x86 linux? dunno, maybe the only one with "treat warning as error" on?).
struct uv_fs_s { // the type of "read_req"
  ssize_t result; // other fields omitted
};
size_t iovcount;
ASSERT(read_req.result == sizeof(test_buf) * iovcount);

and a few line above, there is : ASSERT((size_t)write_req.result == sizeof(test_buf) * iovcount);
by the way, the erronous line has been corrected with libuv 1.9.0, a cast as been added.

Sorry, when I checked, I have check the wrong line :-1: the master is still missing the cast.

Yep, it's https://github.com/libuv/libuv/blob/229b3a4cc150aebd6561e6bd43076eafa7a03756/test/test-fs.c#L2604 that's causing the error. I've tried digging into whats wrong with the build when WREN_NAN_TAGGING is disabled. It looks like method dispatch is broken quite badly. Just trying to run the resulting build as a repl fails:
$ bin/wrend-32
\\/"-
 \_/   wren v0.1.0
> [src/vm/wren_vm.h:210] This code should not be reached in wrenGetClassInline()
Abort trap: 6

I've opened a PR to fixup the libuv compilation error.

Ok, so i've got rid of the 32 builds with NAN_TAGGING disabled and things are looking a lot healthier. Unfortunately i'm not able to reproduce the precision issues that travis seems to see with cos.wren and sin.wren locally with a VM. I am however able to reproduce the leaked handles from the GCC build and the apparent inadvertent cast to 32 bit integer on clang builds in bitwise_not.wren.
To fix the issue with sin and cos failing to return exactly the correct value I'm wondering if we should have an expect statement which takes an epsilon to compare with?

Clang on linux seems to be generating an overflow check when converting values to 32 bit for the not function:
RETURN_NUM(~(uint32_t)AS_NUM(args[0]));
``

// Function prolog
0x0805e8e0 <+0>:	push   %ebp
0x0805e8e1 <+1>:	mov    %esp,%ebp
0x0805e8e3 <+3>:	push   %ebx
0x0805e8e4 <+4>:	sub    $0x24,%esp
// get the function's address to to IP-relative loads
0x0805e8e7 <+7>:	call   0x805e8ec <prim_num_bitwiseNot+12>
0x0805e8ec <+12>:	pop    %eax
// store the address of some constant, at some point relative to the IP, in eacx for later.
0x0805e8ed <+13>:	add    $0x32714,%eax
// Load the parameters to the funciton into registers
0x0805e8f3 <+19>:	mov    0xc(%ebp),%ecx
0x0805e8f6 <+22>:	mov    0x8(%ebp),%edx
// Spill the regsiters
0x0805e8f9 <+25>:	mov    %edx,-0x8(%ebp)
0x0805e8fc <+28>:	mov    %ecx,-0xc(%ebp)
0x0805e8ff <+31>:	mov    %eax,-0x1c(%ebp)
=> 0x0805e902 <+34>:	mov    -0xc(%ebp),%eax
0x0805e905 <+37>:	mov    (%eax),%ecx
0x0805e907 <+39>:	mov    0x4(%eax),%eax
0x0805e90a <+42>:	mov    %esp,%edx
0x0805e90c <+44>:	mov    %eax,0x4(%edx)
0x0805e90f <+47>:	mov    %ecx,(%edx)
// call wrenValueToNum and store the return on the stack
0x0805e911 <+49>:	mov    -0x1c(%ebp),%ebx
0x0805e914 <+52>:	call   0x8061f30
0x0805e919 <+57>:	fstpl  -0x18(%ebp)
/// move the return value into an XMM register
0x0805e91c <+60>:	movsd  -0x18(%ebp),%xmm0
// Get the IP-relative stuff back from the stack into EAX
0x0805e921 <+65>:	mov    -0x1c(%ebp),%eax
// load something from there into xmm1
0x0805e924 <+68>:	movsd  -0xfef0(%eax),%xmm1
// copy the value retuned from valueToNum into xmm2 as well
0x0805e92c <+76>:	movaps %xmm0,%xmm2
// subtract the mystery contant from it
0x0805e92f <+79>:	subsd  %xmm1,%xmm2
// store the result in ecx, but with the sign bit flipped
0x0805e933 <+83>:	cvttsd2si %xmm2,%ecx
0x0805e937 <+87>:	xor    $0x80000000,%ecx
// compare the mystery constant with the valueToNum result
0x0805e93d <+93>:	ucomisd %xmm1,%xmm0
// Move the double from valuetoNum to edx
0x0805e941 <+97>:	cvttsd2si %xmm0,%edx
// overwrite the integer represetion of the return of valtoNum if CF is set
0x0805e945 <+101>:	cmovb  %edx,%ecx
// not whatever integer representation we have
0x0805e948 <+104>:	not    %ecx
// Move it back to fp registers, spill things generally get ready to call
0x0805e94a <+106>:	movd   %ecx,%xmm0
0x0805e94e <+110>:	movsd  -0xfee0(%eax),%xmm1
0x0805e956 <+118>:	movaps %xmm1,%xmm2
0x0805e959 <+121>:	por    %xmm2,%xmm0
0x0805e95d <+125>:	subsd  %xmm1,%xmm0
0x0805e961 <+129>:	mov    %esp,%ecx
0x0805e963 <+131>:	movsd  %xmm0,(%ecx)
0x0805e967 <+135>:	mov    %eax,%ebx
// convert bcak to a value
0x0805e969 <+137>:	call   0x8061f10
0x0805e96e <+142>:	mov    -0xc(%ebp),%ecx
0x0805e971 <+145>:	mov    %edx,0x4(%ecx)
// more stack switcheroo to return the value
0x0805e974 <+148>:	mov    %eax,(%ecx)
0x0805e976 <+150>:	mov    $0x1,%al
0x0805e978 <+152>:	and    $0x1,%al
0x0805e97a <+154>:	movzbl %al,%eax
0x0805e97d <+157>:	add    $0x24,%esp
0x0805e980 <+160>:	pop    %ebx
0x0805e981 <+161>:	pop    %ebp
0x0805e982 <+162>:	ret

I'm not 100% sure of my interpretation of the assembly output here, or if this is correct behavior or not for the code.

I can reproduce the last failure with the following minimal c example:
#include <stdio.h>
#include <stdint.h>

int main(int argc, const char* argv[]) {
	double foo = -4294967293;
	printf("%.14g\n", foo);
	foo = ~(uint32_t)foo;
	printf("%.14g\n", foo);
	return 0;
}

on my laptop it prints:
$ ./foo
-4294967293
4294967292

However on ubutnu/trusty64 it prints the following:
$ ./foo
-4294967293
2147483647

If i enable optimisation however the output is the same as above. This appears to be because the compiler has removed the not instruction & casting entirely and just pre-calculated the result however.

It looks like the prim_num_bitwiseNot is triggering undefined behaviour when we give it numbers that overflow according to StackOverflow. Both behaviours are acceptable and within the standard.

Ok. So i've fixed the bitwise not test by making the overflow behaviour of it a bit more explicit in the code. This seems to stop Clang from emitting runtime checks for overflow and brings back the expected result on all platforms.
This should be good to merge now. Is it OK as is, or shall I sqash it down to fewer commits?

This is so so awesome. I am a bad person for getting distracted by other projects and taking so long to get to this. It's going to take me a little while to wrap my head around it all, but it looks great so far.
Thank you!

This is the bestest thing ever! I tweaked a few things here and there but it's all in now. You rock! 🤘


[Question] Import statement must be top, but this one is not necessary? (So Magic)

Thank you for this wonderful tool!
I have a question, I use this with create-react-app v2.
I discover I can do this!!!
import importAll from 'import-all.macro';
let result
if (someCondition) {
  result = importAll.sync(`./someDir`);
} else {
  result = importAll.sync(`./otherDir`);
}

So weird it works!
Import statement must be top? Isn't it?
I google something like "conditional import in create-react-app", but result is dynamic import.
But importAll.sync is not dynamic import, how did all this work?

I'm not sure exactly why or how it works, sorry.


Intermitent packet loss in ARP app

Follow-up on https://github.com/Igalia/snabb/issues/607
Problem
On running lwaftr check, showlinks reveals lwaftr.v4 sends packets into arp.south, but ARP:push doesn't receive any packets, actually ARP:push is not even called.
$ ./snabb lwaftr check lwaftr-migrated.conf \
   inputv4.pcap empty.pcap outputv4.pcap outputv6.pcap
5 sent on arp.north -> icmpechov4.south (loss rate: 0%)
0 sent on arp.south -> icmpechov4.north (loss rate: 0%)
5 sent on capturev4.output -> reassemblerv4.input (loss rate: 0%)
0 sent on capturev6.output -> reassemblerv6.input (loss rate: 0%)
0 sent on fragmenterv4.output -> output_filev4.input (loss rate: 0%)
2 sent on fragmenterv6.output -> output_filev6.input (loss rate: 0%)
5 sent on icmpechov4.north -> lwaftr.v4 (loss rate: 0%)
0 sent on icmpechov4.south -> fragmenterv4.input (loss rate: 0%)
0 sent on icmpechov6.north -> lwaftr.v6 (loss rate: 0%)
2 sent on icmpechov6.south -> fragmenterv6.input (loss rate: 0%)
0 sent on lwaftr.hairpin_out -> lwaftr.hairpin_in (loss rate: 0%)
2 sent on lwaftr.v4 -> arp.north (loss rate: 0%)
2 sent on lwaftr.v6 -> ndp.north (loss rate: 0%)
0 sent on ndp.north -> icmpechov6.south (loss rate: 0%)
2 sent on ndp.south -> icmpechov6.north (loss rate: 0%)
5 sent on reassemblerv4.output -> arp.south (loss rate: 0%)
0 sent on reassemblerv6.output -> ndp.south (loss rate: 0%)

The relevant bit is:
2 sent on lwaftr.v4 -> arp.north (loss rate: 0%)
0 sent on arp.south -> icmpechov4.north (loss rate: 0%)

It should be:
2 sent on lwaftr.v4 -> arp.north (loss rate: 0%)
2 sent on arp.south -> icmpechov4.north (loss rate: 0%)

outputv4.pcap should contain 2 packets:
reading from file outputv4.pcap, link-type EN10MB (Ethernet)
IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP host <IP_ADDRESS> unreachable, length 36
IP <IP_ADDRESS> > <IP_ADDRESS>: ICMP host <IP_ADDRESS> unreachable, length 36

The issue doesn't happen in lwaftr-snabbvmx branch (which points to a commit by the end of October), but it happens, intermently, in lwaftr.

issue-698.tar.gz

Not able to reproduce it in current tip of lwaftr branch. I run the below command 60 times and I always got the same expected result.
sudo ./snabb lwaftr check ./program/lwaftr/tests/data/icmp_on_fail.conf inputv4.pcap empty.pcap outputv4.pcap outputv6.pcap | grep "arp.south"
   3 sent on lwaftr.v4 -> arp.north (loss rate: 0%)
   3 sent on arp.south -> icmpechov4.north (loss rate: 0%)


Issue with fill command

Hi there @TheBeems!
I have been doing a project for school where I need to build a hub kind of thing and I wanted to speed up the process by using your Github repository. When I managed to get it into Minecraft education everything ran fine but when I start to type "fill" [Whatever Block] it just cuts the air leaving nothing in place. It might just be me not understanding but I have typed it multiple times and nothing has changed whatsoever other than that, I find it really cool that you made this! I would appreciate it if you could find my problem since this is kind of a big project I'm working on at school.
Best Wishes,
@Darkshipreturn

Hi @Darkshipreturn , sorry for the late reply.
In order for the fill command to work, you need to set atleast 2 marks. You can set a mark with the wooden axe (right click). If you don't have a wooden axe, you can receive one by using the command 'wand'.
When you have set 2 marks, you can type in 'fill' and the area between it will filled with the default block.
I hope this helps. Let me know if it worked.
Best Regards


Add warning/error when unnecessary pattern match

What is your idea? Provide a use case.
When having the following code:
type Foo {
  A
  B
}

fn foo(a: foo) {
  when foo is {
    A -> todo
    B -> todo
    _ -> todo
  }
}

The wildard _ option is useless there since the parameter a won't ever have any other value than A or B.
The parser isn't currently providing any warnings or errors on that dead code.
Why is it a good idea?
Code cleanness.
What is the current alternative and why is it not good enough?
No alternative.

@unijad let's discuss ideas first before implementing them.


Automatic checkpoint fetching when starting HeliosLightClient

Check points now are fetch automatically when starting helioLightClient
Pull Request type

Please check the type of change your PR introduces:

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build-related changes
[ ] Documentation content changes
[ ] Testing
[ ] Other (please describe):

What is the current behavior?

Issue Number: N/A
What is the new behavior?

Does this introduce a breaking change?

[ ] Yes
[ ] No

Other information

Codecov Report
Base: 83.63% // Head: 83.64% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (aa7d6d4) compared to base (457c802).
Patch coverage: 68.42% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
+ Coverage   83.63%   83.64%   +0.01%
==========================================
  Files          21       21
  Lines         898      911      +13
==========================================
+ Hits          751      762      +11
- Misses        147      149       +2

Impacted Files
Coverage Δ

beerus_cli/src/main.rs
4.76% <0.00%> (ø)

beerus_rest_api/src/main.rs
5.26% <0.00%> (+0.50%)
:arrow_up:

beerus_core/src/config.rs
91.30% <71.42%> (-8.70%)
:arrow_down:

...ore/src/lightclient/ethereum/helios_lightclient.rs
28.57% <100.00%> (+1.74%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add time cast to SymbolFileDWARFDebugMap

After TimePoint's precision was increased in LLVM we started seeing
failures because the modification times didn't match. This adds a time
cast to ensure that we're comparing TimePoints with the same level of
precision.
Based on r347660 on llvm.org

@swift-ci please test and merge


AGP 3.3.0-beta-01 with suggested classpath fixes

Resolves the issue raised in #526 with the recommended solution from the issue

Thanks @hzsweers !


[q] When you try to test on a different env (not webpack) how you skip svg imports

Scenario
React component working with webpack very regular mstuff but when you try to test without webpack on the equation.
Error: Cannot find module '-!babel!svg-react!./../../assets/waitspinner.svg'

any work around ?
or any official way to skip this or a nice way to include this on the tests?

You should not use require("!./image.svg") then. You should use the loader config in webpack.config.js. Now with node require.extensions (which is a deprecated feature), you can handle how svgs are handled. Or as suggested in the node docs, you can do compilation to replace/remove these requires in a build step before you run the code. You can write a babel plugin that does this. Or if you're determinant of how you'll use the require statements, you can as well write a regex to remove these requires/replace them with something that will make sense in your code.
OR another option would be (I would suggest this) -
To use react-svg-loader without webpack, you can use the CLI and transform you image.svg to image.svg.react.js and use require('./image.svg.react') and it will simply work, cuz the SVG is now transform to a React Component and is just JavaScript.


Known issue on agora_rtc_engine: ^6.0.0

There are some known issues on agora_rtc_engine: ^6.0.0 which will be supported in the future

Flutter web is not supported yet
iOS simulator not supported yet

Errors returned by flutter web                                                 ^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1587:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int captureTimeMs;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1604:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int speech;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1607:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int sendEvenIfEmpty;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1621:33: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external Func_MaxMetadataSize getMaxMetadataSize;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1623:37: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external Func_ReadyToSendMetadata onReadyToSendMetadata;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1625:34: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external Func_MetadataReceived onMetadataReceived;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1636:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int uid;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1639:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int size;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1641:35: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external ffi.Pointer<ffi.Uint8> buffer;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1644:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int timeStampMs;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1702:30: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external Func_AudioOnFrame onFrame;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1711:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int capture_timestamp;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1714:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int samples_per_channel_;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1717:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int sample_rate_hz_;
^
../../Documents/Development/flutter/.pub-cache/hosted/pub.dartlang.org/agora_rtc_engine-6.0.0/lib/src/impl/native_iris_api_engine_bindings.dart:1720:16: Error: Only JS interop members may be 'external'.
Try removing the 'external' keyword or adding a JS interop annotation.
external int bytes_per_sample;

Is there any estimate on when the issue with iOS Simulator will be fixed?

Why isnt the old sample apps not working anymore?

Can also confirm, im not able to launche the app on ios simulator. It's cruel developing on ios atm without a working simulator.

Any updates on the simulator fix?  It's prohibitively impactful that we cannot leverage ios simulator!
Thank you for prompt attention to this.

Using pods configuration in Podfile of example project worked for me.

@littleGnAl  Any estimation for web support on agora 6?

I can't understand if its not supporting web, why the pub page shows another thing?

it works fine for me. Hope it helps you guys Podfile
post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
        config.build_settings['ONLY_ACTIVE_ARCH'] = 'YES'
        config.build_settings["EXCLUDED_ARCHS[sdk=iphonesimulator*]"] = "arm64"
    end
  end
end

It Works for me too. Thank you

Is there any plan on adding support for web on version 6.0.0?

Excluding simulators from build configs is not really a solution for many use cases.
I'm currently testing some solutions for a big education institute (40+ companies) and thanks to such bugs, being unaddressed for 6+ months, Agora makes my decision fast and easy to scrap them off my list. Thanks!

iOS simulator is supported on agora_rtc_engine: ^6.1.0
https://pub.dev/packages/agora_rtc_engine

Flutter web support is planned but there's no timeline guaranteed at this time, you can vote on this issue, which may increase the priority internally.
https://github.com/AgoraIO-Extensions/Agora-Flutter-SDK/issues/844

Are there plans to add support rtm for flutter web?

Plans to add rtm support for flutter web?

Any update on the flutter web?

Hi, is web support currently being worked on?

Hi, is web support currently being worked on? Because in web do not worked this. So can you please provide alternative solution if any available.

It's also not working working on downgrading

Agora product managers..
Please It's a business impact on not having web support for your flutter packages ( RTM as well).
If I could go back i would have NOT leveraged agora in the solution with flutter.  I suspect everyone assumed web would be getting addressed concurrently when we selected flutter and agora's flutter integration into our solutions.
It's  business and technology limiting to not be able to deploy web with common code base.  Can you please prioritize this for RTM/RTC.
thank you.

@nholmvio Unfortunately their support on flutter plugin is unacceptable. We had a product with a custom implementation on their old implementation and it was so stable and promising. Then they introduced their new version of plugin with objective-c and java which ruined everything.
We already moved to LiveKit to overcome the limitations. It's not reliable on network connectivity as Agora has, but their developers are much more responsive and active.
I hope someday Agora understands flutter is growing and they are losing many customers in this area!

@papmodern: it seems LiveKit is more expensive compared to Agora (i.e., $0.02/minute vs. 0.004/minute). Did you install the open source version on-premise?

@github4f Nope. We are using their expensive server

@nholmvio Unfortunately their support on flutter plugin is unacceptable. We had a product with a custom implementation on their old implementation and it was so stable and promising. Then they introduced their new version of plugin with objective-c and java which ruined everything. We already moved to LiveKit to overcome the limitations. It's not reliable on network connectivity as Agora has, but their developers are much more responsive and active.
I hope someday Agora understands flutter is growing and they are losing many customers in this area!

Hi @papmodern ! I work at Agora. I'm so sorry to hear that you had a bad experience on our Flutter SDK. Would you mind share with us what are the issues you have with our new version of plugin? Your experience is important to us. We would like to learn and improve! Thank you in advanced!

It's also not working working on downgrading

@virendracarpenter If you downgrade to 5.3.1 it is working on web https://pub.dev/packages/agora_rtc_engine/versions/5.3.1


Not compatible with electron-pos-printer package

when im combine with electron-pos-printer for printing receipt, its not printing... always time out because tha package will creating another browserwindow for showing the print preview.

Thanks for opening this issue!
Somehow I didn't see a notification for this. Perhaps you opened this just as I ended support for this for this fork. I no longer use or recommend Electron.
That is a very specific use-case you got there. I don't even know what a thermal printer is. I hope you already solved the issue, otherwise I recommend trying your luck again with Tauri


Error page

IBM Cloud toolchain: Delivery Pipeline deployed LRP-web-app to dev, including commit ff386362638fbdc301cfd7891bd194f9677de1eb

IBM Cloud toolchain: Delivery Pipeline deployed LRP-web-app to test, including commit ff386362638fbdc301cfd7891bd194f9677de1eb

IBM Cloud toolchain: Delivery Pipeline deployed LRP-web-app to prod, including commit ff386362638fbdc301cfd7891bd194f9677de1eb


GCC-6.3.0 ptrdiff_t does not name a type

In file included from main.cpp:44:0:
../common/array2.h:28:4: error: ‘ptrdiff_t’ does not name a type
ptrdiff_t ni, nj;
^~~~~~~~~

This worked for me: I added #include <cstddef> to common/array2.h .

I made this change in master. @flgw can you please confirm that it's working now and if so, close this issue?

I made this change in master. @flgw can you please confirm that it's working now and if so, close this issue?


OpenAPI Tool fails to read certain OpenAPI specs.

Description
The OpenAPI tool doesn't always success in parsing an openapi spec (yaml).
Example spec that fails: https://api.drymergeqa.com/test-openapi
Steps to reproduce

Create an agent
Add a OpenAPI tool with the following schema: https://api.drymergeqa.com/test-openapi
Run the agent

Acknowledgements

[X] My issue title is concise, descriptive.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of superagent.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

@elisalimli would you mind having a look at this?
The issue currently is that some OpenAPI specs don't work with the LangChain openapi agent/chain. Perhaps there is a better way to do this. Please research and fix this isssue!

I think OpenAPI Tool's failing to read certain OpenAPI specs is an intended implementation by langchain. The current implementation can call the endpoints derived from the given openapi spec.
Closing this issue for now.


File dialog freezing on macOS

We need to track this winit issue.

I actually tried it on macOS and it's working fine. I guess the issue is only when using it from the winit event loop.


Achieve 100% test coverage for all new pipeline code

100% is an unrealistic stretch goal, but now that I have your attention... we should improve our test coverage from 0% to something more reasonable.
Identify the most critical areas of the pipeline and those with breakages in the past and add tests to improve our resilience. Ensure that the pipeline can gracefully handle malformed or unexpected inputs.

Must have: unit tests
Nice to have: integration tests (e.g. run against a sample of the crawl when it starts)

FYI, I'm a big fan of 100% test coverage - even if that involves a lot of # pragma: no cover comments so it's not actually 100%. Better to be explicit when it's not covered.
This prevents accidental regressions in coverage levels when new code is added and also encourages tests to be added (# pragma: no cover is always there but should not be first choice if it's possible and easy to add a test).


remove compression option for reading csvs

pandas.read_csv infers the compression used (if any) by default. See the documentation at https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html. So we can remove the compression argument.
the .to_csv() method also infers compression. if you pass a filename with a .gz extension, the file will be gzip-compressed.
https://github.com/p-koo/cipher/blob/bbfaaf454eff4c700a0d6eaec50997a072c2e2ce/cipher/preprocess/wrangle.py#L34-L37

I tried removing the compression option and get an error. So keeping it in for now.


Rename: neon-cluster-manager --> neon-cluster-operator

This will align us with standard Kubernetes lingo for this sort of thing.
Let's front-load this change as well as #1155 before we start seriously deploying Loopie clusters.

DONE


Update Apache Airflow jinja template DAG creation logic

Fix to include runtime image parameter in notebook_op

What changes were proposed in this pull request?

How was this pull request tested?

Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

DAG execution now fails with
Broken DAG: [/opt/airflow/dags/hello-generic-world-0722114004.py] The key has to be a string

this is due to incorrect R&R in the template. parsing issues should be resolved now, ran noaa pipeline test on teacloth and its seems to be running now, albeit very slow today for some reason


net/libp2p: Use tokio as the executor and TCP provider

Both Windows and Macos have the same failed test:
---- peer_manager::tests::ban::connect_to_banned_peer_libp2p stdout ----
thread 'peer_manager::tests::ban::connect_to_banned_peer_libp2p' panicked at 'assertion failed: `(left == right)`
  left: `DialError(IoError(ConnectionRefused))`,
 right: `PeerError(BannedPeer("12D3KooWQmmPVC2gBvHhWtst5LbGpNHuWiuyRtFsJ92kw8C6pVpj"))`', p2p\src\peer_manager\tests\ban.rs:164:9

failures:
    peer_manager::tests::ban::connect_to_banned_peer_libp2p

I hope the issue is fixed now. For some reason I thought the TCP configuration was missing port_reuse(false) but I had forgotten that it was removed few months ago exactly because it was causing in issues in Windows.


ENH: remove global vars in adapters

What do these changes do?

Related issue number

Fixes #352
Check code requirements

[x] tests added / passed (if needed)
[x] Ensure all linting tests pass

Could you rebase main branch and resolve the conflicts?

Some conflict needs to be addressed, other parts look great to me.


cannot close reference peek with esc without focusing an editor

It seems that the reference peek widget handles esc key before the keybinding service.

Ii still cannot get why it works VS Code, but not in Monaco. In both cases bubbling event handler is installed. The only difference that in VS Code on window, in Monaco on the standalone editor node. But it should not be matter since the target element belongs to the editor node and should respond first. In Monaco it happens, so esc does not close the reference peek, but in VS Code it does not hit.
@alexandrudima maybe you can give a hint how to fix? Would it be alright to install capturing event handler for Monaco or it will break some overlay widgets?

editor.stablePeek should be set to false

It is repoducible on the playground with 0.18.0.

@akosyakov I am sorry I am not 100% of the precise steps. I have opened find all references and tried the following:

focused the outer editor and pressed Escape => the embedded editor closed
focused the embedded editor and pressed Escape => the embedded editor closed

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


in ElasticKnnSearch added back create_index, add_texts, from_texts

Fixes https://github.com/hwchase17/langchain/issues/7117
Adding back create_index , add_texts, from_texts to ElasticKnnSearch
Quick Test
from langchain.vectorstores.elastic_vector_search import ElasticKnnSearch
from langchain.embeddings import ElasticsearchEmbeddings
Initialize ElasticsearchEmbeddings
model_id = "sentence-transformers__all-distilroberta-v1"
dims = 768
es_cloud_id =
es_user = ""
es_password = ""
test_index = "knn_test_index_012"
embeddings = ElasticsearchEmbeddings.from_credentials(
model_id,
es_cloud_id=es_cloud_id,
es_user=es_user,
es_password=es_password,
)
Initialize ElasticKnnSearch
knn_search = ElasticKnnSearch(
es_cloud_id=es_cloud_id,
es_user=es_user,
es_password=es_password,
index_name= test_index,
embedding= embeddings
)
Test adding vectors
Test add_texts method when index is not created
texts = ["Hello, world!", "Machine learning is fun.", "I love Python."]
knn_search.add_texts(texts)
Test from_texts method  when index is not created
new_texts = ["This is a new text.", "Elasticsearch is powerful.", "Python is fun."]
knn_search.from_texts(new_texts, dims=768)
Correctly throw an exception when index has not been previously created.
 # Test `add_texts` method
texts = ["Hello, world!", "Machine learning is fun.", "I love Python."]
knn_search.add_texts(texts)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/runner/langchain-1/langchain/vectorstores/elastic_vector_search.py", line 621, in add_texts
    raise Exception(f"The index '{self.index_name}' does not exist. If you want to create a new index while encoding texts, call 'from_texts' instead.")
Exception: The index 'knn_test_index_012' does not exist. If you want to create a new index while encoding texts, call 'from_texts' instead.

Correctly create new index
# Test `from_texts` method
new_texts = ["This is a new text.", "Elasticsearch is powerful.", "Python is fun."]
knn_search.from_texts(new_texts, dims=768)

The  mapping is as follows:
{
  "knn_test_index_012": {
    "mappings": {
      "properties": {
        "text": {
          "type": "text"
        },
        "vector": {
          "type": "dense_vector",
          "dims": 768,
          "index": true,
          "similarity": "dot_product"
        }
      }
    }
  }
}

Correctly index texts after index has been created
knn_search.add_texts(texts)

@benwtrent do you have time to do another review? I think I addressed all the issues
I removed ElasticKnnSearch as a subclass but tried to align the methods to be standard. I also return Document type.
I'm also not sure how I picked up 27 other files that show changing

@baskaryan Somehow I picked up 27 other files to change in this PR. Are you able to take a look? It should just be the langchain/vectorstores/elastic_vector_search.py file

Langchain repo underwent a large reorg.
Closing this PR in favor of https://github.com/langchain-ai/langchain/pull/8180


[Color Picker]Support lowercase hex format

📝 support lowercase hex format in color picker
What is the expected behavior of the proposed feature?  What is the scenario this would be used?
--- Add "HEX - #ffaa00" option to the color formats list, which only has "HEX - #FFAA00" currently.
If you'd like to see this feature implemented, add a 👍 reaction to this post.

Alteady tracked in #5379

@niaodan2b
What would be the usecase? Why do you explicit need lowercase hex?

@Aaron-Junker at this point, that issue is deemed closed so fine to have this.
@niaodan2b can you explain why you need this to be lower?  What is the user case here.

I got the same request, apparently lowercase is preferred in web development - https://github.com/martinchrzan/ColorPicker/issues/2

I got the same request, apparently lowercase is preferred in web development - martinchrzan/ColorPicker#2

It's also easier to type ;)

I DONT THINK SO :P

It's not available after 5379, #8305 should make this possible.


ForgedAPI Split Project

What to do?
Any part for a Repo.
Why not use a solution similar to the fabric api?
Because Architectury Loom does not allow forge mod to do this.

还是先把功能做全做完善再split吧（


Just wanted to polish the UI a little bit. :)


Fixed few bugs and updated requirements

Bugs:

DriveWithDiodeBridge(Drive)
Plot legends

@all-contributors please add @HannuHar for code and bug reports


Actions requiring root tag made by root user don't need root tag

Fixes #657 and #660
Instead of checking if user is superuser, the handlers and auths for jobs, gear, groups, and users check if the user's root is set to true.
Tests were modified to use as_user so that root was not true.
Changes Made

PUT, POST, and DELETE user only require admin and not root param (superuser) as well
User can get any other user, not just themselves

(Don't know if this belongs here) https://github.com/scitran/core/issues/660

POST and DELETE gears only require admin
POST, PUT, and GET next jobs only requires admin

Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

The group root tag change was moved to #800 because it would be easier for Dan to make that change at the same time as the group key changes.

Alright, then we can go ahead with what's here. SDK is ready, but needs manual merge work from myself.

Looks good, feel free to merge!


Debugger: Occasional loss of memory regions

This is going to be difficult to reproduce without a long running sample. Basically it is the same issue where you can't see the heap memory but it occurs after having been able to see it previously in the same session. Once it has disappeared it seems to be impossible to get it back. Refreshing the memory node and all of the objects does not work. It seems to happen after hitting a breakpoint, doing some stepping, refreshing some of the objects and then running for a bit until hitting a breakpoint and repeating.
Also a minor nuisance is that some of the object nodes, notably the memory node, stay expanded and automatically re-expand themselves regardless of how many times you collapse it.

@astrelsky Maybe a dumb question, but are those regions gone from the underlying debugger's point of view?  i.e. are these regions that have been freed?  If so, I guess we have several alternatives goign forward - vote for your preference.  (1) is to use the existing trace mechanism to view things, i.e. if you want to see memory from the past, use the Threads view to move back to that point.  (Guessing not your frst choice.) (2) would be to never remove memory.  (2) would not be my first choice, but might be ok if we can label it as stale in some fashion.  Adds a buch of complications for things like goto, i.e. what would the correct behavior be if the user asks to goto memory that is no longer there?  display nulls, display the last non-null value, cange the current tick to the last time in the past where it was good?
Re minor nuisance, that's weird - I have never seen that.  Which target?  All the time?  Reproducible?

The memory is still there, is valid and currently in use. It is not looking back in the trace it is in the present.
I think the tree bug should be as simple to reproduce as expanding the memory node, refreshing the objects, attempting to close the node and then try to look elsewhere in the tree.

No joy reproducing either today, sadly - will take another stab tomorrow.  If you hit upon other hints for reproducing, definitely post 'em.

No joy reproducing either today, sadly - will take another stab tomorrow. If you hit upon other hints for reproducing, definitely post 'em.

I'll cook up something fun to reproduce it.

Well, I did not get very far, I gave up on what should have been a simple sample to work with. Didn't even get the chance to make it fun. The idea here is to set a breakpoint before the final string is destroyed so you have something to look at in memory. The random sleep time is to throw a wrench at the debugger.
#include <iostream>
#include <string>
#include <Windows.h>

int main() {
    for (;;) {
        int t = rand() & 0xff;
        std::string s = std::to_string(t);
        std::string msg = std::string("Sleeping for ") + s + std::string(" ms");
        std::cout << msg << std::endl;
        Sleep(t);
    }
}

OK, got this running - thank you for the code!  And I think I have a fix.  Should be added to master next week.  Not entirely positive  I understand the problem despite the fix, so yell if you see this after the fix.

OK, got this running - thank you for the code! And I think I have a fix. Should be added to master next week. Not entirely positive I understand the problem despite the fix, so yell if you see this after the fix.

I'm always yelling about something so you'll probably here me either way.

LOL - think you've earned the right. :)

LOL - think you've earned the right. :)

I meant more in general. You should hear me when I'm stuck working with python2 or something that hasn't been maintained in 10 years but gets used daily.

It seems to be working ok now. Unless I just got lucky. There's still the occasional comparison contract violation and things get weird when you kill the process from the objects view but I'll open separate tickets once I can get at least a log or way to reproduce those.

@d-millar regarding the fix and punting when setting a breakpoint fails. If the process being debugged is running when you try to set a breakpoint it will always punt. This means that no matter how many times you click ok to set it the popup box for setting a breakpoint will stay there. I actually sat there and held down enter to repeatedly submit it and it never took it. While it is probably an edge case a message saying you can't set it while it is running would be nice.
As a bit of background context the process that I had in the debugger was python which was parked at a breakpoint in a python debugger which is why I didn't realize it was still technically running from Ghidra's viewpoint.

@astrelsky Think I need more context.  Are you setting the breakpoint in Objects, in the Static Listing, in the Dynamic listing, in Breakpoints, or from the Interpreter?  Testing with notepad.exe (ok, not ideal in some sense, but...), breakpoints are disabled in Objects and the Breakpoint viewer, the Static listing succeeds but marks the breakpoint as "ineffective_enabled", the Dynamic listing does nothing from "OK" but you can "Cancel" out, and the Interpreter will hang but set the breakpoint on break.  Does that match what you're seeing?  Those seem sane to me, with the possible exception of bringing up the dialog at all for the Dynamic listing.
On the other question, not sure I know what copy-on-write means in this context.  Refresh depends on the underlying API, which varies from object to object, i.e. it may provide a list, as for the registers in the dbgeng,  or it may force you to iterate, as for the modules.  It always will provide a full set - not a difference.  I can't think of any debugger that provides "new only" values, except through events.  Onec we have the set, we definitely iterate through them to compare the old and new lists.  This is necessary for the trace where we want to know creation and destruction times for the objects.
Sorry to be a little dense - coffee hasn't kicked in yet.

I was trying to set from the static listing. The bookmark gets added but is disabled so the breakpoint wasn't actually set and the popup for setting the breakpoint stays. I usually will just click ok a fewtimes and it will take it.
The copy on write is the name of the class. I don't have the documentation in front of me but you will see what I mean if you dig up the implementation details of GTree.

Gotcha - will put in a ticket for the weird non-disappearing breakpoint dialog.
Re CopyOnWrite, uhh, at a loss still - CopyOnWriteSet, CopyOnWriteArraySet, CopyOnWriteArrayList?

Followed up on the ObjectsTree with folks, here - understand what you were getting at now, but not the issue in this case.

Followed up on the ObjectsTree with folks, here - understand what you were getting at now, but not the issue in this case.

Ok cool. Was just a thought and I wasn't able to check myself at the time.

Yep, was a good thought and definitely worth checking - also will stop me from making that mistake in the future.


Add monitoring for Prometheus servers (that prometheus-operator creates)

Towards https://github.com/giantswarm/roadmap/issues/601 Ensure monitoring for all managed apps
TODO

[ ] ensure it is scraped by Prometheus
[ ] make sure there are basic alerts

This was done in the scope of https://github.com/giantswarm/giantswarm/issues/19310