[dask] [python-package] Search for available ports when setting up network (fixes #3753)

See #3753 for additional details.
The Problem
By default, LightGBM's Python package is built to use TCP sockets for distributed learning (-DUSE_SOCKET=1). When performing distributed learning, LightGBM requires a list of machines and ports, as is documented here.
machine1_ip 12345
machine2_ip 12345

lightgbm.dask currently creates this list by assigning a fixed range of ports. Assuming that you pass parameters local_listen_port=12400, for example, the first worker will open a socket {first_worker_ip}: 12400, the second will open one {second_worker_ip}:12401, and so on.
This approach is fragile, since it makes the assumption that the ports you request will definitely be available. It can lead to an error like this:

Binding to port 12402 failed

This can happen either because of connections not quite cleaned up after previous LightGBM training runs (https://github.com/microsoft/LightGBM/issues/3753#issuecomment-759683870) or because of any other processes running on that machine which might already be using a port that LightGBM wants.
Changes in this PR

lightgbm.dask will now search for open ports on each worker when constructing the network. It uses local_listen_port as a starting point, but doesn't assume that that port or any other specific port is available.

this is now done one time in _train(), before setting up the network. On master today, the code to set up the parameters is re-run in _train_part() (once on each worker) even though it produces the same results

refactors the internals for constructing network parameters
adds tests that you can call .fit() with a Dask model many times in a row with no sleeping in between...this is the primary way that the bug in #3753 shows up
adds type hints to only new code added in this PR (per https://github.com/microsoft/LightGBM/issues/3756)

Notes for reviewers
I tested this with both a LocalCluster with 3 workers and a dask_cloudprovider.aws.FargateCluster, using the steps described in https://github.com/microsoft/LightGBM/issues/3753#issue-784647300 and the notebooks in https://github.com/jameslamb/talks/tree/main/recent-developments-in-lightgbm. ~I think I'll set up a separate small repo with my testing setup so that this can be more easily re-used by contributors in the future.~ This can be tested with the setup in https://github.com/jameslamb/lightgbm-dask-testing.
I captured the screenshot below to show some more evidence that this worked. I had the code print out its decision for ports on each worker, and I ran training 10 times in a row in a for loop like this:
for i in range(10):
    client.restart()
    ...
    dask_reg.fit(...)

You can see that on some runs, it chose 12400, 12401, and 12402. But on others, LightGBM chose to skip up a few ports. That's the "search until you find one" stuff working! The picture on the right side of the screenshot shows the individual training runs succeeding one after another.

Since this worked well on both LocalCluster (where all workers have the same IP address) and FargateCluster (where each worker is a different physical machine, with a unique IP address), I feel confident in this fix.
special thanks to @imatiach-msft and @ffineis for their help figuring this out!

hmmm, never saw this error before. Spotted on the macOS-latest, TASK=REGULAR build. Will try rebuilding, maybe it's a temporary thing.
[NbConvertApp] Converting notebook interactive_plot_example.ipynb to notebook
Traceback (most recent call last):
  File "/Users/runner/miniconda/envs/test-env/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/__init__.py", line 143, in read
    return reads(buf, as_version, **kwargs)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/__init__.py", line 77, in reads
    validate(nb)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 270, in validate
    relax_add_props=relax_add_props):
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 303, in iter_validate
    validator = get_validator(version, version_minor, relax_add_props=relax_add_props)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 57, in get_validator
    schema_json = _get_schema_json(v, version=version, version_minor=version_minor)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 94, in _get_schema_json
    with open(schema_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/v4/nbformat.v4.2.schema.json'

hmmm, never saw this error before. Spotted on the macOS-latest, TASK=REGULAR build. Will try rebuilding, maybe it's a temporary thing.
[NbConvertApp] Converting notebook interactive_plot_example.ipynb to notebook
Traceback (most recent call last):
  File "/Users/runner/miniconda/envs/test-env/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/__init__.py", line 143, in read
    return reads(buf, as_version, **kwargs)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/__init__.py", line 77, in reads
    validate(nb)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 270, in validate
    relax_add_props=relax_add_props):
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 303, in iter_validate
    validator = get_validator(version, version_minor, relax_add_props=relax_add_props)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 57, in get_validator
    schema_json = _get_schema_json(v, version=version, version_minor=version_minor)
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/validator.py", line 94, in _get_schema_json
    with open(schema_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/nbformat/v4/nbformat.v4.2.schema.json'

@jameslamb Just to clarify: I have literally zero experience with Dask. Sorry.

@StrikerRUS it's ok, I understand! I'm not sure if any other maintainers have Dask experience. If there's ever a Dask PR that you feel is large or controversial, we can probably get some reviewing help from our friends at XGBoost or I can ask others in the Dask community.

@jameslamb Just to clarify: I have literally zero experience with Dask. Sorry.

@StrikerRUS it's ok, I understand! I'm not sure if any other maintainers have Dask experience. If there's ever a Dask PR that you feel is large or controversial, we can probably get some reviewing help from our friends at XGBoost or I can ask others in the Dask community.

@jameslamb

we can probably get some reviewing help from our friends at XGBoost or I can ask others in the Dask community.

Asking for a help from outside reviewers or maybe even find someone interested and experienced in LightGBM and Dask to collaborate on improvements in Dask-package on a regular basis supporting you will be just awesome.

@jameslamb

we can probably get some reviewing help from our friends at XGBoost or I can ask others in the Dask community.

Asking for a help from outside reviewers or maybe even find someone interested and experienced in LightGBM and Dask to collaborate on improvements in Dask-package on a regular basis supporting you will be just awesome.

I've moved my test setup out into its own repo: https://github.com/jameslamb/lightgbm-dask-testing. I'm going to test this on LocalCluster and FargateCluster one more time, to be sure I didn't make any mistakes responding to review comments. If all goes well, I'll merge after that and go review #3708

I've moved my test setup out into its own repo: https://github.com/jameslamb/lightgbm-dask-testing. I'm going to test this on LocalCluster and FargateCluster one more time, to be sure I didn't make any mistakes responding to review comments. If all goes well, I'll merge after that and go review #3708

happy to say this worked on LocalCluster and FargateCluster. Thanks for the help and reviews!

happy to say this worked on LocalCluster and FargateCluster. Thanks for the help and reviews!


Inject: Encode labels as json similar to annotations (#9634)

Use json to encode labels

Fix curl image, cherry-pick #9637

(cherry picked from commit 52da3bce56c7ffda2bfcb64dd963881c2a9a01e1)

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers: andraxylia, geeknoid
If they are not already assigned, you can assign the PR to them by writing /assign @andraxylia @geeknoid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS
pilot/OWNERS
pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ayj: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cc @mandarjog - I cherry-picked your fix from the release-1.0 branch. I looks like master also needs this fix.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-local-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-unit-tests

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-local-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-local-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-local-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-unit-tests

prow/istio-presubmit.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: ayj
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers: andraxylia, geeknoid
If they are not already assigned, you can assign the PR to them by writing /assign @andraxylia @geeknoid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS
pilot/OWNERS [ayj]
pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-k8s-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-integ-k8s-tests

prow/istio-unit-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-unit-tests

prow/istio-presubmit.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-presubmit

prow/istio-integ-local-tests.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth

prow/e2e-dashboard.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-dashboard

prow/e2e-bookInfoTests-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-pilot-e2e-envoyv2-v1alpha3-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3-mcp

prow/e2e-mixer-no_auth-mcp.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-mixer-no_auth-mcp

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-e2e-envoyv2-v1alpha3

prow/e2e-simpleTests.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test e2e-simpleTests

prow/istio-pilot-multicluster-e2e.sh
866894dac042c33034844ee49bb2925902679d0b
link
/test istio-pilot-multicluster-e2e

prow/istio-unit-tests.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-unit-tests

prow/istio-presubmit.sh
c142d44ab79c2691074f3364d351afcad4cd53ec
link
/test istio-presubmit

prow/istio-integ-local-tests.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test istio-integ-k8s-tests

prow/e2e-mixer-no_auth-mcp.sh
43f7d117339f5641e7faf35822a4776e56e6c8c0
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-k8s-tests

prow/e2e-mixer-no_auth-mcp.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-k8s-tests

prow/e2e-mixer-no_auth-mcp.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-k8s-tests

prow/e2e-mixer-no_auth-mcp.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test e2e-mixer-no_auth-mcp

prow/e2e-mixer-no_auth.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test e2e-mixer-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e-mixer-no_auth

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test istio-integ-k8s-tests

prow/e2e-mixer-no_auth-mcp.sh
7099427a9b6b3c323640ad9beef32624b21e641e
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Press "Download and Install Now" did not initiate update install on Windows XP

Legacy Update 1.10 is used and it is on a fresh installed Windows XP SP3 (IE6 uesd)

Having this issue too, XP Pro SP3 x86 using IE6. Using Legacy Update 1.10
Clicking the button causes a page error to occur. Inspecting the page's code shows no obvious cause for the issue. Strangely it says KB898461 is downloaded and ready to install after I restarted to see if that would fix it.

Thanks, I made another silly typo. Fixed now. You may need to ctrl-F5 or try again in 15-ish minutes if it still doesn't work.


DoReFa quantizer with higher number of MACs/Ops, Grouped convs as custom ops on LCE 0.7.0

Hello, I have a couple of questions regarding quantizer options for Larq and LCE.
I am designing a BNN using the DoReFa quantizer, however, I noticed a very high number of estimated MACs and Ops when converting the model for ARM64. Changing the quantizer to "ste_sign" dramatically lowered the number of MACs and Ops.
I was wondering if there is a way to use the DoReFa quantizer for training without the serious overhead of operations when converting and running the model for inference in LCE? Is the "ste_sign" quantizer the only viable option for efficient inference?
Thank you for the excellent work and for your attention.

I noticed some issues with the latest version only (0.7.0) but not the one before (0.6.2).
Grouped convolutions (FP or binary) are converted as custom ops in the latest version.
Example:
Grouped (g=2)  convs converter output:

2022-07-26 13:06:17.469686: W external/org_tensorflow/tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1903] The following operation(s) need TFLite custom op implementation(s):
Custom ops: Conv2D
Details:
tf.Conv2D(tensor<1x32x32x64xf32>, tensor<5x5x32x32xf32>) -> (tensor<1x11x11x32xf32>) : {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 3, 3, 1], use_cudnn_on_gpu = true}
See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2022-07-26 13:06:17.469772: I external/org_tensorflow/tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1963] Estimated count of arithmetic ops: 5792  ops, equivalently 2896  MACs
Estimated count of arithmetic ops: 5792  ops, equivalently 2896  MACs

Quantizer small example (2 qconv layers):
Example with ste_sign mode="weights":

2022-07-26 13:14:57.680246: I external/org_tensorflow/tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1963] Estimated count of arithmetic ops: 1.164 M  ops, equivalently 0.582 M  MACs
Estimated count of arithmetic ops: 1.164 M  ops, equivalently 0.582 M  MACs

Changing to DoReFa mode="weights":

2022-07-26 13:16:05.771057: I external/org_tensorflow/tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1963] Estimated count of arithmetic ops: 1.663 M  ops, equivalently 0.831 M  MACs
Estimated count of arithmetic ops: 1.663 M  ops, equivalently 0.831 M  MACs

I was able to successfully benchmark my model with DoReFa and grouped convolutions converted on version 0.6.2 with a better-than-expected efficiency but not the one converted with version 0.7.0
I am using Tensorflow 2.8.0 and larq 0.12.2

Sorry for the late reply.

I noticed some issues with the latest version only (0.7.0) but not the one before (0.6.2).
Grouped convolutions (FP or binary) are converted as custom ops in the latest version.

Unfortunately this was an issue with TensorFlow 2.8 which LCE 0.7.0 uses under the hood. This has been fixed on master since we upgraded to 2.9, but we haven't published a new release with it yet. Sorry about that. For now, I'd recommend sticking with 0.6.2 if grouped convolution support is required.

Is the "ste_sign" quantizer the only viable option for efficient inference?

For binarised convolutions this is recommended for the activation. You can also use custom activation quantisers as well, but to make sure they convert correctly they should be implemented with larq.math.sign which unfortunately is not the case for DoReFa. Regarding weight quantization other quantisers should work fine as long as they binarise to {-1, 1} or {-alpha, alpha}.
I recommend looking at the converted model in Netron to make sure the conversion worked as intended.

I noticed some issues with the latest version only (0.7.0) but not the one before (0.6.2).
Grouped convolutions (FP or binary) are converted as custom ops in the latest version.

@lluevano sorry for the delay. We just release v0.8.0 including a fix for this. Let me know if that works for you.


Customise task status for "Archive all checked tasks"

At the moment, this is either for just [x] or any symbol between "[]. I am using the new status function in Tasks to set a different character for a completed task [&}. If you toggle for all symbols, then it also archives all other statuses, such as [i].
It would be good to be able to customise the symbols used.

Hello, @kmaustral,
Thank you for your input! Check out 0.16.1. Now you can create rules for different statuses:

Thanks so much. It's all working now.


Read an account from the service

As a [customer]
I need [a service that reads my account]
So that [i can see my account details]
Details and Assumptions
* [service will read an account and display details of that account]

Acceptance Criteria
gherkin
Given [a customer needs to see their account details]
When [they call the read api]
Then [they will be able to see their account details]


Add quick fix for when flake8 triggers the "tab before keyword" warning

Ref: https://github.com/microsoft/vscode-flake8/issues/14
There are two things that needs to be done:

[ ] Create quick fix for warning code E274 that triggers the editor.action.formatDocument action
[ ] Add unit test

Code would be similar to this: https://github.com/microsoft/vscode-flake8/blob/c1a69ff81a0c98ea7012d3d00aa0909c8df81a98/bundled/tool/server.py#L239

Hi! I'd love to give this a shot!


error when running docker build on mac os (Apple silicon)

`Dockerfile:10
9 |
10 | >>> RUN wget
11 | >>>     https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
12 | >>>     && mkdir /root/.conda
13 | >>>     && bash Miniconda3-latest-Linux-x86_64.sh -b
14 | >>>     && rm -f Miniconda3-latest-Linux-x86_64.sh

15

ERROR: failed to solve: process "/bin/sh -c wget     https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh     && mkdir /root/.conda     && bash Miniconda3-latest-Linux-x86_64.sh -b     && rm -f Miniconda3-latest-Linux-x86_64.sh" did not complete successfully: exit code: 133`

@apeshimam That's a bit hard to debug without more output, but it seems the miniconda installation didn't work. If you start from a Miniconda image instead of Ubuntu this could work better. I've pushed a change to the Dockerfile. Please let me know what happens when you build after downloading/pulling this change.


Document mongo sink task conf property name is not correct

http://docs.datamountaineer.com/en/latest/mongo-sink.html
connect.mongo.database -> connect.mongo.db

docs fixed


analysis_notebooks

This PR has all the notebooks used to create the figures/plots etc for the CanSig paper


Added tests for lib/verdeps/traverse_package_dir.go

Coverage increased (+4.6%) to 71.675% when pulling 6b2efd1369986cdc742906ecc84d0b8b0ee77ba7 on feature/verdeps-tests into ebb3e517c41361e66b458b67d4bbd85fc7e7258a on master.


Fix #3360: RuntimeWarning: 'numba.runtests' found in sys.modules

Fixes #3360 by moving most of numba.runtests into numba._runtests and import only the latter in numba.__init__.
If you think some other change makes more sense, please feel free to close this PR or push to this its branch as you see fit!
ref: @mbargull in https://github.com/conda-forge/numba-feedstock/pull/15#issuecomment-425769334:

everywhere:lib/python3.6/runpy.py:125: RuntimeWarning: 'numba.runtests' found in sys.modules after import of package 'numba', but prior to execution of 'numba.runtests'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))

This is more or less just a nuisance since numba.runtests uses if __name__ == '__main__' for the non-declarative part. If you wanted to get rid of those warnings, you could put the declarations into a separate module, e.g., numba._runtests, and import them from that module in numba.runtests and numba.__init__ so to remove the numba.runtests import in numba.__init__.

Codecov Report

Merging #3362 into master will decrease coverage by 0.06%.
The diff coverage is 19.11%.

@@            Coverage Diff             @@
##           master    #3362      +/-   ##
==========================================
- Coverage   80.71%   80.64%   -0.07%
==========================================
  Files         389      390       +1
  Lines       77965    78035      +70
  Branches     8797     8807      +10
==========================================
+ Hits        62927    62935       +8
- Misses      13658    13719      +61
- Partials     1380     1381       +1


[BUG]: Doesn't work with Vlad fork because localization object is not defined.

Is there an existing issue for this?

[X] I have searched the existing issues

Bug Description
Refuses to load in vlad fork, gives error below in firefox.
Uncaught (in promise) ReferenceError: localization is not defined
Dn http://<IP_ADDRESS>:7860/file=extensions/sd-webui-better-styles/javascript/betterStyles.js?1684803623.031734:6
Corrected thanks to chatgpt.
added const localization = {}; to the start of your BetterStyles.js
Looking forward to trying it, hopefully that was enough to get it going.
Steps to Reproduce
Install, try to use. Doesn't show up. Error in dev console.
Screenshot (optional)
No response

@Aptronymist ChatGPT suggested fix did the trick for me.
const localization = {}; to the start of your betterStyles.js gets the little icon to show up.

In my case I now have it right after the first var, so like this:
var BetterStyles=function(){"use strict";const localization = {};
and make sure to have the "hide original styles" checked, as well as having 1.2.0 selected. I don't know for sure what fixes what beyond the localization addition, but those are my current settings and it works fine.

Note: Egg on face time;
Placing my extensions behind a symlink caused a lot of issues. Those issues vanished when I removed the symlink and copied the extension back to the main folder.
Closeable.


Should I make a collective for my org? Or for my repo?

OpenCollective.com presents two options:

[ ] Create a collective for the repository
[ ] Create a collective for the organization

I couldn't find any recommendations on which to pick. What's the advantage of one over the other? When should we pick one or the other?
Can we switch options, or enable both, later?

Hi @trusktr, this choice doesn't make a huge difference. If your project is something that mainly rest in a single repo (ie. a code library) then "repository" should be the right choice. If it's a set of multiple repos (ie. a website with a frontend + an API) then "organization" should be picked. The one you choose will affect:

What the Github icon on your profile will link to
What will be used for the stars verification: if you pick "repository", it needs to have 100+ stars. If you pick "organization", one of the repositories under this org needs to have 100+ stars.


WDSL codeunit parameter names in lowercase

Sorry but my english is not good.
Recently we updated one Navision 2018 from CU 10 to 18 and now the parameters of the codeunits published as web services start with lowercase (with the CU 10 they started with capital letters). This generates us a problem since an integration that we have has stopped working. This codeunit is developed in AL extension.
Has anyone the same problem?
Thks

Hi! This change is by design. The AL compiler has changed how parameters are named in the emitted metadata. Changing this at this point in time would break existing consumers.


I can't move blocks

Description of the issue
I can't get past the blocks and I can't find a way to activate the no-clip
Expected behavior
No response
Steps to reproduce

Open a singleplayer world
Enable Freecam by pressing F4
...

Log output
No response
Crash report
No response
How prevalent is the issue?
Every time I try to occupy it
Freecam version
1.2.1
Are you using "Modrinth Edition"?
Yes
Minecraft version
1.20.1
Modding platform

[X] Fabric
[ ] Neoforge
[ ] Forge

Sorry if it's written wrong, I speak Spanish and I had to use the translator (I didn't see anyone use Spanish)

Thanks for reporting your issues!
This is a Modrinth Edition restriction.
Due to Modrinth's anti-chrating policies, Freecam's collision settings are restricted to singleplayer and servers where you are op.
You can download Freecam from GitHub or CurseForge instead.


Is there an option to disable mouse wheel for Player?

Scrolling the browser window may trigger moving to next/previous moves. Is there an option to disable move navigation triggered by mouseWheel?

Sure, see the scroll_wheel_navigation setting documented here: https://github.com/adambuczynski/ngGo/wiki/Player-configuration


Python files are marked as saved when the netlist is saved

Describe the bug
When saving a netlist as .hal file, all open python editors with pending changes are marked saved. This is only partially correct: When the HAL file is saved, the unsaved Python files are included, so no work is lost. However, the actual Python files are not saved individually.
To Reproduce
Steps to reproduce the behavior:

Open any netlist
Open a .py file in the python editor and type something, this will cause an asterisk to appear in the tab
Hit [Ctrl-S]
See that the asterisk is gone

Expected behavior
Users will expect the asterisk to disappear iff the actual script file has been saved.
Additional context
We should clearly document what gets saved where and when.

I think we need to discuss when an asterisk has to be added to the tab or define what the asterisk actually means. Usally it means there are unsaved changes. Save the .hal and it's technically saved. In my opinion the whole system how py script are saved and associated with the .hal file is the cause of confusion here.

I think python scripts should only be written to their individual source files. I do not know of any other editor that saves opened files twice, once in the actual file and once in some kind of project file (excluding temporary backups). Seems unnatural and overcomplicated to me.
The only benefit of the current system seems to be that the python scripts are still available if you move or delete (intentionally or accidentally) the source files, but...
a) How often does that happen? It seems kind of contrived.
b) Even if it does happen, it's not our problem. Thats the users problem. If you delete your files, your stuff is gone - thats how it works. If the user wants additional security he is free to use any backup or redundancy system he wants.
One could argue that there is the additional benefit of being able to share your entire project with other people, python scripts included, by sending them the project file, but once again thats a non problem. Zipping a folder takes like 3 seconds, we dont need a custom solution for that.

I have previously suggested splitting this saving mechanism between never saved files and modified files. The first ones go into the HAL file. The second only go to their script file, but HAL remembers their path in the HAL file. Thus, unsaved files are not lost and external modifications of scripts while HAL is closed don't collide with what is saved in the HAL file. If the user deletes the script, it's gone.


Update the emoji packs

Seems like there's been quite a few additions to the emojis. I would like to see them updated and natively working in chatterino :)

Duplicate of #869


http3 does not work with other than default TLS settings

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

configured traefik with h3 and opened in chrome 101:

[x] works fine if using default TLS settings (i.e. no settings applied)

configured http3 for a docker backend

added named tls settings using file provider (tls-settings-prod.yml)
applied settings using e.g. docker label traefik.http.routers.traefik-sec.tls.options: prod@file

Note: UDP receive buffers have been incresed using sysctl -w net.core.rmem_max=2500000
What did you see instead?
As soon as any TLS setting other than default is applied, h3 does not seem to work anymore. It actually does not seem to matter what options are contained in the file (tried minVersion, preferServerCipherSuites, sniStrict, cipherSuites).
Is there a way to make h3 work with adjusted TLS settings or is this a limitation of the current (experimental) implementation?
What version of Traefik are you using?
Version:      2.7.0
Codename:     epoisses
Go version:   go1.17.10
Built:        2022-05-24T17:07:05Z
OS/Arch:      linux/amd64
What is your environment & configuration?
tls-settings-prod.yml
tls:
  options:
    prod:
      minVersion: VersionTLS13

docker labels on traefik dashboard:
traefik.http.routers.traefik-sec.tls.options: prod@file
traefik.http.routers.traefik-sec.tls.certresolver: prod

If applicable, please paste the log output in DEBUG level
No response

I experienced the same behavior.
I tried to enable http3 for my traefik v2.9.5 but it only worked kind of: http3check.net said that http3 would be supported (it also says that QUIC is not supported but that seems to be a known bug at their side). And also when I disable the 443 port forwarding for udp at my router (fritzbox), http3check sais (correctly) that http3 is not supported. But in any case, Chrome never used the http3 protocol (according to its developer tools).
Only when I (found this issue and) removed the custom tls settings for traefik, http3 is also used by Chrome.

Hi all,
I was able to reproduce, thanks for the report.


Possibility of undefined from optional properties is not honoured in spread

TypeScript Version:  3.6.3

Search Terms: spread optional undefined object properties property
Code
Here's an example of spreading an object with optional properties:
const myQueryDefaults: { a: number; } = { a: 1 };
const myQuery: { a?: number; } = {};

const result = { ...myQueryDefaults, ...myQuery };

// type is `number`
// runtime value is `number`
result.a;

The above example makes sense. However, an optional property does not only mean "this property may not exist" ({})—it also means "this property may exist with a value of undefined" ({ a: undefined }). (Related: https://github.com/Microsoft/TypeScript/issues/13195.)
As a result of this, the type information may mismatch the actual types at runtime:
const myQueryDefaults: { a: number; } = { a: 1 };
const myQuery: { a?: number;  } = { a: undefined };

const result = { ...myQueryDefaults, ...myQuery };

// type is `number`, but runtime value is `undefined`!!
// expected type: `number | undefined`
result.a;

… which could result in runtime exceptions:
result.a.toString(); // runtime exception, because `result.a` is `undefined`

Ideally optional properties would only mean "this property may not exist" (https://github.com/Microsoft/TypeScript/issues/13195), in which case the existing behaviour with spreading makes sense. However, in the interim, perhaps the behaviour of spreading should be changed?
Extracted from https://github.com/Microsoft/TypeScript/issues/13195#issuecomment-373178677

However, in the interim, perhaps the behaviour of spreading should be changed?

That would most likely break every codebase out there.

#13195 is the actionable version of this. This would be an unbelievably large and unergonomic breaking change.


"Goiești"  is listed in the drop-down menu

Description:
"Goiești" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Goiești" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Goiești, DOLJ".


ImportError: cannot import name 'template_generic' from 'pyqtgraph.console'

Traceback (most recent call last):
File "/home/user/.local/bin/nodedge", line 5, in
from nodedge.main import main
File "/home/user/.local/lib/python3.10/site-packages/nodedge/main.py", line 14, in
from nodedge.nodedge_app_window import NodedgeAppWindow
File "/home/user/.local/lib/python3.10/site-packages/nodedge/nodedge_app_window.py", line 11, in
from nodedge.mdi_window import MdiWindow
File "/home/user/.local/lib/python3.10/site-packages/nodedge/mdi_window.py", line 21, in
from nodedge.console_widget import ConsoleWidget
File "/home/user/.local/lib/python3.10/site-packages/nodedge/console_widget.py", line 8, in
from pyqtgraph.console import template_generic as ui_template
ImportError: cannot import name 'template_generic' from 'pyqtgraph.console' (/home/user/.local/lib/python3.10/site-packages/pyqtgraph/console/init.py)

I got the same error on Windows with Python 3.8
python -m pip install nodedge
Collecting nodedge
  Downloading nodedge-0.5.0-py3-none-any.whl (224 kB)
     ---------------------------------------- 224.1/224.1 kB 2.7 MB/s eta 0:00:00

Successfully installed PySide6-Essentials-6.3.1 asammdf-7.3.14 canmatrix-1.0 coloredlogs-15.0.1 humanfriendly-10.0 isal-1.2.0 lz4-4.3.2 nodedge-0.5.0 pyqtconsole-1.2.2 pyqtgraph-0.13.3 pyreadline3-3.4.1 shiboken6-6.3.1 types-PyYAML-<IP_ADDRESS>

nodedge
===== 2023.08.16 14:08:06 =====
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python38\Scripts\nodedge.exe\__main__.py", line 4, in <module>
  File "C:\Program Files\Python38\lib\site-packages\nodedge\__main__.py", line 14, in <module>
    from nodedge.nodedge_app_window import NodedgeAppWindow
  File "C:\Program Files\Python38\lib\site-packages\nodedge\nodedge_app_window.py", line 11, in <module>
    from nodedge.mdi_window import MdiWindow
  File "C:\Program Files\Python38\lib\site-packages\nodedge\mdi_window.py", line 21, in <module>
    from nodedge.console_widget import ConsoleWidget
  File "C:\Program Files\Python38\lib\site-packages\nodedge\console_widget.py", line 8, in <module>
    from pyqtgraph.console import template_generic as ui_template
ImportError: cannot import name 'template_generic' from 'pyqtgraph.console' (C:\Program Files\Python38\lib\site-packages\pyqtgraph\console\__init__.py)

The installed wheel from pypi downloads the latest dependencies for nodedge. It means that pyqtgraph is installed with version 0.13.3, whereas frozen version uses 0.13.1.
I guess one should stick with the frozen version. We can do with a pip install -r requirements.txt before installing nodedge

Hello Guillaume,
Thank you for your message.
I am updating the project right now: #204
The console is now integrated into pyqtgraph, so we don't need our own console widget in Nodedge.


Permission Denied Error on Windows

When using the qa example code from the cookbook repo, uploading a file results in a Permission Denied error.
Also there is a file size upload limit of 2MB. Is there a way to disable this feature?
Full traceback:
$ chainlit run app.py -w
2023-05-26 16:42:56 - Loaded .env file
2023-05-26 16:43:06 - Your app is available at http://localhost:8000
    yield from self.parser.parse(blob)
  File "c:\users\ifeanyi pc\desktop\chat-with-github-repo\env\lib\site-packages\langchain\document_loaders\base.py", line 87, in parse
    return list(self.lazy_parse(blob))
  File "c:\users\ifeanyi pc\desktop\chat-with-github-repo\env\lib\site-packages\langchain\document_loaders\parsers\pdf.py", line 16, in lazy_parse
    with blob.as_bytes_io() as pdf_file_obj:
  File "C:\Users\IFEANYI PC\.pyenv\pyenv-win\versions\3.8.10\lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "c:\users\ifeanyi pc\desktop\chat-with-github-repo\env\lib\site-packages\langchain\document_loaders\blob_loaders\schema.py", line 86, in as_bytes_io
    with open(str(self.path), "rb") as f:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\IFEANY~1\\AppData\\Local\\Temp\\tmp1_q_q2sk'

For the max file size you can use the max_size_mb parameter in ask_for_file, see https://docs.chainlit.io/api-reference/ask/ask-for-file
For the permission error that looks like a langchain error at first glance. Not sure that is the issue but could you try running the chainlit run command in a terminal with admin permissions?

Also can you try to run chainlit run without the -w option?

I tried both options(running a terminal with admin permissions and without the -w flag) and still got the same error. Tested this on a Linux cloud environment though and it works as expected.

Could you try to:

install chainlit from the wheel provided in this PR https://github.com/Chainlit/chainlit/pull/40
run chainlit hello

I would like to see if the rework fix those issues. Note that this version is not merged yet. The infos to install are at the end of the PR's description.

Should be fixed in the latest version 0.3.0. Please note that it contains breaking changes. We prepared a migration guide to make it easy for everyone.

I am facing the same issue, My code is pretty much the same as the example given at https://docs.chainlit.io/api-reference/ask/ask-for-file I tried running as admin, and also without the -w:
async def start():
    await cl.Message(content="Upload a file to chat with your file").send()
    files=None
    while files is None:
        files=await cl.AskFileMessage(
            content=welcome_message,
            accept=['text/plain','application/pdf'],
            max_size_mb=20,
            timeout=180
        ).send()

    file=files[0]
    # file=file.content.decode("utf-8")
    msg=cl.Message(content=f"Processing {file.name}...")
    await msg.send()

    docsearch = await cl.make_async(get_docsearch)(file)

    chain = RetrievalQAWithSourcesChain.from_chain_type(
        ChatOpenAI(temperature=0.8,
                   streaming=True,
                   chain_type="stuff",
                   retriever=docsearch.as_retriever(max_tokens_limit=4097)
                   )
        )

gives:
07:08:22 pm
Processing 2308.07037.pdf...
Error
07:08:22 pm
[Errno 13] Permission denied: 'C:\Users\tanma\AppData\Local\Temp\tmpkgtc4us6'

I think the problem is about python temp files on windows. You can avoid using temp files. Also would be helpful to search if there is a clean solution for this

Yes, it is the problem of Python writing to Temp directory.  I had the code like below:
with tempfile.NamedTemporaryFile() as tempfile:
tempfile.write(file.content)
loader = Loader(tempfile.name)
It was giving Permission Denied error. I replaced with as below and it worked,
with open('filename.ext', 'wb') as file_output:
file_output.write(file.content)
loader = Loader('filename.ext')


Create SQL Alchemy connection string using Okta IdP

Driver version
redshift_connector 2.0.889
Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.31651
Client Operating System
mac OS Monterery
Python version
3.9.7
Problem description
import redshift_connector

# An Okta auth page will pop up when initiating this connector
conn = redshift_connector.connect(
    host='<cluster_identifier>.ap-southeast-2.redshift.amazonaws.com',
    port=5439,
    region='ap-southeast-2',
    database='<db_name>',
    credentials_provider='BrowserSamlCredentialsProvider',
    login_url='<saml_url>',
    cluster_identifier='<cluster_identifer',
    preferred_role='arn:aws:iam::<iam-id>:role/<role-name>',
    user='',        # left empty string
    password='',    # left empty string
    iam=True,
    ssl=True
 )

cursor: redshift_connector.Cursor = conn.cursor()

Currently, I can connect to Redshift using the above credentials but when trying to utilise the SQL Alchemy connection string - it causes an identification error (using sqlalchemy-redshift). Is there a way to generate an appropriate URL for sqlalchemy to parse for IdP authentication?
I've noticed that Redshift was recently updated to work with sqlalchemy-redshift but have had no luck generating an appropriate URL login.

Hey @nam-tonthat-afterpay ,
Below is an example showing how to establish a connection to Redshift using Okta Browser authentication. Please note that the user and password field are no longer required (i.e. as you're using the browser authentication plugin they can be excluded).
import sqlalchemy as sa
from sqlalchemy.engine.url import URL

url = URL.create(
drivername='redshift+redshift_connector', # indicate redshift_connector driver and dialect will be used
)

conn_params = {
"ssl": True,
"iam": True, # must be enabled when authenticating via IAM
"credentials_provider": "BrowserSamlCredentialsProvider",
"database": '<db_name>', # Amazon Redshift database
"region": "ap-southeast-2",
"cluster_identifier": "<cluster_identifer>",
"login_url": "<saml_url>"
}

engine = sa.create_engine(url, connect_args=conn_params)

with engine.connect() as connection:
    result = connection.execute("select 1")
    for row in result:
        print(row)

engine.dispose()

Please let me know if you see any issue or have any questions.

Hey @Brooke-white,
I appreciate that! That works running it by itself but once integrated into Amundsen where the botocore/client.py takes the api_call, it returns this error.
File "/Users/nam.tonthat/miniconda/envs/amundsen/lib/python3.9/site-packages/redshift_connector/iam_helper.py", line 371, in set_cluster_credentials
    response = client.describe_clusters(ClusterIdentifier=info.cluster_identifier)
  File "/Users/nam.tonthat/miniconda/envs/amundsen/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/nam.tonthat/miniconda/envs/amundsen/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ClusterNotFoundFault: An error occurred (ClusterNotFound) when calling the DescribeClusters operation:

More context, I've used the following versions:

amundsen-common = 0.21.0
amundsen-databuilder  = 6.3.1
amundsen-rds  = 0.0.6
sqlalchemy = 1.4.22
sqlalchemy-redshift = 0.8.8
botocore = 1.23.2

Hi @nam-tonthat-afterpay ,
Thanks for providing this additional information.  I'd recommend to try the following:

Enable debug logging for redshift_connector and verify nothing looks suspicious in the logs (e.g. region, cluster_identifier)
Are the cluster_identifier and region used for establishing the connection inline with what is seen in the debug logs? As a sanity check, does the cluster exist in the given region?

The error provided is being thrown after the authentication process w/ Okta has completed (i.e. we have temporary IAM credentials), when redshift_connector tries to retrieve the host and port of the cluster with the given cluster_identifier. It's a good sign we're able to check if the cluster if cluster_identifier exists or not.
In my experience, this exception occurs when authentication has been successful but there is some mis-match between cluster_identifier and region.
If you still see issue after trying the above, please let me know and I will work to reproduce on my end.

Hey @Brooke-white, do you have any tutorials on how to enable debug logging? I'm still newish to logging.

Hey @Brooke-white, do you have any tutorials on how to enable debug logging for redshift_connector? I'm still newish to logging.
I've tried using the Python tutorial but haven't had much luck.

Hey @nam-tonthat-afterpay ,
Apologies for the late response. Here's an example of configuring debug logging to output to stdout.
This would likely be easier that outputting the logs to a file given your Amundsen instance is probably running in a docker container, and a log file would show up there rather than your local file system :)
import redshift_connector
import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)  # will write all logs with a level >= DEBUG
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)

with redshift_connector.connect(
...
) as conn:
    with conn.cursor() as cursor:
        cursor.execute("select 1")
        print(cursor.fetchall())

For me, this outputs:
2021-11-19 09:31:02,869 - redshift_connector - DEBUG - ===================================
2021-11-19 09:31:02,869 - redshift_connector - DEBUG - User provided connection arguments
2021-11-19 09:31:02,869 - redshift_connector - DEBUG - ===================================
2021-11-19 09:31:02,869 - redshift_connector - DEBUG - {{'access_key_id': None, 'allow_db_user_override': False, 'app_id': None, 'app_name': 'amazon_aws_redshift',....}

ref: https://stackoverflow.com/a/14058475

Hey Brooke, thanks for the heads up; I've done some more testing with this and the cluster_identifer and region look correct but then when the query starts; it returns the ClusterNotFound error.
2021-11-26 15:16:40,716 - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeClusters) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.18.21 Python/3.9.7 Darwin/21.1.0 Botocore/1.21.41'}, 'body': {'Action': 'DescribeClusters', 'Version': '2012-12-01', 'ClusterIdentifier': 'dwh'}, 'url': 'https://redshift.ap-southeast-2.amazonaws.com/', 'context': {'client_region': 'ap-southeast-2', 'client_config': <botocore.config.Config object at 0x7fa1685b5040>, 'has_streaming_input': False, 'auth_type': None}}
2021-11-26 15:16:40,716 - botocore.hooks - DEBUG - Event request-created.redshift.DescribeClusters: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa1685aff70>>
2021-11-26 15:16:40,716 - botocore.hooks - DEBUG - Event choose-signer.redshift.DescribeClusters: calling handler <function set_operation_specific_signer at 0x7fa168342b80>
2021-11-26 15:16:40,717 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-11-26 15:16:40,717 - botocore.auth - DEBUG - CanonicalRequest:
POST
/
2021-11-26 15:16:41,091 - urllib3.connectionpool - DEBUG - https://redshift.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 404 287
2021-11-26 15:16:41,092 - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '934b5778-7693-4be6-ae70-7efdbdcf975b', 'Content-Type': 'text/xml', 'Content-Length': '287', 'Date': 'Fri, 26 Nov 2021 04:16:40 GMT'}
2021-11-26 15:16:41,092 - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="http://redshift.amazonaws.com/doc/2012-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    ClusterNotFound\n    <Message>Cluster dwh not found.</Message>\n  </Error>\n  <RequestId>934b5778-7693-4be6-ae70-7efdbdcf975b</RequestId>\n</ErrorResponse>\n'
2021-11-26 15:16:41,094 - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '934b5778-7693-4be6-ae70-7efdbdcf975b', 'Content-Type': 'text/xml', 'Content-Length': '287', 'Date': 'Fri, 26 Nov 2021 04:16:40 GMT'}
2021-11-26 15:16:41,095 - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="http://redshift.amazonaws.com/doc/2012-12-01/">\n  <Error>\n    <Type>Sender</Type>\n    ClusterNotFound\n    <Message>Cluster dwh not found.</Message>\n  </Error>\n  <RequestId>934b5778-7693-4be6-ae70-7efdbdcf975b</RequestId>\n</ErrorResponse>\n'
2021-11-26 15:16:41,095 - botocore.hooks - DEBUG - Event needs-retry.redshift.DescribeClusters: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa1685b5970>
2021-11-26 15:16:41,095 - botocore.retryhandler - DEBUG - No retry needed.
2021-11-26 15:16:41,096 - redshift_connector.iam_helper - ERROR - ClientError: An error occurred (ClusterNotFound) when calling the DescribeClusters operation: Cluster dwh not found.

Any ideas for this error?

Hey @nam-tonthat-afterpay ,
Could you verify that IAM SAML federation role was created inline with the instructions here:

To set up the service provider, complete steps 5–7 from Federate Amazon Redshift access with Okta as an identity provider.

For step 7, the Amazon Redshift connector needs the following additional permission:

{
            "Sid": "DescribeClusters",
            "Effect": "Allow",
            "Action": "redshift:DescribeClusters",
            "Resource": "*"
}

If the role does not provide permissions to DescribeClusters, this could prevent us from being able to see the dwh cluster.

Hi @Brooke-white ,
I have checked and found the permission to allow the access to Action: redshift:DescribeClusters. I've done some debugging - on the first run of a sample script like this
import redshift_connector
import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)  # will write all logs with a level >= DEBUG
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)

import sqlalchemy as sa
from sqlalchemy.engine.url import URL

url = URL.create(
drivername='redshift+redshift_connector', # indicate redshift_connector driver and dialect will be used
)

conn_params = {
"ssl": True,
"iam": True, # must be enabled when authenticating via IAM
"credentials_provider": "BrowserSamlCredentialsProvider",
"database": 'vega', # Amazon Redshift database
"region": "ap-southeast-2",
"cluster_identifier": "dwh",
"login_url": "<okta_login>"
}

engine = sa.create_engine(url, connect_args=conn_params)

with engine.connect() as connection:
    result = connection.execute("select 1")
    for row in result:
        print(row)

engine.dispose()

It throws the same error on the first run but every subsequent run always returns (1,) as the output.
However, repeating it for the amundsen query (adjusting the sample_postgres_loader.py to suit), will always cause a time out.
Any ideas on what could be the issue?

Hi @Brooke-white ,
I have checked and found the permission to allow the access to Action: redshift:DescribeClusters. I've done some debugging and made some progress but am thrown with another error.
On a simple query like this, it returns the ClusterNotFound error on the first run but on subsequent runs, it always returns (1,) as the output which is fantastic.
import redshift_connector
import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)  # will write all logs with a level >= DEBUG
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)

import sqlalchemy as sa
from sqlalchemy.engine.url import URL

url = URL.create(
drivername='redshift+redshift_connector', # indicate redshift_connector driver and dialect will be used
)

conn_params = {
"ssl": True,
"iam": True, # must be enabled when authenticating via IAM
"credentials_provider": "BrowserSamlCredentialsProvider",
"database": 'vega', # Amazon Redshift database
"region": "ap-southeast-2",
"cluster_identifier": "dwh",
"login_url": "<okta_login>"
}

engine = sa.create_engine(url, connect_args=conn_params)

with engine.connect() as connection:
    result = connection.execute("select 1")
    for row in result:
        print(row)

engine.dispose()

However, repeating it for the amundsen query (adjusting the sample_postgres_loader.py to suit), will always cause a ClusterNotFound connection error.
``
Any ideas on what could be the issue?

Hi @nam-tonthat-afterpay,
Can you confirm if the Python script which only uses redshift_connector is now consistently connecting?

On a simple query like this, it returns the ClusterNotFound error on the first run but on subsequent runs, it always returns (1,) as the output which is fantastic.

When this behavior occurs, what environment(s) are used to run the Python script (e.g. Jupyter)?

However, repeating it for the amundsen query (adjusting the sample_postgres_loader.py to suit), will always cause a ClusterNotFound connection error.

Could you share the modified sample_postgres_loader.py so I can work on reproducing this behavior? Also, how is sample_postgres_loader.py being run? Is the idea that it is manually invoked or done so as a scheduled task?

Hi @nam-tonthat-afterpay ,
Following up to see if you're still experiencing issues here.

Hi @Brooke-white , yes I am - I'm just a little tied up with some tickets that I have to close off - will update this thread by tomorrow! Apologies for the wait :'(.

Hi @Brooke-white, did a fresh install this morning and tried. Looks like everything is working perfectly. Closing the thread. The debug logging helped me iron out any issues. Thank you again for your help and patience on this!

yay! I'm happy to hear this, @nam-tonthat-afterpay. Best wishes on your project :)


Fix various types in module

Annotations/labels can only be map(string)
Make Webhook Ingress Hosts more flexible

Moving this to a new "home": https://github.com/SPHTech-Platform/terraform-aws-lb-controller


Refactor contributing doc

Closes #521

📚 Documentation preview 📚: https://earthaccess--533.org.readthedocs.build/en/533/

@andypbarrett I'm going to merge this one into update-documentation and re-do the remaining PRs to make them more reviewable. Sorry about that! Even though this PR will be closed feel free to continue discussions, I'll get notified.


fix dataplane and udp server builds on arm64

this needs to be llvm-19, now 😔

this is ready for another round of review @shaneutt  @EandrewJones

@aryan9600 Still failing 😢

From the logs it looks like the arch might be getting set incorrectly. Do you want it to be aarch64 or x86_64 for CI testing? The latter, correct? Well where is TARGETARCH being set?

Tell that to Flynn ;)
But I agree. arm64 builds are notoriously painful to maintain, in my
experience.
Best
Evan Jones
Website: www.ea-jones.com
On Fri, Oct 4, 2024 at 11:54 AM Shane Utt @.***> wrote:

@.**** commented on this pull request.
If this isn't going to be super straightforward (I say this because it
seems like we've had some issues) then I'm almost inclined to say we should
wait until someone else comes along and requests arm64 builds before we
spend more time on this?
—
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/blixt/pull/261#pullrequestreview-2348896310,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ2T6AMFWZEQOSW7ODWZSXDZZ3QABAVCNFSM6AAAAABK6UAWKGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNBYHA4TMMZRGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Oh I didn't realize Flynn needed this (the description was empty! there wasn't an issue for it!). I'll go chat with him about it, if this unblocks him then I would be in favor of pressing on actually, as I don't want anyone who's interested in the project to be "locked out" because of their system architecture if we can avoid it within a reasonable amount of effort.

My understanding is @aryan9600 initially hopped on this because Flynn
mentioned the build was broken for him. And Sanskar has been chasing llvm
versions ever since. We should confirm.
Best
Evan Jones
Website: www.ea-jones.com
On Fri, Oct 4, 2024 at 1:58 PM Shane Utt @.***> wrote:

Oh I didn't realize Flynn needed this (the description was empty! there
wasn't an issue for it!). I'll go chat with him about it, if this unblocks
him then I would be in favor of pressing on actually, as I don't want
anyone who's interested in the project to be "locked out" because of their
system architecture if we can avoid it within a reasonable amount of effort.
—
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/blixt/pull/261#issuecomment-2394577936,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ2T6AJTXE3XBSSK5U3NT6DZZ36PPAVCNFSM6AAAAABK6UAWKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJUGU3TOOJTGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

So yeah I talked with Flynn and all his development/testing is done on ARM64 systems so unblocking a community member is a great reason to continue. I did realize however that without the finished Rust control-plane the value of any builds is a bit dubious so just keep that in mind (but otherwise godspeed).


policy: Avoid crashing due to index out of range

Check Key.TrafficDirection for invalid values before using it to index an array. Return from denyPreferredInsertWithChanges() without doing anything if the given key has an invalid traffic direction. Dump a stack trace into logs if this happens so that we can trace where the invalid key originates from.
Fixes: #33313

/test

As I understand this is not a regression, only affects main so no need to backport.


Uncaught Error: dlopen(/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/...

[Enter steps to reproduce:]

...
...

Atom: 1.57.0 x64
Electron: 9.4.4
OS: Mac OS X 10.15.7
Thrown From: Atom Core
Stack Trace
Uncaught Error: dlopen(/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app.asar.unpacked/node_modules/keyboard-layout/build/Release/keyboard-layout-manager.node, 1): image not found
At electron/js2c/asar.js:140

Error: dlopen(/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app.asar.unpacked/node_modules/keyboard-layout/build/Release/keyboard-layout-manager.node, 1): image not found
    at process.func [as dlopen] (electron/js2c/asar.js:140:31)
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1034:18)
    at Object.func [as .node] (electron/js2c/asar.js:149:18)
    at Module.load (internal/modules/cjs/loader.js:815:32)
    at Module._load (internal/modules/cjs/loader.js:727:14)
    at Function.Module._load (electron/js2c/asar.js:769:28)
    at Module.require (/app.asar/static/index.js:72:46)
    at require (internal/modules/cjs/helpers.js:74:18)
    at customRequire (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:1:756278)
    at get_KeyboardLayoutManager (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:14:2890405)
    at get_manager (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:14:2890507)
    at Object.getCurrentKeyboardLayout (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:14:2890633)
    at e.keystrokeForKeyboardEvent (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:14:1118314)
    at KeymapManager.keystrokeForKeyboardEvent (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:11:1224088)
    at KeymapManager.handleKeyboardEvent (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:11:1222073)
    at WindowEventHandler.handleDocumentKeyEvent (/private/var/folders/87/mdrg_pkx6zq_1q5k316whnb00000gn/T/AppTranslocation/8950F105-A43C-403F-BCF3-EEC14DBD791A/d/Atom.app/Contents/Resources/app/static/<embedded>:11:286551)

Commands

Non-Core Packages
japanese-menu 1.15.0

Thanks for taking the time to contribute!
We noticed that this is a duplicate of https://github.com/atom/atom/issues/21285. You may want to subscribe there for updates.
Because we treat our issues list as the Atom team's backlog, we close duplicates to focus our work and not have to touch the same chunk of code for the same reason multiple times. This is also why we may mark something as duplicate that isn't an exact duplicate but is closely related.
For information on how to use GitHub's search feature to find out if something is a duplicate before filing, see the How Can I Contribute? section of the Atom CONTRIBUTING guide.


Update regression tests that use obsolete attribute values.

If you look at the testsuite log file, there are many warnings such as:
'backgroundFlag' is obsolete. Use 'opaqueColor'.
'foregroundFlag' is obsolete. Use 'meshColor'.
'specifyTextColor1' is obsolete. Trying to accommodate.
While I understand in part leaving them in the test, to ensure old scripts don't completely break when attributes change, the regression tests also serve as examples of how to do things in VisIt.  As such wouldn't it be better to update any test using obsolete values?
Cleaning these up would also make it easier to find actual problems with tests, as it would reduce the warning messages in the log file.
If we still wanted to ensure old scripts work (via testing), we could possibly introduce a new test, with the sole purpose of testing obsolete things.  'OldAtts' test or something like that. Say that the OldAtts test existed when the way that Mesh plot's colors were specified.  Any existing mesh plot test would have been updated to the new way, and a Mesh plot color specific test would be added to the OldAtts test, perhaps with a 'remove by version' tag.  It could be a bit of a maintenance headache though.

BTW...if we have a remove by version tag, that would be  cool too. There are a couple of places in VisIt C++ code I've done that. I think we need to be committed to removing old code and so we'd want to not only remove testing of the old logic but to also remove that old logic from our code. I discuss an idea for this here
BTW...just happened to be looking at this yesterday and by lines of code count, I think the only developer among us who consistently removes (old) code is @biagas and I think she should be highly regarded for that and that we should all try to follow her example. Removing unneeded code is at least as important if not more so than adding new code for new features. It is what helps keep the product maintainable/sustainable.

We should probably go back and remove commands that have been obsoleted a long time ago (like 10+ years ago).

To locate files that have code that can possibly be removed immediately: grep for VersionLessThan.
It is used in the ProcessOldVersions method for attributes.
For Py atts, can grep for Try and handle legacy fields but that may not be used by all.
Just need to decide the immediate cut-off version. anything for version < 2.0 ? 2.13?  3.0?

For future I'm thinking we can wrap all ProcessOldVersions and python legacy field handling in
#if VISIT_VERSION_GE(x,y,z)
#error Time to remove obsolete field support
#else
  // handle newly obsoleted field
#endif

Just need a standard policy: set removal for next major release? two major releases from current?

Makes sense to me. Do you see any value in trying to both have compile time behavior you describe but also be able to somehow disable this via either a single, global CMake flag or (more complex) code-block-specific CMake flags?

I'm not quite certain how something like this could be handled by CMake.   It seems very VisIt version specific, and dependent upon which VisIt version a field is removed/renamed.  Could you explain a bit more your ideas for CMake usage?

Whoops...didn't mean to suggest there is any support for something like this in CMake itself. Just that we'd define CMake-time, or make make-time variables that allow for disablement of the disablements 😮  if that makes sense.

You mean a code-parser that would look for something like "remove this code block at version x.y.z", and automagically remove that code block?  That would be something great to run outside of the build cycle, and could be done once at each change-in-major version.   It would also remove the onus from whichever developer happened to be the first to compile once the magic version number is reached.  Someone would still have to vet the changes and create a PR, though.

You mean a code-parser that would look for something like "remove this code block at version x.y.z",

No, I am not suggesting anything that sophisticated. All I am asking is whether it makes sense, in addition to what you describe in the above comment that we have the ability to override the effect of the #error Time to remove obsolete field support at either CMake-time or maybe make-time by providing appropriate CMake or make variables like so...
cmake -DVISIT_OVERRIDE_OBSELETE_CODE:BOOL=Yes ...

or
make OVERRIDE_OBSELETE_CODE=Yes

So, instead of using VISIT_VERSION_GE directly we'd have a new macro
#ifndef VISIT_OVERRIDE_OBSOLETE_CODE
#define VISIT_OBSOLETE_AT_VERSION(A,B,C) VISIT_VERSION_GE(A,B,C)
#else
#define VISIT_OBSOLETE_AT_VERSION(A,B,C) 0
#endif
.
.
.
#if VISIT_OBSOLETE_AT_VERSION(4,0,0)
#error This code is obsolete in this version. Please remove it.
#else
// old code here
#endif

such that, for example, something like make VISIT_OVERRIDE_OBSOLETE_CODE=1 has the effect of disabling the #if portion, above

@bartlettroscoe recommended these practices regarding deprecation of code. It might be worth looking over this to find pieces appropriate for our use case.

The practices suggested in the above ref are specified within the context of a very large eco-system of integrated codes. So, it is likely overkill or includes a lot of pieces not necessarily appropriate for VisIt as a single, stand-alone application. But, I'd bet there are some nuggets of functionality there worth stealing from.

Another colleague, @frobnitzem reminds me of some key pieces to this puzzle.

issueance of warning messages of some kind alerting users to functionality that is on its way to be deprecated

Such messages should include info about replacement functionality, if any, as well as drop dead date for when functionality will be fully removed.

removal of deprecated code should also then trigger removal of test code that tested the deprecated code

Results of team meeting #17052 was created, guidelines for immediate code cleanup.
After #17052 is complete, we should revisit any code block still handling legacy fields:

determine a version for its removal
wrap in deprecation/removal logic that includes warning message for user

Also to be done( see @markcmiller86  comment above):

add CMake var for turning on the code removal (to be used at beginning of each release cycle to determine code that should be removed).
add documentation for this new process, including policy that legacy field handling logic should be scheduled for removal for 2 point releases after deprecation (eg if deprecated in 3.3, then removal set for 3.5)

There is a problem getting the deprecation message from ProcessOldVersions to be viewable. avtCallback::IssueWarning is what I'm currently using. It prints out fine to the terminal window, but this doesn't work for a release version of VisIt on Windows, where there is no terminal window in which to print.  If not using the CLI, then ideally the message would show up in the GUI's output window.  That isn't happening on startup (when ProcessOldVersions is processed  for config files).  The red ! indicates a message waiting, but nothing is there. The message does show up in the gui log file.


doc: bluetooth: Update overview

Update the Bluetooth overview with additional content.
Signed-off-by: Carles Cufi<EMAIL_ADDRESS>

if we are updating the docs as part of this release, can we also merge this? https://github.com/zephyrproject-rtos/zephyr/pull/54124

if we are updating the docs as part of this release, can we also merge this? #54124

I approved it and requested a review from @jhedberg

Should the ICS tables be updated as well? I dont think they have been updated in a while?

Should the ICS tables be updated as well? I dont think they have been updated in a while?

Yes, but for that I need more time and the help of @cvinayak and @Thalley. Let's do that in a separate PR.


Write tests

(Try to) Write BDD tests.

Partially done in 0.0.9.
To add more tests and maybe add some E2E tests


How about add the blogs about tikv posted in Wechat ?

I've been seen several blogs about details of tikv in wechat, what makes me excited. It's wonderful to collect and sort out that articles and add them to doc. I wonder it should be a great tutorial for beginners :)

Thanks @XiaoZYang
Maybe you can watch our blog https://pingcap.github.io/blog/ now.
Later, we will publish more docs about the architecture, the key design of the modules in the TiKV repo directly to let the user know it easily.

Thanks for this suggestion, it's so kind of you~


Start or end point on polygon boundary

When the start or end point is on the boundary of a polygon, VisGraph.shortest_path() throws an error, as demonstrated in this test case:
import pyvisgraph as vg
polys = [[vg.Point(3.0,2.0), vg.Point(6.0,2.0), vg.Point(6.0,7.0), vg.Point(3.0,7.0)]]
g = vg.VisGraph()
g.build(polys)
s = vg.Point(3.0,5.0)
e = vg.Point(8.0,5.0)
path = g.shortest_path(s, e)
print path

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-64-23f53555023b> in <module>()
      5 s  = vg.Point(3.0,5.0)
      6 e = vg.Point(8.0,5.0)
----> 7 path = g.shortest_path(s, e)
      8 print path

pyvisgraph\vis_graph.py in shortest_path(self, origin, destination)
    128             for v in visible_vertices(destination, self.graph, origin=orgn):
    129                 add_to_visg.add_edge(Edge(destination, v))
--> 130         return shortest_path(self.visgraph, origin, destination, add_to_visg)
    131
    132     def point_in_polygon(self, point):

pyvisgraph\shortest_path.py in shortest_path(graph, origin, destination, add_to_visgraph)
     68         path.append(destination)
     69         if destination == origin: break
---> 70         destination = P[destination]
     71     path.reverse()
     72     return path

KeyError: Point(8.00, 5.00)

Not sure if this is the intended behavior.
It seems unusual that pyvisgraph will run the shortest path along a polygon boundary, e.g.  as in the illustration below, yet not allow the start or end point to be on the boundary.

(source code for this illustration is here)
In this application of a pixelated map with integer coordinates, it's not a trivial issue: the start and end points must be 1-pixel outside the polygon, yet the interior points of the path can be on the polygon boundary.
My clunky workaround is to use Shapely to move the start and/or end points outward along the edge normal by 0.1 units. This avoids the error, and gives the correct result when the path coordinates are rounded back to integers.

I'm having the same problem, but moving the points doesn't seem to be working for me.

I confirm running to the same issue. My workaround was to generate Shapely buffer around the point, iterate over the buffer's points and pick the one the most distatnt from the polygon.


Pasting Images on iOS

Describe the bug
When pasting images from clipboard on iOS they are not included/embedded.
To Reproduce
Steps to reproduce the behavior:

Copy image to clipboard
Paste in editor

Expected behavior
Image to be included/uploaded as per desktop functionality
Smartphone (please complete the following information):

Device:  iPhone XS & iPad Pro
OS: 13.4.1
Browser chrome, safari
Version React (also tested with the examples on here)

Error on pasting of image on iOS.

@Dag0n Currently, the Editor is not perfecting mobile device support. So it would be difficult to solve this issue right now.


ERROR in ./~/medium-editor/dist/css/themes/default.css Module parse failed: /home/bouhmid/Desktop/ctf_platform/react-hot-boilerplate/node_modules/medium-editor/dist/css/themes/default.css Unexpected token (1:0) You may need an appropriate loader to handle this file type. SyntaxError: Unexpected token (1:0)

Use css-loader https://github.com/webpack-contrib/css-loader