az debug commands leak user provided password (x-ms-secret fields)

Describe the bug

az commands run with the debug flag display the password in the terminal. My assumption is the logger az uses knack.cli, cannot distinguish secrets from other string inputs.
Currently there is no way to mask the user provided secret value (marked as x-ms-secret in the swagger) while developing az extensions
To Reproduce

An example is
az vm create --debug -g snaskar-rg --image UbuntuLTS -n test-vm-01 --admin-password Open-Secret-1

An example involving an az extension:
az scvmm vmmserver connect --debug -g snaskar-rg -l eastus2euap -cl '/subscriptions/204898ee-cd13-4332-b9d4-55ca5c25496d/resourcegroups/snaskar-rg/providers/microsoft.extendedlocation/customlocations/arcvmm-1201-cl' --fqdn '<IP_ADDRESS>' --username 'host\user' --password 'Open-Secret-1' -n arcvmm-1201-vmmserver

Expected behavior

Open-Secret-1 should not be displayed in the debug logs. This will be helpful if the user wants to share the logs for debugging any issues, also if the user is running the command sharing his screen.
Environment summary

Additional context

The complete logs can be found here.

Compute


 So, here is what's happened: __brewcomp_prev function finds previous non-switch word typed by the user on the command line. Currently it is not used by other Bash completion functions in Homebrew/brew, so I removed it in https://github.com/Homebrew/brew/pull/14545. It is, however, used by Bash completions in this repo (Homebrew/services). So, we can either restore it in Homebrew/brew (with a note that it might be used in other repos) or add it to Bash completions here.

@MikeMcQuaid, a gentle ping about the ☝🏻 Our options are: (1) restore __brewcomp_prev in Homebrew/brew or (2) add __brewcomp_prev here.

It should be here if this is the only user

@Nesze, thank you for reporting the issue! The fix (#551) has been merged in -- please update homebrew/services, re-source Bash completion script, and let us know if the issue has been resolved.

@maxim-belkin thank you for the quick fix! I can confirm that auto-complete now works w/o any warnings.


[clkmgr/dif] Align DIF to use mubi type

It appears that this has slipped through the CI cracks and is breaking TOT at the moment.
Signed-off-by: Michael Schaffner<EMAIL_ADDRESS>

thanks, i think there was some PR crossing.
On Wed, Nov 3, 2021 at 2:14 PM Timothy Trippel @.***>
wrote:

@.**** approved this pull request.
LGTM, thanks @msfschaffner https://github.com/msfschaffner !
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/pull/9034#pullrequestreview-797105072,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSSBESAOLSR3YNAV3ODUKGX2XANCNFSM5HJ6H4PA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


bolt: doesn't pass tests after latest version bump

Describe the bug
bolt 0.9 appears to have broken/flaky tests. As of #103213 I cannot rebuild my system. Hydra fails in the same way: https://hydra.nixos.org/build/130437438
To Reproduce
Build bolt-0.9, (hopefully) observe test failure.
Expected behavior
Bolt passes tests/has tests disabled.
Screenshots
N/A
Additional context
N/A
Notify maintainers
@callahad
Metadata
 nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.9.8, NixOS, 21.03pre-git (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.8`
 - channels(root): `"nixpkgs-maxine, home-manager"`
 - channels(maxine): `"home-manager"`
 - nixpkgs: `/var/run/current-system/systems/nixpkgs`

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

cc @peterhoeg

This is now blocking the nixos-unstable channel. One restart didn't fix it.
Upstream issue https://gitlab.freedesktop.org/bolt/bolt/-/issues/167. No response for two weeks.

The perceived nondeterminism of the test may have been caused by the presence of noatime mount options on some/most of the builders.
I've proposed to remove atime-dependent tests in #106113 and upstream (linked from there)


Cannot build INLA from source on Windows

If we try and build INLA from source, we get the error message below:-
> install.packages("INLA",
                 repos = c(getOption("repos"),
                           INLA = "https://inla.r-inla-download.org/R/stable"),
                 type = "source",
                 dep = TRUE)

Installing package into ‘C:/Users/wrh1/Documents/R/win-library/4.4’
(as ‘lib’ is unspecified)
Warning in install.packages :
  dependencies ‘Rgraphviz’, ‘graph’ are not available
trying URL 'https://inla.r-inla-download.org/R/stable/src/contrib/INLA_24.05.10.tar.gz'
Content type 'application/x-gzip' length 61448956 bytes (58.6 MB)
downloaded 58.6 MB

* installing *binary* package 'INLA' ...
C:\Windows\cp.exe: invalid option -- )
Try `C:\Windows\cp.exe --help' for more information.
ERROR: installing binary package failed
* removing 'C:/Users/wrh1/Documents/R/win-library/4.4/INLA'
* restoring previous 'C:/Users/wrh1/Documents/R/win-library/4.4/INLA'
Warning in install.packages :
  installation of package ‘INLA’ had non-zero exit status

This is easy to work around with R 4.4 - just don't build from source, and the binary gets downloaded - if the ready-build version is available. But for users still on R 4.3, there is no matching binary of 24.05.10 in https://inla.r-inla-download.org/R/stable/bin/windows/contrib/4.3/, so install.packages tries to build 24.05.10 from source. They have to either upgrade to R 4.4, or specify version 24.05.01-1 manually, because there's a binary for that.
So the workarounds are not hard, but is seems there's a problem in the source version for building at the moment.

The build process for the INLA R package requires a manual step of first building the standalone inla binary executable, so no INLA package version can be built purely using the normal R build process. When installing from the repositories, one cannot "build" the package in a working format, as that will not include the executable, so "building from source" will not work on any R version, or any existing INLA source code version, without manual intervention. This may be improved in the future, but the build process is difficult to fully automate.
The "stable" repository cannot yet be used with R 4.4, as only the testing version has been built for R 4.4, currently one must use testing with 4.4, and stable with 4.3. But yes, the missing binary for 24.05.10 will cause an issue. R should let you specify that you only want "binary" installation, which should work around the issue without needing to specify the version (as that's not supported by install.packages()):
install.packges("INLA", repos = ..., type = "binary")

The documentation for install.packages contains a further option:

For a binary install from a repository, the function checks for the availability of a source package on the same repository, and reports if the source package has a later version, or is available but no binary version is. This check can be suppressed by using
  options(install.packages.check.source = "no")

and should be if there is a partial repository containing only binary files.

OK - thanks for these workarounds - but just to highlight the immediate problem - So if we want to use the stable version, we should use R 4.3 for now - fine - but if an R 4.3 user follows the installation instructions from here (https://www.r-inla.org/download-install) - which say:
install.packages("INLA",repos=c(getOption("repos"),INLA="https://inla.r-inla-download.org/R/stable"), dep=TRUE)

that will fail with the error message I got above because it will try to install 24.05.10 from source. So I think for now, either updating the installation instructions with the workaround to make install.packages not do that, or filling in the missing binary in the 4.3 folder might be helpful to do.

Further - I tested on R 4.3 with options(install.packages.check.source = "no") - but it made no difference:-
> options(install.packages.check.source = "no")
> install.packages("INLA",repos=c(getOption("repos"),INLA="https://inla.r-inla-download.org/R/stable"), dep=TRUE)
Warning in install.packages :
  partial argument match of 'dep' to 'dependencies'
Installing package into ‘E:/Rlib’
(as ‘lib’ is unspecified)
Warning in install.packages :
  dependencies ‘Rgraphviz’, ‘graph’ are not available

  There is a binary version available but the source version is later:
         binary   source needs_compilation
INLA 24.05.01-1 24.05.10             FALSE

installing the source package ‘INLA’

trying URL 'https://inla.r-inla-download.org/R/stable/src/contrib/INLA_24.05.10.tar.gz'
Content type 'application/x-gzip' length 61448956 bytes (58.6 MB)
downloaded 58.6 MB

* installing *binary* package 'INLA' ...
C:\WINDOWS\cp.exe: invalid option -- )
Try `C:\WINDOWS\cp.exe --help' for more information.
ERROR: installing binary package failed
* removing 'E:/Rlib/INLA'
Warning in install.packages :
  installation of package ‘INLA’ had non-zero exit status

The downloaded source packages are in
	‘E:\Rtmp\Rtmp4ows8C\downloaded_packages’

Did the type = "binary" method work?

I see in the NEWS.md file that 24.05.10 was the first R 4.4 build, so the error with that one is that it shouldn't be present in any form in the 4.3 folder.

Have you found a method for specifying a specific INLA version that actually works? I don't see an option for that in install.packages, and I don't have access to try remotes::install_version() on Windows.

type = "binary" on the install.packages did work - it picked the most recent binary correctly.
trying URL 'https://inla.r-inla-download.org/R/stable/bin/windows/contrib/4.3/INLA_24.05.01-1.zip'

Yes, so the issue then is why does it still try to build 24.05.10 from source in R 4.3

It tries to build it because it finds a newer source package. I would have expected turning off that check to have fixed it; that may be an error in the install.packages documentation for that feature.

The most generally useful approach on windows and macos seems to be for such users to always specify type = "binary" when installing INLA, so that seems to be the hint that needs to be added to the webpage. That way, the instruction will work also in future cases, and not having to rely on specific version instructions.

Have you found a method for specifying a specific INLA version that actually works? I don't see an option for that in install.packages, and I don't have access to try remotes::install_version() on Windows.

Yes, you're right - and further remotes::install_version requires type="source". So type="binary" indeed seems the best option, since building from source on Windows does not seem likely to be possible.
(On our cluster, we use pkgdepends, which can let us specify both the repo and version)

I see that https://www.r-inla.org/download-install already does mention the remotes::install_version() option, specifically in the context of the transition period when INLA gets built for new R versions. But I think as long as the available packages in the repository section for each version are correct, then the install.packages(..., type = "binary") option would be sufficient. So removing the 24.05.10 source package from the 4.3 section might make the type argument the only thing needed to handle the issue for all R versions.

I think all the instructions for remotes::install_version probably need reviewing.
From R 4.3, the instructions there do the following:-
> remotes::install_version("INLA", version="23.05.30",repos=c(getOption("repos"),INLA="https://inla.r-inla-download.org/R/testing"), dep=TRUE)
Trying https://cloud.r-project.org
Trying https://inla.r-inla-download.org/R/testing
Error in download_version_url(package, version, repos, type) :
  version '23.05.30' is invalid for package 'INLA'

and for R 4.4, since remotes::install_version has default type='source' that can't be changed to binary, we get the familiar error:-
> remotes::install_version("INLA", version="24.05.10",repos=c(getOption("repos"), INLA="https://inla.r-inla-download.org/R/testing"), dep=TRUE)
Downloading package from url: https://inla.r-inla-download.org/R/testing/src/contrib/INLA_24.05.10.tar.gz
Installing package into ‘C:/Users/wrh1/Documents/R/win-library/4.4’
(as ‘lib’ is unspecified)
* installing *binary* package 'INLA' ...
C:\Windows\cp.exe: invalid option -- )
Try `C:\Windows\cp.exe --help' for more information.
ERROR: installing binary package failed

Incidentally, for R 4.3 the issue is that version 23.05.30 doesn't exist - the earliest is 23.05.30-1 - and the most recent 24.05.01-1 - but again, you can't install any of these with remotes::install_version because that would require compilation from source. (I haven't tested R 4.1)
So at present, the only way you can install any INLA package on any R version is
install.packages("INLA",repos=c(getOption("repos"),INLA="https://inla.r-inla-download.org/R/stable"), dep=TRUE, type="binary")

which will get the most recent binary for your R version. There is no way I think to get a specific older version, or to use any method that involves compiling from source.

There's always a way: downloading the needed tar/zip file manually and installing from the file instead of the repository. But the type="binary" option is easier.


”Borleasa”, ”Cireași”, ”Lunca Sătească” are listed in the drop-down menu

Description:
When user writes ”Borleasa”, ”Cireași”, ”Lunca Sătească” the locations appear in the drop-down menu next to the comune and the county witch is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Borleasa"

Expected results
The location appears in the drop-down menu as "Borleasa, Bistrița-Năsăud (Târlișua)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Cireași"

Expected results
The location appears in the drop-down menu as "Cireași, Bistrița-Năsăud (Târlișua)".


Problem in downloading Xgboot  in Windows

Thanks for participating in the XGBoost community! We use https://discuss.xgboost.ai for any general usage questions and discussions. The issue tracker is used for actionable items such as feature proposals discussion, roadmaps, and bug tracking.  You are always welcomed to post on the forum first :)
Issues that are inactive for a period of time may get closed. We adopt this policy so that we won't lose track of actionable issues that may fall at the bottom of the pile. Feel free to reopen a new one if you feel there is an additional problem that needs attention when an old one gets closed.
For bug reports, to help the developer act on the issues, please include a description of your environment, preferably a minimum script to reproduce the problem.
For feature proposals, list clear, small actionable items so we can track the progress of the change.

Hi,
Facing problem in downloading xgboot  and mu OS is windows 10 : showing below error msg  appear in jyuter

Hi,
Facing problem in downloading xgboot and mu OS is windows 10 : showing below error msg appear in jyuter note book
Collecting xgboost
Using cached xgboost-1.4.2.tar.gz (716 kB)
Requirement already satisfied: numpy in c:\users\13213\anaconda3\lib\site-packages (from xgboost) (1.19.2)
Requirement already satisfied: scipy in c:\users\13213\anaconda3\lib\site-packages (from xgboost) (1.5.2)
Building wheels for collected packages: xgboost
Building wheel for xgboost (setup.py): started
Building wheel for xgboost (setup.py): finished with status 'error'
Running setup.py clean for xgboost
Failed to build xgboost
Installing collected packages: xgboost
Running setup.py install for xgboost: started
Running setup.py install for xgboost: finished with status 'error'

Can you run
conda list -f pip

and paste the output?


Hooks

After installing it should run the after hooks.
https://github.com/tinytacoteam/dotties-vim/blob/818cb6d3caa6b216158301a70a381d9d5374509f/.dotties.yml#L4

Fixed with https://github.com/tinytacoteam/dotties-core/commit/585a647c2520139a0e126feee02917af99b3eeb0


Device manager stopped listing devices and templates (kubernetes)

On a kubernetes environment, for some reason, after a while device manager stopped replying the GUI with devices and templates configuration data.
All calls to the service result in a 500 error being returned. After a forced restart of the service everything seemed to be working again.
Service logs for such failed calls follow:
[2018-04-10 19:13:53,998] ERROR in app: Exception on /device [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "devices" does not exist
LINE 2: FROM devices
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/src/app/DeviceManager/DeviceHandler.py", line 586, in flask_get_devices
    result = DeviceHandler.get_devices(request)
  File "/usr/src/app/DeviceManager/DeviceHandler.py", line 190, in get_devices
    page = db.session.query(Device).paginate(**pagination)
  File "/usr/local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 491, in paginate
    items = self.limit(per_page).offset((page - 1) * per_page).all()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2726, in all
    return list(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2901, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "devices" does not exist
LINE 2: FROM devices
             ^
 [SQL: 'SELECT devices.id AS devices_id, devices.label AS devices_label, devices.created AS devices_created, devices.updated AS devices_updated, devices.persistence AS devices_persistence \nFROM devices \n LIMIT %(param_1)s OFFSET %(param_2)s'] [parameters: {'param_1': 1000, 'param_2': 0}] (Background on this error at: http://sqlalche.me/e/f405)
<IP_ADDRESS> - - [10/Apr/2018:19:13:53 +0000] "GET /device?page_size=1000 HTTP/1.1" 500 25 "http://<IP_ADDRESS>/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
[2018-04-10 19:13:54,804] ERROR in app: Exception on /template [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "templates" does not exist
LINE 3: FROM templates
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/src/app/DeviceManager/TemplateHandler.py", line 224, in flask_get_templates
    result = TemplateHandler.get_templates(request)
  File "/usr/src/app/DeviceManager/TemplateHandler.py", line 64, in get_templates
    error_out=False)
  File "/usr/local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 491, in paginate
    items = self.limit(per_page).offset((page - 1) * per_page).all()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2726, in all
    return list(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2901, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "templates" does not exist
LINE 3: FROM templates
             ^
 [SQL: 'SELECT anon_1.templates_id AS anon_1_templates_id, anon_1.templates_label AS anon_1_templates_label, anon_1.templates_created AS anon_1_templates_created, anon_1.templates_updated AS anon_1_templates_updated, attrs_1.id AS attrs_1_id, attrs_1.label AS attrs_1_label, attrs_1.created AS attrs_1_created, attrs_1.updated AS attrs_1_updated, attrs_1.type AS attrs_1_type, attrs_1.value_type AS attrs_1_value_type, attrs_1.static_value AS attrs_1_static_value, attrs_1.template_id AS attrs_1_template_id, attrs_1.parent_id AS attrs_1_parent_id \nFROM (SELECT templates.id AS templates_id, templates.label AS templates_label, templates.created AS templates_created, templates.updated AS templates_updated \nFROM templates \n LIMIT %(param_1)s OFFSET %(param_2)s) AS anon_1 LEFT OUTER JOIN attrs AS attrs_1 ON anon_1.templates_id = attrs_1.template_id'] [parameters: {'param_1': 20, 'param_2': 0}] (Background on this error at: http://sqlalche.me/e/f405)
<IP_ADDRESS> - - [10/Apr/2018:19:13:54 +0000] "GET /template HTTP/1.1" 500 25 "http://<IP_ADDRESS>/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
[2018-04-10 19:13:54,817] ERROR in app: Exception on /device [GET]

This error originates from a prior failure on the PostgreSQL cluster where one of the nodes died and the access to the leader became corrupted until a new leader was assigned.
The log for the postgres error can be seen below:

2018-04-10 17:55:31,426 WARNING: Connection dropped: outstanding heartbeat ping not received
2018-04-10 17:55:31,427 WARNING: Transition to CONNECTING
2018-04-10 17:55:31,428 INFO: Zookeeper connection lost
2018-04-10 17:55:31,599 INFO: Connecting to zk-client:2181
2018-04-10 17:55:34.894 27 LOG {ticks: 90, maint: 0, retry: 0}
2018-04-10 17:55:40,823 ERROR: get_cluster
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/kazoo/retry.py", line 123, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/patroni/dcs/zookeeper.py", line 148, in _inner_load_cluster
    nodes = set(self.get_children(self.client_path(''), self.cluster_watcher))
  File "/usr/local/lib/python3.4/dist-packages/patroni/dcs/zookeeper.py", line 132, in get_children
    return self._client.get_children(key, watch)
  File "/usr/local/lib/python3.4/dist-packages/kazoo/client.py", line 1079, in get_children
    return self.get_children_async(path, watch, include_data).get()
  File "/usr/local/lib/python3.4/dist-packages/kazoo/handlers/utils.py", line 72, in get
    raise self._exception
kazoo.exceptions.SessionExpiredError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/patroni/dcs/zookeeper.py", line 196, in _load_cluster
    self._client.retry(self._inner_load_cluster)
  File "/usr/local/lib/python3.4/dist-packages/kazoo/client.py", line 273, in _retry
    return self._retry.copy()(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/kazoo/retry.py", line 136, in __call__
    raise RetryFailedError("Exceeded retry deadline")
kazoo.retry.RetryFailedError: Exceeded retry deadline
2018-04-10 17:55:40,827 ERROR: Error communicating with DCS
waiting for server to shut down....2018-04-10 17:55:41.353 27 ERROR connection error: PQsendQuery
2018-04-10 17:55:41.353 27 ERROR libpq: server closed the connection unexpectedly
		This probably means the server terminated abnormally
		before or while processing the request.
2018-04-10 17:55:41.353 27 WARNING dojot_auth: default timeout
2018-04-10 17:55:41.354 27 ERROR connection error: PQsendQuery
2018-04-10 17:55:41.354 27 ERROR libpq: server closed the connection unexpectedly
		This probably means the server terminated abnormally
		before or while processing the request.
2018-04-10 17:55:41.354 27 WARNING kong: default timeout
2018-04-10 17:55:41.354 27 ERROR connection error: PQsendQuery
2018-04-10 17:55:41.354 27 ERROR libpq: server closed the connection unexpectedly
		This probably means the server terminated abnormally
		before or while processing the request.
2018-04-10 17:55:41.354 27 WARNING dojot_devm: default timeout
....2018-04-10 17:55:45.171 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:45.171 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:45.171 27 WARNING dojot_devm: default reconnect
....2018-04-10 17:55:49.584 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:49.584 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:49.584 27 WARNING kong: default reconnect
..2018-04-10 17:55:51,623 WARNING: Connection dropped: socket connection error: Name or service not known
.2018-04-10 17:55:52,346 INFO: Connecting to zk-client:2181
..2018-04-10 17:55:54.602 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:54.602 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:54.602 27 WARNING dojot_devm: default reconnect
2018-04-10 17:55:54.602 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:54.603 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:54.603 27 WARNING kong: default reconnect
2018-04-10 17:55:54.605 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:54.605 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:54.605 27 WARNING dojot_auth: default reconnect
2018-04-10 17:55:54.614 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:54.614 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:54.616 27 ERROR connection error: PQconnectPoll
2018-04-10 17:55:54.616 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:55:54.616 27 WARNING postgres: default timeout
.........2018-04-10 17:56:03.083 27 ERROR connection error: PQconnectPoll
2018-04-10 17:56:03.083 27 ERROR libpq: FATAL:  the database system is shutting down
2018-04-10 17:56:03.083 27 WARNING dojot_auth: default reconnect
..2018-04-10 17:56:04.896 27 LOG {ticks: 18, maint: 0, retry: 0}
..2018-04-10 17:56:07,368 WARNING: Connection dropped: socket connection error: Name or service not known
.2018-04-10 17:56:08,131 INFO: Connecting to zk-client:2181
2018-04-10 17:56:08,143 WARNING: Session has expired
2018-04-10 17:56:08,143 INFO: Zookeeper session lost, state: EXPIRED_SESSION
.2018-04-10 17:56:08,966 INFO: Connecting to zk-client:2181
2018-04-10 17:56:08,976 INFO: Zookeeper connection established, state: CONNECTED
...2018-04-10 17:56:11,979 CRIT reaped unknown pid 56)
 done
server stopped
2018-04-10 17:56:12,906 CRIT reaped unknown pid 57)
2018-04-10 17:56:12,927 INFO: closed patroni connection to the postgresql cluster
2018-04-10 17:56:13,002 INFO: postmaster pid=30048
2018-04-10 17:56:13 UTC [30048]: [1-1] 5accfabd.7560 0     LOG:  redirecting log output to logging collector process
2018-04-10 17:56:13 UTC [30048]: [2-1] 5accfabd.7560 0     HINT:  Future log output will appear in directory "../pg_log".
localhost:5432 - accepting connections
localhost:5432 - accepting connections
2018-04-10 17:56:14,141 INFO: demoted self because DCS is not accessible and i was a leader
2018-04-10 17:56:14,145 WARNING: Loop time exceeded, rescheduling immediately.
2018-04-10 17:56:14,158 INFO: establishing a new patroni connection to the postgres cluster
2018-04-10 17:56:14,175 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:14,175 INFO: does not have lock
2018-04-10 17:56:14,185 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:56:14,199 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:14,199 INFO: changing primary_conninfo and restarting in progress
2018-04-10 17:56:14,206 INFO: rewind flag is set
waiting for server to shut down....2018-04-10 17:56:14,238 CRIT reaped unknown pid 30048)
2018-04-10 17:56:14,239 CRIT reaped unknown pid 30050)
2018-04-10 17:56:15.173 27 ERROR connection error: PQconnectStart
2018-04-10 17:56:15.173 27 ERROR libpq: could not connect to server: No such file or directory
		Is the server running locally and accepting
		connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
2018-04-10 17:56:15.173 27 WARNING dojot_devm: default reconnect
 done
server stopped
2018-04-10 17:56:15,528 INFO: running pg_rewind from user=postgres host=<IP_ADDRESS> port=5432 dbname=postgres sslmode=prefer sslcompression=1
servers diverged at WAL position 0/640005A0 on timeline 1
rewinding from last common checkpoint at 0/640004F8 on timeline 1
2018-04-10 17:56:19.589 27 ERROR connection error: PQconnectStart
2018-04-10 17:56:19.590 27 ERROR libpq: could not connect to server: No such file or directory
		Is the server running locally and accepting
		connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
2018-04-10 17:56:19.590 27 WARNING kong: default reconnect
2018-04-10 17:56:24,188 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:24,196 INFO: changing primary_conninfo and restarting in progress
Done!
2018-04-10 17:56:25,612 INFO: closed patroni connection to the postgresql cluster
2018-04-10 17:56:25,679 INFO: postmaster pid=30077
2018-04-10 17:56:25 UTC [30077]: [1-1] 5accfac9.757d 0     LOG:  redirecting log output to logging collector process
2018-04-10 17:56:25 UTC [30077]: [2-1] 5accfac9.757d 0     HINT:  Future log output will appear in directory "../pg_log".
localhost:5432 - accepting connections
localhost:5432 - accepting connections
2018-04-10 17:56:26,853 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:26,854 INFO: does not have lock
2018-04-10 17:56:26,854 INFO: establishing a new patroni connection to the postgres cluster
2018-04-10 17:56:26,880 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:56:33.106 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:56:34.899 27 LOG {ticks: 0, maint: 0, retry: 0}
2018-04-10 17:56:36,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:36,842 INFO: does not have lock
2018-04-10 17:56:36,845 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:56:41.371 27 LOG dojot_auth: pgq version ok: 3.2.6
2018-04-10 17:56:41.376 27 LOG dojot_devm: pgq version ok: 3.2.6
2018-04-10 17:56:41.377 27 LOG kong: pgq version ok: 3.2.6
2018-04-10 17:56:45.183 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:56:46,840 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:46,840 INFO: does not have lock
2018-04-10 17:56:46,846 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:56:49.601 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:56:53.116 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:56:56,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:56:56,842 INFO: does not have lock
2018-04-10 17:56:56,845 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:04.900 27 LOG {ticks: 72, maint: 3, retry: 0}
2018-04-10 17:57:05.197 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:06,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:06,841 INFO: does not have lock
2018-04-10 17:57:06,844 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:09.612 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:13.129 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:16,840 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:16,841 INFO: does not have lock
2018-04-10 17:57:16,843 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:25.212 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:26,842 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:26,842 INFO: does not have lock
2018-04-10 17:57:26,845 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:29.625 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:33.140 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:34.901 27 LOG {ticks: 90, maint: 0, retry: 0}
2018-04-10 17:57:36,840 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:36,841 INFO: does not have lock
2018-04-10 17:57:36,844 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:45.225 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:46,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:46,841 INFO: does not have lock
2018-04-10 17:57:46,844 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:57:49.635 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:53.152 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:57:56,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:57:56,841 INFO: does not have lock
2018-04-10 17:57:56,844 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:58:04.902 27 LOG {ticks: 90, maint: 0, retry: 0}
2018-04-10 17:58:05.237 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:58:06,840 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:58:06,840 INFO: does not have lock
2018-04-10 17:58:06,847 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:58:09.644 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:58:13.166 27 ERROR dojot_auth: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:58:16,841 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:58:16,841 INFO: does not have lock
2018-04-10 17:58:16,848 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:58:25.248 27 ERROR dojot_devm: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement
2018-04-10 17:58:26,840 INFO: Lock owner: postgres-1; I am postgres-0
2018-04-10 17:58:26,841 INFO: does not have lock
2018-04-10 17:58:26,843 INFO: no action.  i am a secondary and i am following a leader
2018-04-10 17:58:29.656 27 ERROR kong: ERROR:  cannot execute LOCK TABLE during recovery
	CONTEXT:  SQL statement "lock table pgq.retry_queue in share update exclusive mode"
	PL/pgSQL function pgq.maint_retry_events() line 20 at SQL statement

The issue  is related to this repeating message on ZooKeeper:
2018-04-09 13:20:11,905 [myid:2] - WARN  [NIOServerCxn.Factory:<IP_ADDRESS>/<IP_ADDRESS>:2181:NIOServerCnxnFactory@188] - Too many connections from /<IP_ADDRESS> - max is 60

Where the presented IP is associated with the data broker POD.


Feat: Trade Advancements

Added Trade Advancement for picking up an incomplete trade - to hint to the player to trade
Added auto-generating advancements (hidden) to be used for backwards compatibility

Test 123

way to many updates...

No such thing as too many updates @JonerCrack1


[Bug] Can't load file for file token

Insomnia Version: 5.14.3
Operating System: win10

Details

Create request
Add a file token and click it
Choose file click Done
File token is in warning state

Quick look at error message says:
ENOENT: no such file or directory, open 'C:\Users\Owner\AppData\Local\insomnia\app-5.14.6\UsersOwnerDesktopimage.png'
So looks like it's resolving file path wrongly!
Thanks 😄

Another thing to note is that the live preview takes literally minutes to load and is blocking.

Exporting data shows path of
"value": "{% file 'C:\\\\Users\\\\Owner\\\\Desktop\\\\image.png' %}"

If I change this to
"value": "{% file 'C:/Users/Owner/Desktop/image.png' %}"

it works fine

Hmm, I hesitate to simply replace all backslashes with forward slashes. I assume this hack would break if a file or folder name contained a forward slash?
It would be nice to figure out the actual cause of this issue. I don't currently have a Windows machine set up for development. Any chance you'd be interested in working on a fix for this?

Sure, I'll give it a go 😄

In case you want/need a place to start... I'd first figure out if the value of path here is what you expect: https://github.com/getinsomnia/insomnia/blob/b3e473071824f0e48e7c26507163182acbe01703/plugins/insomnia-plugin-file/index.js#L18

Thanks, saved me digging 😄

I believe I've replicated the root cause on MacOS as well when having a file with a backslash in it. Happy to try fix this but digging into the code it seems there is some tokenization and untokenization which could be the root cause as I do see some backslash handling/manipulations
https://github.com/Kong/insomnia/blob/1bbbf38a9653792fd137662d015846f4334b1989/packages/insomnia-app/app/templating/utils.js#L133-L141
https://github.com/Kong/insomnia/blob/1bbbf38a9653792fd137662d015846f4334b1989/packages/insomnia-app/app/templating/utils.js#L168-L186
Select file with Backslash

Initial loading and preview works

Bring up modal again and preview doesn't work

Request works
The request using the file selected still works so the problem looks to be related to the display/preview

This problem seems to be an issue with backslashes in general as a similar problem shows up when trying to Base64encode (similar to above, requests sents do have the correct data, so the problem lies in the just the view portion it seems)

Sorry folks! This was incorrectly tagged as a feature request along with other bugs. I'm going back through and cleaning up, sorry for the inconvenience.


MicEnabled bug fix and Invisible

MicEnabled bug fix and added Invisible property
Description

MicEnabled ==='1' // instead of 1

Added Invisible to same part

Invisible === '1' // instead of 1

was already done in beta - these  changes replace the beta stuff.

Your checklist for this pull request
🚨 Please review the guidelines for contributing to this repository.

[ x ] Make sure you are requesting to pull a topic/feature/bugfix branch (right side). Don't request your master!
[ x ] Make sure you are making a pull request against the beta branch (left side). Also you should start your branch off svrooij/node-sonos-ts/beta.
[ ] Check the commit's or even all commits' message styles matches our requested structure.
[ ] Check your code additions will fail neither code linting checks nor unit test.
[ ] Check you add tests for added code.

This error is fixed in #129, which will be released to beta any minute.


Misuv 2862 liability calculation model

Description
Simply creates the LiabilityResponse calculation model
Add a link to the relevant story in Jira
(https://jira.tools.tax.service.gov.uk/browse/MISUV-2750)
Checklist PR Reviewer
Before Reviewing

[x]  Have you pulled the branch down?
[x]  Have you assigned yourself to the PR?
[x]  Have you moved the task to “in review” on JIRA?
[x]  Have you checked to ensure all dependencies are up to date?
[x]  Have you checked to ensure its been rebased against the current version of main?

Whilst Reviewing

[x]  Have you run the tests?

[x]  Have you looked at the JIRA story to make sure all Acceptance Criteria has been met?

After Reviewing

[x]  Have you checked for merge conflicts or any changes in the current main that may affect the current pull request? i.e. does it need another rebase?
[x]  Have you checked to make sure there are no builds in the pipeline before you merge?
[x]  Have you moved the task to “in pipeline” on Jira?

[x]  Have you rebased against the current version of main?
[x]  Have you checked code coverage isn’t lower than previously?

After PRs been raised

[x]  Have you checked the PR Builder passes?

Please squash the commits. If for some reason you want to keep some separate, ensure at least the following are squashed:

MISUV-2862 initial model creation and tests
working model and tests
fix tests
MISUV-2862 fix scalastyle
fix test import


Predefined migration not detected in pnpm monorepo

Link to reproduction
.
To Reproduce

Configure pnpm monorepo
npx create-payload-app w/ mongo
pnpm --filter my-payload run payload migrate:create --file @payloadcms/db-mongodb/versions-v1-v2
The generated migration will show as a blank migration (empty up and down funcs).

Describe the Bug
In a pnpm monorepo, generating the @payloadcms/db-mongodb/versions-v1-v2 predefined migration generates a blank migration instead of finding the predefined file to copy.
This means it is not finding the file's path properly. We should show an error if the file is not found.
Payload Version
2.0.3
Adapters and Plugins
db-mongodb

User was able to test multiple scenarios: Discord Thread

I played with empty project and here is what I found:

pnpm run --filter ... works from root.
pnpm run ... works from payload folder.
npx ... works from payload folder.
npm run ... gives empty migration from payload folder.

It looks like we are already logging an error and exiting if the file can't be found.
https://github.com/payloadcms/payload/blob/main/packages/db-mongodb/src/createMigration.ts#L45
It looks like the way the script was being run was dropping the --file param, possibly.


LSTM Returning Questionable Results

What is wrong?
I have trained an LSTM model but am getting poor results when testing strings (potentially two issues). The first is that results that one would presume to be negative instead are showing positive. Secondly, there are inputs tested with net.run() that don't return any result at all.
Where does it happen?
I am running a linux machine using node. It occurs after training the data when loading that data with net.fromJSON() and running future examples with net.run().
How do we replicate the issue?

I have included sample data and code here: https://jsfiddle.net/farverio/z378bnoa/6/
How important is this (1-5)?
In the greater scheme of things, probably a 0-1 as its for a school project not a production environment but it feels like a 3-4 today and any help would still be appreciated :)
Expected behavior (i.e. solution)
With any net.run(testString), I would presume the testString input is coerced to a positive or negative output (I only have two outputs).
Secondly, I'd have thought that the results might be more accurate. In the domain example, I am categorizing the interest of sample email responses as 'interested' or 'not interested'. When I type the exact copy of a not interested input that I used to train ("No interest."), it successfully returns not interested. However, when I remove the period at the end of the input, ("No interest"), it instead returns interested.
Is this normal behavior? How should I fix to produce more meaningful results?
Other Comments
I tried to follow documentation pretty closely so am at a loss about both types of errors. Any help in resolving this would be greatly appreciated!!

Its a little late, I'll take a look in the morning, but the first thing that stands out is iterations: 100.  Crank that up to iterations: 5000 or even iterations: 10000, and your results will vastly improve.

Thanks @robertleeplummerjr . Was looking over a lot of previous posts from you and hoping you'd reply. Appreciate you taking some time out of the holiday weekend to check things out :)

After playing with it a bit, I think the problem is just lack of data and overfitting.  If we look at your training set, the following have punctuation:
  {"input":"Thank you but we are not interested at this time.", "output":"Not Interested"},
  {"input":"No interest.", "output":"Not Interested"},
  {"input":"Dont contact me further!!", "output":"Not Interested"},
  {"input":"Please remove me from your list !!!!", "output":"Not Interested"},
  {"input":"No interest please remove us.", "output":"Not Interested"},
  {"input":"Please tell me more!", "output":"Interested"},
  {"input":"Are you free for a call?", "output":"Interested"},
  {"input":"How do I sign up?", "output":"Interested"},
  {"input":"Let's set up an introductory meeting.", "output":"Interested"}

The following to not have punctuation:
  {"input":"Not interested", "output":"Not Interested"},
  {"input":"No thank you", "output":"Not Interested"},
  {"input":"Please stop emailing", "output":"Not Interested"},
  {"input":"For now I am not interested", "output":"Not Interested"},
  {"input":"I'd like to hear more", "output":"Interested"},
  {"input":"Busy this week but let's schedule some time next week", "output":"Interested"},
  {"input":"I am open for a quick 10 minute call", "output":"Interested"},
  {"input":"What is the total cost", "output":"Interested"},
  {"input":"I forwarded your email to our CEO - he'll follow up", "output":"Interested"},

I don't think there is enough data yet for it to form a solid basis for a reply, but I could be wrong.  There are three sentences that end with period, and possibly that is having causing it to weigh towards "NotInterested".  I'm playing around with a deeper net to see if I can decouple this problem.

I cranked up the hidden layers, put iterations at 2k, errorThresh at 1.1, and added another input, it seemed to give a better result: https://jsfiddle.net/robertleeplummerjr/z378bnoa/29/
So in short, a deeper net and adjusting training settings should put you where you want to be.

Thanks for the reply! Can you help me understand how the hidden layers add more value? That just means essentially that it's testing more things throughout the iterations to find an "optimal" solution, right?
I'm running a test with actual cleansed data (not the test inputs), and am currently at 4100 iterations and running since I started after your last message yesterday. Training error has been at "Infinity" for some time now, so seems like it's struggling to find correlations?
Appreciate the continued help -- excited to be learning with Brain.js! Thanks again for all your help with the project!!

Hidden layers are like abstractions of ideas.  The more hidden layers, the more the next can abstract.  I think the net doesn't understand what you are wanting it to do.  Try giving it more layers, (like in the example I included) and see if that helps.


VM that extends BaseI18N failing to load as dialog VM

Example VM to be loaded into modal
@autoinject
export class UserSettings extends BaseI18N
{
    public user: IUserOwn;

    constructor(
        private _i18n: I18N,
        private _ea: EventAggregator,
        public controller: DialogController)
    {
        super(_i18n, Element, _ea);
    }

    activate(user: IUserOwn)
    {
        this.user = user;
    }
}

When attempting to use UserSettings as the viewModel for a dialog, the following error occurs:
i18n.js:136 Uncaught (in promise) TypeError: el.querySelectorAll is not a function
    at I18N.updateTranslations (http://localhost:52058/jspm_packages/npm/aurelia-i18n@0.5.3/i18n.js:136:22)
    at UserSettings.attached (http://localhost:52058/jspm_packages/npm/aurelia-i18n@0.5.3/base-i18n.js:33:17)
    at Controller.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:2947:24)
    at View.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:1016:25)
    at ViewSlot.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:1472:15)
    at DialogController.dialogController.showDialog (http://localhost:52058/jspm_packages/npm/aurelia-dialog@0.6.1/renderers/dialog-renderer.js:114:31)
    at eval (http://localhost:52058/jspm_packages/npm/aurelia-dialog@0.6.1/renderers/dialog-renderer.js:70:35)

I think this is just an oversight on our part - we aren't registering the
viewslot element with the child container. I'll mark this as a bug.
2016-05-18 15:01 GMT-04:00 Stefan Buys<EMAIL_ADDRESS>
Example VM to be loaded into modal
@autoinject
export class UserSettings extends BaseI18N
{
public user: IUserOwn;
constructor(
    private _i18n: I18N,
    private _ea: EventAggregator,
    public controller: DialogController)
{
    super(_i18n, Element, _ea);
}

activate(user: IUserOwn)
{
    this.user = user;
}

}
When attempting to use UserSettings as the viewModel for a dialog, the
following error occurs:
i18n.js:136 Uncaught (in promise) TypeError: el.querySelectorAll is not a function
at I18N.updateTranslations (http://localhost:52058/jspm_packages/npm/aurelia-i18n@0.5.3/i18n.js:136:22)
at UserSettings.attached (http://localhost:52058/jspm_packages/npm/aurelia-i18n@0.5.3/base-i18n.js:33:17)
at Controller.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:2947:24)
at View.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:1016:25)
at ViewSlot.attached (http://localhost:52058/jspm_packages/npm/aurelia-templating@1.0.0-beta.1.2.7/aurelia-templating.js:1472:15)
at DialogController.dialogController.showDialog (http://localhost:52058/jspm_packages/npm/aurelia-dialog@0.6.1/renderers/dialog-renderer.js:114:31)
at eval (http://localhost:52058/jspm_packages/npm/aurelia-dialog@0.6.1/renderers/dialog-renderer.js:70:35)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/aurelia/dialog/issues/150

This is actually related to https://github.com/aurelia/dialog/issues/15 and I believe it's the same issue. Due to the refactors we've done we can resolve this.


2.4.x

I suggest closing this PR. The main problem here is that programmatic use should not be adressed by adding CLI options which make no sense. Also, if resolve() ends with errors, it's best to throw an exception encapsulating whatever relevant information is available, rather than blithely carrying on.

I think it would be great "not" to have an additional CLI option, but then you cannot have a System.exit(1) terminating the client code (https://github.com/apache/ant-ivy/blob/2.4.x/src/java/org/apache/ivy/Main.java#L313) just because of an error.
The errors (and the successes) can be queried out of the ResolveReport.  A client can programmatically make decisions based on the data returned. And the run() methods still throw their respective exceptions which could be caught by the client consumer.
The new option is to alleviate the problematic design of having an error terminate a client consumer  with a System.exit.
Perhaps a design would be to copy/paste Main.java to IvyClient.java and remove the System.exit - but then it would be less normalized, which has its own flaws.

I also spam this pull request. (why not open issue tracker here?)
Ivy CLI says: -types                comma separated list of accepted artifact types
java -jar ivy-2.4.0.jar -cache ./ivy -types source,jar ...
does NOT work. What does work is a white space separated list of artifacts.

Issue tracker for the project is JIRA. Apache has own Git infrastructure, GitHub is use to increase exposure and as easy means to suggest code changes (rather than uploading patches).
Please check the latest Ivy build. I cannot verify your report right now, but the value of types should be split correctly in master.

@supertick FYI, Ivy 2.5.0 RC is on the way, @nlalevee has kindly taken the role of release manager. Your latest commits are not merged to master, though. If you are interested in using 2.5.0, please let us know.

Check out the nightlies
I was commenting here because @supertick is apparently pushing changes to 2.4.x release branch, but the RC is from the master.


dynamic grouping error

Describe the Bug
Hello. Grouping error at https://reactflow.dev/docs/examples/nodes/dynamic-grouping/
I made a video, everything is clearer there.
Your Example Website or App
No response
Steps to Reproduce the Bug or Issue
At the beginning I add blocks and then a group block. Then I move these blocks to the group block and when I click on the block on the last one, the menu does not appear where it is needed. And when I delete the block, a general error occurs because of this.
Expected behavior
In my video, when I click on the last block, the menu should appear above it. This bug is in chrome and opera. I couldn't replicate the error in firefox.
Screenshots or Videos
recording.webm
Platform

OS: [Windows]
Browser: [Chrome]
Version: [108.0.5359.125]

Additional context
No response

thanks @lexoring! It was an issue with the node sorting. I just fixed it.


:warning: Replace Positive and Negative outcomes with True and False

What kind of change does this PR introduce?
structured result breaking change

[X] PR title follows the guidelines defined in our pull request documentation

What is the current behavior?
Outcomes use Positive and Negative which is a confusing mix of both safe/unsafe and yes/no
What is the new behavior (if this is a feature change)?**
Outcomes use True / False which is a less ambiguous equivalent of yes/no

[ ] Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes
Fixes #3866 (see https://github.com/ossf/scorecard/issues/3866#issuecomment-2043506236)

Special notes for your reviewer
Does this PR introduce a user-facing change?
For user-facing changes, please add a concise, human-readable release note to
the release-note
(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

NONE

/scdiff generate Dangerous-Workflow,Fuzzing,SAST,Security-Policy,Packaging,Signed-Releases,Branch-Protection,Token-Permissions,CI-Tests,Code-Review,Contributors,Binary-Artifacts,Maintained,Pinned-Dependencies,CII-Best-Practices,Dependency-Update-Tool,License,Vulnerabilities

The current change is a find/replace for Positive -> True and Negative -> False. All of the behavior changes will be in a follow-up PR.
The unit and e2e tests pass, and the scdiff was a match.
I will highlight a few important places:
The new definitions / documentation for OutcomeTrue and OutcomeFalse:
https://github.com/ossf/scorecard/blob/17d84c4d7ccb43e7f489d0d35787237f4d55a6e9/finding/finding.go#L60-L70
We currently only surface remediation data when the probe is negative. So this will likely need to go away in a followup.
https://github.com/ossf/scorecard/blob/17d84c4d7ccb43e7f489d0d35787237f4d55a6e9/finding/finding.go#L220-L232
While some of the probes are phrased so OutcomeNegative corresponded to a no, hasDangerousWorkflowUntrustedCheckout wasn't and will need to be fixed
https://github.com/ossf/scorecard/blob/17d84c4d7ccb43e7f489d0d35787237f4d55a6e9/probes/hasDangerousWorkflowUntrustedCheckout/impl.go#L57-L62


Fix interleaveWithNodeStream streaming flow to be compatible with React 18 api

As mentioned here - https://github.com/styled-components/styled-components/issues/3658
And here - https://github.com/styled-components/styled-components/issues/4211
New React "renderToPipeableStream" stream api decides how much HTML to send by itself and it’s not a fact that it can be valid, for example, the middle of the svg path, large base64 image or just a long text.
So in this pr I suggest you a solution to work compatible with the new react streaming api.

Thank you for working on this, I have no practical test bed for streaming testing at this point but I'd be happy to post a test release under a tag for people to try and report back

Published under styled-components@react-18-streaming-prototype for those interested in testing

Oh, i really forgot about this pr. It's really unstable now.
But if you don't mind i can continue work on it in the next month.
The main problem now is that it will trigger hydration issue without any scripts which will move styles outside the root react component. We should somehow give users way to control how should it work.

Of course! That's why it's a test build 👍

Hi @ignatiqq,
I wrote a Transform stream implementation for handling server-side rendering of styled-components based on this PR. I noticed a couple issues with this PR due to updating the React 16 streaming approach to work with React 18 streaming:

Original CLOSING_TAG regex is invalid

This regular expression only checks if the chunk begins with a closing tag (which works for React 16 streaming), not if the closing tag exists anywhere in the chunk

Search for end of closing tag position is incorrect

Current approach looks for first '>' character in the chunk, which may be part of a starting tag rather than a closing tag

Need a final method to flush the styles buffer of any remaining style tags before stream closure

I can submit my code as a separate PR, but these are the only three required changes to patch the bugs I noticed.

@burnsdy You are right. This code is buggy. But the main reason i do not push any commits here. Becuase i find problem which i cant solve by myself here. The problem related to SSR error during new react 18 stream api. It means React will start hydrate page partially and we cant remove all "style" tags from server.
After reading the instructions from React maintainers and paying attention to other CSS in JS libraries, the only solution that I could come up with at the moment is to use a script that moves the “style” tag behind the “root” component. But this solution is not perfect and has many trade-offs, such as: performance (both in the case of multiple tag styles, if there are not many "Suspense" on the page, and if there are many "Suspense" borders on the page, we will need to execute the script after processing the css every time and process it using the sc.rehydrate function)
@quantizor What do you think about that?

@burnsdy You are right. This code is buggy. But the main reason i do not push any commits here. Becuase i find problem which i cant solve by myself here. The problem related to SSR error during new react 18 stream api. It means React will start hydrate page partially and we cant remove all "style" tags from server.
After reading the instructions from React maintainers and paying attention to other CSS in JS libraries, the only solution that I could come up with at the moment is to use a script that moves the “style” tag behind the “root” component. But this solution is not perfect and has many trade-offs, such as: performance (both in the case of multiple tag styles, if there are not many "Suspense" on the page, and if there are many "Suspense" borders on the page, we will need to execute the script after processing the css every time and process it using the sc.rehydrate function) @quantizor What do you think about that?

Do you have any solution for this yet @ignatiqq?
Would you mind sharing your code that moves streamed (after first hydration) to top here?

@ignatiqq @quantizor https://github.com/styled-components/styled-components/pull/4327 should fix the empty styles and streaming seems to be mostly broken because of those
We still have rehydration issue on chunk that streamed in after initial hydration


bug: volume's device path cannot be found for volumes that are reported to be attached

What happened:

Occasionally, NodeStageVolume repetitively fails without any recovery sequence due to nodeserver not being able to find the volume's device path with the given lun.
CloudProvisioner.PublishVolume reports success and Azure Portal also lists the volume under given node's list of attached volumes, but volume cannot be mounted due to the abovementioned cause.
Also, it has been observed that detaching and reattaching the volume from/to node fixes the issue.

What you expected to happen:

We expect to be able to identify volume's device path if the volume is reported successfully attached to node.

How to reproduce it:

Such behavior has been observed in scale tests and pod failover tests.

Anything else we need to know?:
Environment:

CSI Driver version:
Kubernetes version (use kubectl version):
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

@sunpa93 Is there anything that needs to be done here since the workaround was implemented in #218?

No, I think we have done what we can on the driver side. I left the issue open so that we can track CRP issue and close upon its fix.

We can go ahead and close this one. The issue has been identified and we have a commitment from CRP for a fix. There's no further action we need to take here.


estimate: Fix recursion and improve var naming

The fix/functional change is in lines 135 and following: The recursion did nothing before, as visit is the default (noop) function at that point. You need a separate function variable to get actual recursion.
The rest is just variable renaming (took too many brain cycles to understand what's going on there ;) ) and filtering unrelated forward dependencies in the first loop.

The fix/functional change is in lines 135 and following: The recursion did nothing before, as visit is the default (noop) function at that point. You need a separate function variable to get actual recursion.

I don't understand this. I write a poc code, it seems to work. https://play.golang.org/p/P5GUXzQ1jOi

You are clearly right, I don't understand what I was thinking back then. I remember having an example, but it clearly must have been wrong. The variable in the closure is evaluated when called, so it isn't a problem at all. Sorry for wasting your time (if you would like to keep the naming changes, let me know and I'll remove the bogus stuff).


Custom error message for cli

How to use 'say' function is wrong.
When building an environment with docker etc, it is difficult to understand the error message in an environment where git is not installed.

thanks!


fix(slp-token-media): Pushing update

:tada: This PR is included in version 2.4.3 :tada:
Your semantic-release bot :package::rocket:


2024/07/02: Jenkins regular patching cycle - update to  2.452.2

Process to follow: https://github.com/adoptium/infrastructure/blob/master/README.md#jenkins
Current version: 2.440.3
New version:  2.452.2
Jenkins event calendar
Previous patching issue: https://github.com/adoptium/infrastructure/issues/3548
Notes:
Part of https://github.com/adoptium/infrastructure/issues/3380

Previous plugin versions:
pluginVersions.txt.gz

All upgraded except for the thinbackup plugin which is changing it's configuration file names so we'll save that upgrade until after everything else is verified as ok.


Build OpenSearch 1.1 branch in CI

opensearch-project/security-dashboards-plugin pull request intake form
Please provide as much details as possible to get feedback/acceptance on your PR quickly

Category: (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)
Maintenance

Github Issue # or road-map entry, if available:

Description of changes:

Why these changes are required?

What is the old behavior before changes and new behavior after changes? (Please add any example/logs/screen-shot if available)

Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)

TO-DOs, if any: (Please describe pending items and provide Github issues# for each of them)

Is it backport from main branch? (If yes, please add backport PR # and commits #)

By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #827 (96522ad) into main (e867e91) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 96522ad differs from pull request most recent head 38edbcb. Consider uploading reports for the commit 38edbcb to get more accurate results

@@           Coverage Diff           @@
##             main     #827   +/-   ##
=======================================
  Coverage   71.98%   71.98%
=======================================
  Files          87       87
  Lines        1906     1906
  Branches      242      242
=======================================
  Hits         1372     1372
  Misses        480      480
  Partials       54       54

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e867e91...38edbcb. Read the comment docs.


Write an Analyzer

The compiler supports only a subset of the C#, so there are many language features that will never be fully supported. It would be helpful to have an analyzer point out in the IDE if a user tries to use an unsupported feature.

Figured out one way to build a custom CoreLib: https://github.com/Yttrmin/CustomCoreLibExample
I think it would be much simpler to build against a custom CoreLib that only exposes what is actually available, versus writing an analyzer to accomplish the same thing.
I don't yet know how well Roslyn will deal with a custom CoreLib though. Hopefully very well.

The custom CoreLib works great when it's the only library your project references. Referencing a second, normal, library though completely breaks it.
This is a problem if we want to share types like AssemblyLine between the compiler (where it has to actually execute), and the game projects (where it doesn't have to execute).
At this point going with an analyzer is probably best.


elli_http:chunk_loop/1 cannot match message {tcp_closed, Socket} in its receive block

Around line https://github.com/knutin/elli/blob/master/src/elli_http.erl#L258
we have
...
chunk_loop(Socket) ->
    receive
        {tcp_closed, Socket} ->
            {error, client_closed};
....

Where Socket is a tuple of type
-type socket() :: {plain, inet:socket()} | {ssl, ssl:sslsocket()}.

but the message being received is of the form { tcp_closed, X:socket()}.
The fix is easy, I can provide a pr.
I found this while sketching an Elixir Plug Adapter for Elli, does anyone knows if an "official" adapter
is under development.
Cheers,
Andrea

Thanks for catching this! I would be very happy to accept a Pull Request.
There's at least two projects wrapping elli in Elixir, but I don't know anything about development of a plug adapter except what you'll find on Google.
https://github.com/mme/ellihandler
https://github.com/pigmej/exelli

Ok, yes I saw exelli.
Anyway you have pr #89

I have been working on an alternative to plug. https://github.com/CrowdHailer/raxx which might be of interest on the subject of elixir adapters for elli.
Currently I have reached an impasse. Is it possible to get arbitrary messages out of the request processes mailbox? the equivalent of a handle_info on the request handler. I have been looking through the source code but can't see any way. I think i need something like the following in the chunk loop
chunk_loop(Socket) ->
    receive
        {tcp_closed, Socket} ->
            {error, client_closed};
        other ->
            Mod.handle_info(other)


Add dev tool for ruff

Short description

This PR adds a dev tool for ruff
Proposed changes

Add dev tool for ruff

Side effects

At the moment we use ruff still in parallel to the other formatters and linters. AFAIK over time we want to move towards using only one?
Yet only check is enabled also with fix as extra parameter. Maybe over time we want to include formatting and also include the option to pass arguments such as --fix manually

Resolved issues

Fixes: One part of #2442

Pull Request Review Guidelines

Hmm, as described in #7242, I think adding more dev tools to perform tasks that already can be done by one pre-commit command is kind of superfluous...

Okay. Do you suggest to close this then? :)

Hmm, as described in #7242, I think adding more dev tools to perform tasks that already can be done by one pre-commit command is kind of superfluous...

Invoking the pre-commit manually in order to check for things you need to fix is a bit burdensome though, + the same could be said for most existing dev tools

Okay. Do you suggest to close this then? :)

Yes, I personally would suggest that, but I'm open for other opinions.

Invoking the pre-commit manually in order to check for things you need to fix is a bit burdensome though, + the same could be said for most existing dev tools

Hmm, I'd say that
pre-commit run ruff

is not much longer than:
./tools/ruff.sh

If you still think it's inconvenient, we could set bash aliases for this command rather than separate bash scripts?

My two cents: I basically ignored the pre-commit hooks entirely, always running the code style tools individually or as code_style.sh, because it took way too long and got in the way especially during rebases, if I remember correctly. So I at least got very accustomed to tools/. But my opinion on the pre-commit hooks might very well change once rust fully replaces most tools.

Hmm, I'd say that
pre-commit run ruff

is not much longer than:
./tools/ruff.sh

True, but then why not do the same for eslint, prettier, black, isort,...?
I'd wager most of us are used to using the dev tools for these tasks, and I am having a little trouble seeing the benefit in changing this workflow.

My two cents: I basically ignored the pre-commit hooks entirely, always running the code style tools individually or as code_style.sh, because it took way too long and got in the way especially during rebases, if I remember correctly. So I at least got very accustomed to tools/.

👍🏼

I think we can just merge this, tools/ became kind of the default location I look for, well, tools like that. If it's not in a script, I just straight up assume it's not set up for the project.
If we organize this differently in the future, I guess it makes little difference whether to also have to delete this ruff.sh or not.


NullPointerException when calling ManageSieveClient#isConnected before ManageSieveClient#connect

I'm building a simple UI to edit my sieve scripts and to avoid opening too many connections to the server, I want to check if the client is already/still connected. But if the client is not yet connected or the underlying socket was closed, I get a NullPointerException and I have to guard against this by doing:
    try {
        if (client.isConnected()) return;
    } catch(NullPointerException e) {
        // this is fine if we are not yet connected
    }
    ManageSieveResponse resp1 = client.connect(host, port);
    ...

If ManageSieveClient.java#L171 would check if the socket exists (like ManageSieveClient.java#L171), the NPE could be prevented and I could check if the connection is still open.

Nice catch, thank you. I'll get an updated version uploaded later today.


Introduce pydantic model for channels

A first version of this PR is ready, but I'll need to review it a bit more carefully.
Also, I'll have to double check it as part of https://github.com/fractal-analytics-platform/fractal-demos/pull/54.
If this all goes smoothly, then a bunch of related issues should be "easy" to address (#414, #412, #404, #401).


Nudge the placeholder ribbon left 2px, past the border

To my eye, the ribbon should fold against the outside edge of the border.
Before:

After:

(The "In Use" ribbon on artwork looks right because it's placed on a higher element, before the border is applied.)

Good catch. But let's tweak the CSS a little - we want the .ribbon to be reusable and the exception case is the placeholder, which has a thicker border. So we should only adjust the position for placeholders, not every single ribbon. (For example, in case in the future we want to add a ribbon to a regular ebook?)

Thanks for explaining that, and I see what you mean. I changed the CSS to apply only to placeholders.
(For fun, I tried to add ribbons to regular ebooks, but they are still missing some of the styling that placeholders have. So in the future there would still be more work for regular ebooks, but these placeholder rules won't get in the way.)

Thanks!


More permissions

This includes the branch from #323 but goes a step further and uses the configured attributes as defaults for the consul_definition and consul_watch resources. I made this a separate pull request since it wasn't something we discussed in #322.

You're going to need to rebase

Thanks @johnbellone, rebased and updated.


Feature/online news

Author: Tobias Ritter<EMAIL_ADDRESS>on behalf of MBition GmbH.
Addition of Data Loader for the Online News Popularity Data Set
The program was tested solely for our own use cases, which might differ from yours.
The submission is provided under the main project license (LICENSE file in root of project).
Provider Information

Hi @gunnar-mb,
since this is my first public Pull Request, I wanted to know if this adheres to the common process. If everything looks fine to you, I will not bother you for future Pull Requests anymore.
Thank you very much for your support!
Best regards
Tobias

The format of the PR text looks fine.

If I were you, I would squash these commits during the merge to main branch (maybe you plan to...) since the second one is just fixing the first.
I think it is possible to write more descriptive commit messages, and if you want to I can give you some tips, but since you are the maintainer it is really your policy that applies 😉
Also, it doesn't hurt to have someone review the commit content, so if you don't have another colleague doing that, you can also ask me.


Paginate releases. MVP.

MVP. Needs tests.

Aside from those comments, and the failing tests, this is great!

Amazing. :shipit:!


Privacy settings

Then we lose zsh autocompletion features for gpg if include 2 --options file arguments
Enabling it manuallyis too much work for something custom.


Handle edge case when ka = 0

When this occurs, velocity can be treated as a model input to avoid
numerical instability. The feedback gains produced appear to be what the
original model converges to as ka approaches zero.
With the alternate model, kp = kv * K[0, 0] is used because the gain
produced by LQR is for velocity. We can use the feedforward equation
u = kv * v to convert velocity to voltage. kd = 0 because velocity
was an input; we don't need feedback control to command it.

You'll have to manually increment the setup.py version number; other than that this looks good


perf(ring_outlier_filter): a cache friendly impl

Description
This is a "cache friendly" reimplementation of ring outlier filter. See https://github.com/autowarefoundation/autoware.universe/issues/3269 for more details.
The code is based on @sykwer WiP improvements (see https://github.com/autowarefoundation/autoware.universe/pull/3014), which I rewrote to improve memory access (hopefully).
This implementation produce slightly different output than the original implementation:

output points are in same order than input. In the original implementation, points are reordered and copied by ring.
Fixes https://github.com/autowarefoundation/autoware.universe/issues/3217
Fixes https://github.com/autowarefoundation/autoware.universe/issues/3218

Related links
https://github.com/autowarefoundation/autoware.universe/issues/3269
https://github.com/autowarefoundation/autoware.universe/pull/3014
Tests performed
Note: these tests are fairly old and refer to older implementation of ring outlier filters. See recent comments for comparisons between the current implementation and this PR.
Simple test on a few random rosbags. Plot data is the output of /ring_outlier_filter/debug/processing_time_ms. In blue the current implementation (with hashmap), in orange the proposed implementation (see the more recent comments for more accurate benchmarks):

On VLP-16 data (~20000 points per message):

On VLP-32 data (~50000 points per message):

In both cases, my implementation is roughly 3 times faster than the original one. Note that I am not using the TLSF allocator.
This is of course just a very simple benchmark and I will need to test more. In particular, I want to:

compare cache performance with old implementation
compare performance with @sykwer implementation (after his changes have been merged)
compare performance on vehicle, with all autoware modules running

I don't necessarily expect this implementation to shine when the filter is the only process running on the machine. However I expect it will perform way better when the machine is stressed.
Notes for reviewers

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.
[ ] The PR has been properly tested.
[ ] The PR has been reviewed by the code owners.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.
[ ] The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Using the setup described here https://github.com/autowarefoundation/autoware.universe/issues/3269#issuecomment-1499897019, I can confirm this implementation is "cache friendly".
Compared to the initial implementation, cache-miss share of RingOutlierFilterCompoment dropped from 12.59% to 4.40% (count of cache miss also dropped a lot):
Samples: 8K of event 'cache-misses', Event count (approx.): 158292133
Overhead  Command          Shared Object                                              Symbol
  32.79%  component_conta  libc.so.6                                                  [.] __memset_avx2_unaligned_erms                                                                    ◆
   8.03%  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::CropBoxFilterComponent::faster_filter                                  ▒
   6.84%  component_conta  libc.so.6                                                  [.] __memmove_avx_unaligned_erms                                                                    ▒
   6.29%  component_conta  libc.so.6                                                  [.] pthread_mutex_lock@@GLIBC_2.2.5                                                                 ▒
   4.40%  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter                              ▒
   3.73%  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::PointCloudConcatenateDataSynchronizerComponent::convertToXYZICloud     ▒

The biggest source of cache miss in the filter is now a data copy (accessed linearly):
Biggest cache miss is on a memcpy:
       │      output_size += sizeof(PointXYZI);                                                                                ▒
  0.52 │        add      $0x14,%r9                                                                                ▒
       │      memcpy():                                                                                ▒
       │        movups   %xmm5,(%rdx)                                                                                ▒
  4.35 │        mov      0x10(%rax),%ecx                                                                                ▒
  3.20 │        mov      %ecx,0x10(%rdx)                                                                                ▒
       │      pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter(std::shared_ptr<sensor_msgs::msg::PointCloud2_<std::allocator<void> > const> const&, std::shared_ptr<std:▒

One big hint the code is more cache friendly is that there is no big source of cache miss anymore.

I have tested and benchmarked a few variations. In particular I benchmarked my impl against 82262b9.

I tried a variation using a "fast isCluster" implementation. However the performance is mostly the same than the accurate version. I guess most processing time is spent on moving data rather than some math.
I tested a version with a fix for #3218. The fix has no significant performance impact.

The benchmark setup is described here. CPU clock is locked 3GHz, and I am using the same VLP-32 rosbag data for all.

| original | 82262b9 | proposal | w/ fast isCluster | w/ fix 3218
-- | -- | -- | -- | -- | --
average (ms) | 2.708554 | 0.787365 | 0.758712 | 0.741636 | 0.737616
stdev | 0.227331 | 0.06549 | 0.047685 | 0.044937 | 0.048045
All variations have approximately the same performance. The proposed implementation is more stable (but not by a huge amount) The big difference is that 82262b9 cache miss count is roughly twice as big the proposed implementation.
Using perf to monitor the process over 120 seconds each. I measured:

82262b9: 393 caches misses in pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter
proposed implementation: only 107 cache misses

So even though the individual performance is roughly the same, there is a big difference in term of cache performance (which impacts the performance of other processes).

Rebased the code, then benchmarked again following protocol described here https://github.com/autowarefoundation/autoware.universe/issues/3269#issuecomment-1499897019
The rosbag used for benchmark this time is one of autoware's demo rosbag , whose pointcloud data scan contains each ~218000 points (from a VLP 128 I guess?). It is pretty short though (~30 seconds).

cache friendly PR
current

average (ms)
4.26097894736842
6.2111043956044

stddev
0.756442711825567
1.58342325326683

This rosbag is maybe a bit unfair for the current implementation, because the data is really heavier.
Cache-misses on current implementation:
Samples: 28K of event 'cache-misses', Event count (approx.): 829442212
Overhead       Samples  Command          Shared Object                                              Symbol
  18.78%          3246  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter
  14.19%          2658  component_conta  libc.so.6                                                  [.] __memset_avx2_unaligned_erms
  11.17%          1595  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::CropBoxFilterComponent::faster_filter
   8.76%          1829  component_conta  libc.so.6                                                  [.] __memmove_avx_unaligned_erms
   4.46%          1418  component_conta  libc.so.6                                                  [.] pthread_mutex_lock@@GLIBC_2.2.5
   2.49%           374  component_conta  libvelodyne_rawdata.so                                     [.] velodyne_rawdata::RawData::unpack_vls128

Same as my previous report, the float azimuth_diff = next_azimuth - current_azimuth line is responsible for mosts cache-misses:
       │      walk_last_idx = idx;                                                                                ▒
  0.18 │        mov      %esi,%r11d                                                                                ▒
  0.67 │        mov      %esi,-0x8e8(%rbp)                                                                                ▒
       │      float azimuth_diff = next_azimuth - current_azimuth;                                                                                ▒
  0.09 │        lea      0x0(%r13,%rdi,1),%rdx                                                                                ◆
  0.12 │        movss    (%rdx,%r12,1),%xmm5                                                                                ▒
 36.99 │        subss    0x0(%r13,%rcx,1),%xmm5                                                                                ▒
       │      azimuth_diff = azimuth_diff < 0.f ? azimuth_diff + 36000.f : azimuth_diff;                                                                                ▒
  2.93 │        comiss   %xmm5,%xmm2                                                                                ▒
  0.62 │      ↓ jbe      7e6                                                                                ▒
       │        addss    %xmm4,%xmm5

Now with the current PR, the number of cache-miss in RingOutlierFilterComponent::faster_filter drops a lot:
Samples: 25K of event 'cache-misses', Event count (approx.): 794706274
Overhead       Samples  Command          Shared Object                                              Symbol
  16.73%          2925  component_conta  libc.so.6                                                  [.] __memset_avx2_unaligned_erms
  12.55%          1703  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::CropBoxFilterComponent::faster_filter
  10.78%          1979  component_conta  libc.so.6                                                  [.] __memmove_avx_unaligned_erms
  10.13%          1888  component_conta  libpointcloud_preprocessor_filter.so                       [.] pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter
   4.32%          1233  component_conta  libc.so.6                                                  [.] pthread_mutex_lock@@GLIBC_2.2.5
   2.84%           379  component_conta  libvelodyne_rawdata.so                                     [.] velodyne_rawdata::RawData::unpack_vls128

It is unclear where exactly is the biggest source of cache miss, I feel it is in one of the lambda helpers (e.g. getPointAzimuth()):
       │      pointcloud_preprocessor::RingOutlierFilterComponent::faster_filter(std::shared_ptr<sensor_msgs::msg::PointCloud2_<std::allocator<void> > const> const&, std::shared_ptr<std::vector<int, std::allocator<int> > > const&, senso▒
       │      ↓ je       1320                                                                                ▒
  0.41 │        mov      -0x9d8(%rbp),%rax                                                                                ▒
  0.73 │        mov      0x8(%rdx),%ecx                                                                                ▒
 10.77 │        mov      (%rdx),%rbx                                                                                ▒
  0.34 │        mov      (%rdx,%rax,1),%eax                                                                                ▒
  6.12 │        shl      $0x20,%rax                                                                                ▒
  1.03 │        or       %rcx,%rax                                                                                ▒
  1.97 │        mov      -0x9c0(%rbp),%rcx                                                                                ▒

Note that these functions could be made faster if Autoware enforced PointCloud2 data field format (see https://github.com/autowarefoundation/autoware.universe/pull/2618#issuecomment-1499824261).
The output data "looks ok", but I don't have any real test. I don't have many rosbag either, so any help would be appreciated.

Are performance improvements measured without the influence of the operating system? It not so, could you please refer to this document?
https://autowarefoundation.github.io/autoware-documentation/main/how-to-guides/integrating-autoware/tuning-parameters-and-performance/evaluating-real-time-performance/#performance-measurement

@sykwer Very interesting reading. https://github.com/sykwer/ros2_single_node_replayer and https://github.com/sykwer/pmu_analyzer look very nice.
I let you guess have not used these ressources for my setup ^^.
What I have done to reduce interferences:

only launch the pointcloud preprocessing pipeline (from the sample sensor kit)
only run rosbag and perf
used fixed CPU clock speed

So very far from compiling a custom kernel.
Regarding node isolation, I am not sure it makes sense in this case. In the original issue, I pointed out that cache performance was most likely an issue with the filter implementation. In a first attempt to quantify the problem, I used a rosbag to feed data directly to the filter. Doing so blows up the cache-misses count, because the input data is never in the cache. Even though cache access is made more efficient, it is difficult to see any improvement with this setup.
On the other hand, when Autoware is running, it is likely the input data is hot in the cache when the filter tries to access it. This is why I decided to run the whole preprocessing pipeline for my benchmark (the last 2 components could have worked as well). Not only that, but inefficient cache usage is most impactful when many nodes are running: when a cache line is invalidated, many processes are impacted, not just the one responsible for the cache miss. As I reported in the original issue, simply skipping the ring outlier filter on our vehicle reduced htop cpu usage by 20%... yet the filter was not using 20% of the cpu itself.
In any case, it seems that this PR is currently broken (from the last few rebase, but after my last benchmark). The ring index is not read properly anymore (or input data is invalid) and my code reports:
[component_container_mt-1] [WARN<PHONE_NUMBER>.418574920] [sensing.lidar.top.ring_outlier_filter]: 59649 points had ring index over max_rings_num (128) and have been ignored. (faster_filter() at /home/sig/autoware/src/universe/autoware.universe/sensing/pointcloud_preprocessor/src/outlier_filter/ring_outlier_filter_nodelet.cpp:350)

I need to investigate that.

@VRichardJP I've set it to draft since you've reported that it stopped working correctly after the last rebase.

@xmfcx Sorry I don't have much time to work on Autoware recently.
During my recent tests however, I have observed the ring field of ring outlier filter input cloud is filled with garbage data. Maybe I got unlucky in my rebase, but I don't think this was related to this PR.

@VRichardJP I would like to troubleshoot the garbage data issue, and verify and complete the pull request if that is okay with you!

@mojomex Sorry, I really can't find much free time to work on Autoware recently.
I have pretty much pushed everything and documented all the details, so feel free to take over this PR.


wrong cpu load

here is what I see in grafana

and here is what i see in proxmox webui

VMs CPU usage wrong too
pve-exporter 3.0.2, pve 8

Just change the CPU history widget to have a percentage 0-1.0 value with a softmax and max of 1.0; then on the Current CPU widget, change the value to "pve_cpu_usage_ratio{instance="$instance"} and on(id) pve_node_info".

Thank you! Works like a charm =)

Sorry, but i also need to fix Guest CPU Usage. Real value is ~30% in grafana i see ~6%

I'm seeing the same issue with guest CPU usage. The current query is:

((pve_cpu_usage_ratio{instance="$instance"} / pve_cpu_usage_limit) * on(id, instance) group_left(name, type) pve_guest_info) and on(id, instance) pve_up == 1

If I remove / pve_cpu_usage_limit I look to get data that matches the built-in Proxmox graphs. pve_cpu_usage_limit looks to contain the number of cores each guest is allocated, so it's artificially dividing any multi-core guest CPU load.
@dormancygrace can you confirm the same?

@dormancygrace can you confirm the same?

Yes