Migrate uptime service to AWS and support different payload revisions

This PR migrates the usage of GCP storage to AWS S3 in the uptime service app. It also changes how some values are loaded (bucket name, whitelist spreadsheet ID, credentials) and adds support for different payload revisions. The node will always send the latest version it knows of that is supported by the backend. The project structure was also changed to accommodate new subprojects.
Summary of changes:

In the node:

Small structure change to have payload definition on its module.
Include a block_producer_version field in the payload with the Git commit SHA-1.
Query backend for the latest payload version supported.

In the BPU app:

Removed unneeded migration app.
Migrate logic from GCP storage to AWS S3.
Bucket name and whitelist spreadsheet ID loaded from config file.
Change bucket directory structure.
Change project structure.
Create different payload revisions with a common interface to verify them and process them into the value saved to S3.
Create a new payload version with the block_producer_version field.
Add GET endpoint that lists the latest payload version supported.
Add and fix tests (all green).

This was tested by running the tests defined in the codebase, running a node and BPU app, and checking the sent payloads for the new information. Tested communication between different node/backend versions (including the ones live on mainnet) and they all are able to send and process payloads.
Closes #12965, #13402 and #13059.

I'm not the right person to review the delegation backend changes. I'll figure out who the right person is and ask them to review.

@georgeee maybe, he worked on this codebase. FYI @Smorci also reviewed this.

!ci-build-me

@joaosreis @Smorci will not IP fix be included in this PR? Is there a separate one?

@georgeee the fix for the IP was just merged in a separate PR (#12955)

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

IMHO i think this PR contains to many changes that in my opinion are too loosely coupled.  I can identify 4 granular changes that should be separate:

migration from GCP  to AWS S3
variables loading (bucket name, whitelist spreadsheet ID, credentials)
support for different payload revisions.
The node version awarness

Is there particular reason it is bundled in single PR?

@dkijania I agree those are the 4 main changes in this PR. We started by porting the GCP functionality to AWS, and shortly after that, we began working on everything else. We wanted to deploy and test these changes using our AWS infrastructure, so all the other functionality we implemented depended on the GCP -> AWS migration. We could have opened different PRs initially branched off from this branch, but we ended up doing it all on this one. Other than that, I believe there was no other reason to bundle everything together.

@joaosreis Ok, understand. And thanks for code improvements!
Regarding testing i see that there are unit tests for delegation backend and I understand that you perform manual integration tests. I don't want to block this work by requiring automated test for integration part, so i will be satisfied if you can create issues or paste here test steps you were performing. This will help us pay automation debt . Extra question is: i see that uptime service is loosely coupled with node.. if it possible (in the future of course) to create automated tests for uptime service -> delegation backend -> aws s3 communication with mocked node component and vice versa : node -> uptime service -> delegation backend -> local storage (instead of aws s3). Having architecture which is not too tighly coupled will help us to test better

@dkijania The steps for the integration testing I have done are the following:
Requirements: AWS and Google Spreadsheet API auth keys.

Generate a BP keypair.
Create a whitelist spreadsheet with the generated pubkey.
Run uptime service backend configured with AWS/GSpreadsheet keys and whitelist spreadsheet.
Run the BP node with the generated keypair.
Wait until blocks are produced and sent to the backend.
Observe that there are no errors in either node or backend.
Observe S3 buckets for submitted payloads.

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

I believe the plan is to backport this to compatible as well, and when it's shipped it should not be enabled by default.

!ci-build-me

!approved-for-mainnet


User unable to update IOTA verified address

discord ticket: 578
User reports repeat issue in inability to change verified IOTA address, however changing SMR address has no issues.  I think others have reported in past.
User EVM account: 0x2390731d0eDA70a349DA1fd70941475b87329eF2
These were the transactions which did work for updating the IOTA verified address: https://explorer.iota.org/mainnet/transaction/0x39bb49b55a3c5aa23fea78932ba5ed7d76668969a936f477c3c3bf1cd07f99fb
https://explorer.iota.org/mainnet/transaction/0x862aba8f0fc892ff8f62f291b7b9887c19e3a764ac612e651c7249d69a57f5af

Could you ask the user to retry, it should work now?


Fix SelectField choice backwards compatibility

Related to: https://github.com/wtforms/wtforms/issues/811
Fixes backwards compatiblity to SelectField choices to handle render_kw argument.

This patch solves so many problems with other libraries using WTF (Flask ecosystem). So please accept it as soon as possible!

I added a deprecation warning and released this under 3.1.1
Thank you for your feedbacks


update ly to unstable version and add nixos/ly module

Description of changes

Tow notes:

I updated ly to the unstable one due to the fixes of config reading.
I made a wrapper for all ly necessary files which is linked into /run/current-system/sw/ due to the line length restriction of configator used by ly.

Test method
NixOS Test
nix-build -A nixosTests.ly <your nixpkgs repo on this pr>
This is successed.
NixOS VM
An example flake is here:
{
  inputs = {
    nixpkgs.url = "github:Vonfry/nixpkgs/init-ly-module";
  };
  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { localSystem = system; };
      flakeSpecialConfig = { pkgs, ... }: {
        nix = {
          package = pkgs.nixFlakes;
          registry = {
            sys.flake = nixpkgs;
            nixpkgs.flake = nixpkgs;
            nixos.flake = nixpkgs;
          };
          channel.enable = false;
          nixPath = [ "nixos-config=${./.}" "nixpkgs=${nixpkgs}" ];
          extraOptions = ''
            flake-registry = /etc/nix/registry.json
            experimental-features = nix-command flakes
          '';
        };
      };

      nixosConfig = { ... }: {
        nixpkgs.hostPlatform = "x86_64-linux";

        users.users.alice = {
          isNormalUser = true;
          description = "Alice Foobar";
          password = "foobar";
          uid = 1000;
          extraGroups = [ "wheel" ];
        };

        i18n.defaultLocale = "en_US.UTF-8";
        time.timeZone = "Asia/Shanghai";
        console.keyMap = "dvorak-programmer";

        virtualisation.vmVariant.virtualisation.cores = 8;
        virtualisation.vmVariant.virtualisation.memorySize = 8192;
        services.xserver.enable = true;
        services.xserver.displayManager.ly.enable = true;
        services.xserver.displayManager.ly.settings = {
          load = false;
          save = false;
        };
        services.xserver.displayManager.defaultSession = "none+icewm";
        services.xserver.windowManager.icewm.enable = true;

        environment.systemPackages = with pkgs; [ vim ];
      };
      nixosOutputs = {
        nixosConfigurations.nixos-desktop = nixpkgs.lib.nixosSystem {
          modules = [
            flakeSpecialConfig
            nixosConfig
          ];
        };
      };
    in nixosOutputs // {
      devShells."${system}".default = pkgs.mkShell {
        packages = [ ];
        SHARED_DIR = "/tmp/shared";
        NIX_DISK_IMAGE = "/tmp/nixos.qcow2";
      };
    };
}

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[x] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

 The only parts config in /etc/ly/config.ini  takes effect. According to strace, the file is read successfully. However, some configured dirs are read but some aren't (the default one is used).

 NixOS tests passed and it works in a nixos vm!

Result of nixpkgs-review pr 297434 run on x86_64-linux 1

1 package blacklisted:

nixos-install-tools

1 package built:

ly

Result of nixpkgs-review pr 297434 run on x86_64-linux 1

1 package blacklisted:

nixos-install-tools

1 package built:

ly

Confirmed that the test works. The previous iteration was already working
but the success criteria wasnt right yet and the test timed out. Now it's
working nice. Thank you!
Em sex., 22 de mar. de 2024 10:36, Lucas Eduardo @.***>
escreveu:

@.**** approved this pull request.
—
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/pull/297434#pullrequestreview-1954807962,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADXXO6FRCZ26UK5RRMRLHQ3YZQXV3AVCNFSM6AAAAABE7IJRSWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNJUHAYDOOJWGI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Update:

move ncurses from systemd path to package wrapper to increase compatibility.
documentation update.

Result of nixpkgs-review pr 297434 run on x86_64-linux 1

1 package blacklisted:

nixos-install-tools

1 package built:

ly

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/3671

ping @SuperSandro2000

if there is no more problems, would someone merge this pr?

I'm not sure why a TTY display manager would required xorg, but this
is why I'm not going to use it even though I like it.

It is not an issue on ly, but nixos modules. Wayland sessions and
xsessions (desktop files) are generated in services.xserver.displayManager
module (/services/x11/display-manager/default.nix defines
options.services.xserver.displayManager.sessionData).
Without xserver, no desktop files on wayland is generated currently.

It is not an issue on ly, but nixos modules. Wayland sessions and xsessions (desktop files) are generated in services.xserver.displayManager module (/services/x11/display-manager/default.nix defines options.services.xserver.displayManager.sessionData). Without xserver, no desktop files on wayland is generated currently.

Oh, that's too bad. I like having a package set as small as possible, so I'll keep an eye out for that to be possible with ly and wayland. I'm pretty happy with tuigreet at the moment, so I'll keep using that for now.
Regarding the PR: I still think you should add the xserver requirement somewhere, either in documentation or in the module itself (with a clear comment on why it's necessary).

Oh, that's too bad. I like having a package set as small as possible, so I'll keep an eye out for that to be possible with ly and wayland. I'm pretty happy with tuigreet at the moment, so I'll keep using that for now.
Regarding the PR: I still think you should add the xserver requirement somewhere, either in documentation or in the module itself (with a clear comment on why it's necessary).

PR #291913 is working on the seperation. We can wait for that and update this pr with the new display-manager module.

The separation is merged! 🎉

Update: merge to new services.displayManager module and the nixos tests pass (only for x).

Nice, I've tested this again and now it's working without xserver enabled. Thanks!
However, I have a few warnings when rebuilding my system with it:
trace: Obsolete option `services.xserver.displayManager.autoLogin' is used. It was renamed to `services.displayManager.autoLogin'.
trace: Obsolete option `services.xserver.displayManager.sessionData' is used. It was renamed to `services.displayManager.sessionData'.

Additionally, nix-tree tells me that xorg-server and xauth are still installed on my system by ly-data-wrapper, even if services.xserver.enable = false;. I think it would be good to omit those from ly's config when xserver isn't enabled, thus limiting the amount of installed packages.
Lastly, I've found that ly isn't starting Hyprland correctly, while the same wayland-session file works fine when started from sddm. But that probably isn't caused by this PR but rather by ly itself.

trace: Obsolete option services.xserver.displayManager.autoLogin' is used. It was renamed to services.displayManager.autoLogin'.
trace: Obsolete option services.xserver.displayManager.sessionData' is used. It was renamed to services.displayManager.sessionData'.

Sorry, I forget to rename some configs. It has been updated in the new commit.

Additionally, nix-tree tells me that xorg-server and xauth are still installed on my
system by ly-data-wrapper, even if services.xserver.enable = false;. I think it
would be good to omit those from ly's config when xserver isn't enabled, thus limiting
the amount of installed packages.

I add a condition check for the default config generation, and should
avoid to install x11 packages if services.xserver.enable is false.

Lastly, I've found that ly isn't starting Hyprland correctly, while the same
wayland-session file works fine when started from sddm. But that probably isn't caused
by this PR but rather by ly itself.

I find some relative issue in hyprland and ly. I think it caused by ly itself.

This works very well now, thanks! Now I have to find out how to fix the hyprland login... :)

This works very well now, thanks! Now I have to find out how to fix the hyprland login... :)

@arjan-s I use Hyprland and it works fine for me. I'm not sure what I could be doing differently that's making it work.
I use the version of Hyprland in nixpkgs, and I am using this PR via flake-utils-plus's nixpkgs patching feature with a patch obtained by adding .patch to the end of this PR's url. Here's my config if it helps: https://github.com/Anomalocaridid/dotfiles.

@Anomalocaridid Thanks! Looking at your config helped analyzing my problem!
I know how to build my system with this PR, that works fine. Hyprland itself starts fine for me when logging in through ly, but somehow all dependencies (waybar, gnome-keyring, etc) are started too early, before Hyprland is done starting.
The difference between our configs is that I'm relying on systemd user services to start things like waybar, and you're starting your eww bar from the Hyprland config with exec-once.
I think what's happening is that ly somehow decides too early that graphical-session.target is reached, when Hyprland isn't ready for that yet. I haven't looked into the reason or solution for that yet.

Update: fix conflictions in release-notes.

This module seems to cause issues with the hypridle service from home-manager, due to its dependency on graphical-session.target. It starts the service too early, causing it to be skipped due to not meeting a requirement for a Wayland session.

What I found out about this problem is that home-manager creates a systemd user target which is bound to graphical-session.target and is started immediately at pam login due to this module setting security.pam.services.ly.startSession = true;. So that happens before Hyprland is started, which causes these issues.
I don't know how to solve it in a clean way though.

Inspired by this post I took a look at how the sddm module does it. They don't set security.pam.services.ly.startSession, so perhaps that's not necessary here either? I haven't tested removing it, but it may just resolve the issue.

Thinking about it, just adding security.pam.services.ly.startSession = lib.mkForce false; would be enough to test this theory.

Just tried it. Unfortunately, it breaks ly. When I try to log in, I get a blank screen for a few seconds and then I'm taken back to ly with the message "logged out" at the top of the username and password box.

Hmm, that's too bad. Apparently ly needs the systemd user session to be started. In that case you could experiment with the home-manager setting to disable systemd support

Hmm, that's too bad. Apparently ly needs the systemd user session to be started. In that case you could experiment with the home-manager setting to disable systemd support

Strangely, setting wayland.windowManager.hyprland.systemd.enable = false does not seem to have any noticeable effect.
I also tried working around it by setting wayland.windowManager.hyprland.systemd.extraCommands = [ "systemctl --user restart hypridle.service" ], but that did not seem to change anything either.

Update:

ly is rewritten in zig in newest commits. This pr migrate to it which fix many issues in the old implementations.
remove some wrappers in ly modules due to it is not necessary in the new implementation
tested by the nixos test (only x11)

Update: move release notes to 24.11 and rebase on master again.

I updated to try this out and I got a hash mismatch error:
building the system configuration...
error: hash mismatch in fixed-output derivation '/nix/store/ymq3zqnq2mv3szrk52gxqhzjcvbd0hgf-source.drv':
         specified: sha256-UoVZ6wG3zY7e6vl7O4cm9JOEAUJDCVEwDaTYHnYrpUY=
            got:    sha256-tlpTmtEDEriJgY6Xl9f9omnkGtVK9lwDY9u8vh9u7QA=
error: 1 dependencies of derivation '/nix/store/mxrm3d5cvmr4ph00nqcy2w5xy781qj2n-zig-packages.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hwdcxwhar01jzfwyf775mara820apzp3-ly-0.6.0-unstable-2024-05-10.drv' failed to build
error: 1 dependencies of derivation '/nix/store/biqaff8dkah23r09xclgx0baindsavsf-dbus-1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fj48jwvdcz1vkma4xpcrrpmw3cgdgz8c-ly-0.6.0-unstable-2024-05-10_fish-completions.drv' failed to build
error: 1 dependencies of derivation '/nix/store/bc9nn8dv98fm4nqja17p0j4kr1y1xlhq-man-paths.drv' failed to build
error: 1 dependencies of derivation '/nix/store/65nanvj2db8xj7iskrh2jc37x379gasy-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/6v8jyh1dcpddy5100wlja17in4x1hs30-nixos-system-home-pc-24.11pre-git.drv' failed to build

I updated to try this out and I got a hash mismatch error:

It should be fixed now. Wrong hash is used for fetchzip.

I think the module is missing a new default value for term_restore_cursor_cmd, default value.

It is added.

Update:
1. ly is rewritten in zig in newest commits. This pr migrates to it which fix many issues in the old implementation.

2. remove some wrappers in ly module because they are not necessary in the new implementation

3. tested by the nixos test (only x11)

4. rebase on master

The above issues on pam and wayland can be tested again with the new implementation. They may be fixed now?

Just tested it. Unfortunately, just like with the previous version, disabling pam for ly makes it unable to log in and disabling systemd for hyprland appears to do nothing.

I tested some more and found that other desktops don't even start. I enabled all of these:
programs.hyprland.enable = true;
services.xserver.desktopManager.xfce.enable = true;
services.xserver.windowManager.icewm.enable = true;
services.xserver.desktopManager.cinnamon.enable = true;
services.xserver.windowManager.qtile.enable = true;
services.xserver.windowManager.awesome.enable = true;

Sadly, only Hyprland worked (somewhat, see previous comment). The rest didn't start and ly just showed this error on the terminal:
/run/current-system/sw/bin/bash: line 1: add: command not found
/run/current-system/sw/bin/bash: line 1: :0: command not found

I tested some more and found that other desktops don't even start. I enabled all of these:
programs.hyprland.enable = true;
services.xserver.desktopManager.xfce.enable = true;
services.xserver.windowManager.icewm.enable = true;
services.xserver.desktopManager.cinnamon.enable = true;
services.xserver.windowManager.qtile.enable = true;
services.xserver.windowManager.awesome.enable = true;

Sadly, only Hyprland worked (somewhat, see previous comment). The rest didn't start and ly just showed this error on the terminal:
/run/current-system/sw/bin/bash: line 1: add: command not found
/run/current-system/sw/bin/bash: line 1: :0: command not found

I've tested this and I was able to hit the issue as well. The problem is with missing services.xserver.enable = true;. Once the option is enabled it seems everything is working just fine.

Oops, sorry for not answering. Yes, you're right, adding services.xserver.enable = true; fixed the error I reported.
There's still the same issue with ly starting the systemd graphical-session.target too soon. But I don't think that's an issue with this PR, so I vote to merge this.

There's still the same issue with ly starting the systemd graphical-session.target too soon. But I don't think that's an issue with this PR, so I vote to merge this.

Do we have any idea of what could be causing this or is it still yet to be determined?

Do we have any idea of what could be causing this or is it still yet to be determined?

Yes, I documented the reason in this comment: https://github.com/NixOS/nixpkgs/pull/297434#issuecomment-2113323877
We don't have a solution yet though since ly seems to require the user session to be started (tested by you).
I haven't had time myself to further dive into it, but I'm hoping I will soon.

ly 1.0.0 came out today. Can you update the pr for it? And let's push for merge after?

Is there a proper way to test this change? I am doing git cherry-pick on top of the unstable branch and using that nixpkgs commit but I couldn't get it working.
As of now the build is failing with the latest zig version, and before that when it used to compile it didn't show me ly but just the tty login shell.
I'll close this comment as outdated/resolved after a while.

As of now the build is failing with the latest zig version

The release version and previous commits cannot work on zig 0.13 which is the default version in nixpkgs master.

before that when it used to compile it didn't show me ly but just the tty login shell.

It seems that your display-manager doesn't start. Could you check your service log?

Result of nixpkgs-review pr 297434 run on x86_64-linux 1

2 packages blacklisted:

nixos-install-tools
tests.nixos-functions.nixos-test

1 package built:

ly

Functionality wise LGTM

v1.0.1 released and rebased on master.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-to-use-ly-display-manager/13753/8

https://github.com/fairyglade/ly/releases/tag/v1.0.2
Since this PR still didn't get merged... :(

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/4365

Result of nixpkgs-review pr 297434 run on x86_64-linux 1

2 packages blacklisted:

nixos-install-tools
tests.nixos-functions.nixos-test

1 package built:

ly

It has landed in nixos-unstable!
Thanks @Vonfry for this module, time to replace sddm/gdm/whatever on all my systems.

For those still following this PR, or those arriving here in the future: I finally found out why the graphical-session.target is starting too early, causing all kinds of errors with graphical tools starting before the window manager is ready.
It's caused by this line:
https://github.com/NixOS/nixpkgs/blob/fafef5049e2a7bcc36802e1ce72cd2f51d386388/nixos/modules/services/x11/display-managers/default.nix#L48
This is meant to be helping in case the user starts a window manager without built-in systemd support. However, since Hyprland is not in the list (although I think it should be), and neither is Qtile (my main), in those cases the problem described above happens.
The solution:
systemd.services.display-manager.environment.XDG_CURRENT_DESKTOP = "X-NIXOS-SYSTEMD-AWARE";

Additionally, I added the following line in my Qtile autostart script, to make sure graphical-session.target is reached after Qtile finishes starting up and the tray is available:
( sleep 1 && systemctl --user start nixos-fake-graphical-session.target ) &


pycarwings3~=0.7.1 not found on HASS 2024.8.0

System Health details
System Information

version
core-2024.8.0

installation_type
Home Assistant OS

dev
false

hassio
true

docker
true

user
root

virtualenv
false

python_version
3.12.4

os_name
Linux

os_version
6.6.31-haos-raspi

arch
aarch64

timezone
Europe/Amsterdam

config_dir
/config

Home Assistant Community Store

GitHub API
ok

GitHub Content
ok

GitHub Web
ok

GitHub API Calls Remaining
5000

Installed Version
1.34.0

Stage
running

Available Repositories
1387

Downloaded Repositories
8

Home Assistant Cloud

logged_in
false

can_reach_cert_server
ok

can_reach_cloud_auth
ok

can_reach_cloud
ok

Home Assistant Supervisor

host_os
Home Assistant OS 12.4

update_channel
stable

supervisor_version
supervisor-2024.08.0

agent_version
1.6.0

docker_version
26.1.4

disk_total
439.4 GB

disk_used
266.1 GB

healthy
true

supported
true

host_connectivity
true

supervisor_connectivity
true

ntp_synchronized
true

virtualization

board
rpi4-64

supervisor_api
ok

version_api
ok

installed_addons
Mosquitto broker (6.4.1), File editor (5.8.0), ESPHome (2024.7.3), NGINX Home Assistant SSL proxy (3.9.0), Let's Encrypt (5.0.26), Home Assistant Google Drive Backup (0.112.1), SQLite Web (4.2.0), InfluxDB (5.0.0), Samba share (12.3.2), Terminal & SSH (9.14.0)

Dashboards

dashboards
3

resources
1

views
5

mode
storage

Recorder

oldest_recorder_run
April 11, 2024 at 20:39

current_recorder_run
August 8, 2024 at 03:43

estimated_db_size
9907.01 MiB

database_engine
sqlite

database_version
3.45.3

Checklist

[X] I have enabled debug logging for my installation.
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of any previous issues..

Describe the issue
After updating to HASS 2024.8.0
Using version e33c46b
Logger: homeassistant.setup
Source: setup.py:269
First occurred: 03:43:58 (1 occurrences)
Last logged: 03:43:58

Setup failed for custom integration 'nissan_carwings': Requirements for nissan_carwings not found: ['pycarwings3~=0.7.1'].

Reproduction steps

Update to HASS 2024.8.0
View error in notification

Debug logs
On request.

Diagnostics dump
Is not there.

There is an issue in pycarwings 0.7.1 with the aiohttp version being incompatible with the version used by the current HASS. I am about to solve this issue in pycarwings3 and I am looking forward to provide a fix soon..

@calamarain a simple HASS restart should fix it! I was able to fix the issue in carwings3, a restart should pick the current patch level release (0.7.2) from PyPI.

I don't know how you did it, but it is fixed now after a restart.
I've restarted earlier today, it didn't solve the problem then.
Thanks for the response!


Error on warnings.

Fixes #1823

I see this error on Python 2. I'm just going to suppress it.


Dual Watch-related proposals

Two things related to Dual Watch that might be good to have:

Option in the menu to adjust the Dual Watch interval (how fast the switching between the two monitored VFOs happen). It can be controlled from here .
Virtual VFOs - split one VFO in two and use RSSI sampling to implement virtual Dual Watch between the two selected channels. Use menu option to pick the two channels part of the virtual VFO. Keep the regular dual watch between the two default VFOs.
That way, we can have 4 channels monitored in the same time with a minimal trade off for the switching time, because the virtual VFOs will be RSSI sampling-based instead of switching the entire VFO like in the regular scanner mode.

There isn't really two VFO's, the radio only has one receiver which is scanned between the two 'virtual' VFO's. It's not really a dual-RX radio.

I was thinking of adding some time settings in the menu, like how long to pause before resuming scanning after a signal vanishes.
I have to watch how much flash space I'm using up though, I want to add the band scope, but need to finish the AM fix first, seems it has a new bug I need to find.

There isn't really two VFO's, the radio only has one receiver which is scanned between the two 'virtual' VFO's. It's not really a dual-RX radio.

Yes, i am aware, but my idea is do further split the already virtual VFO into two more virtual VFOs, so a total of 4 channels to monitor at once in standard mode (2 VFOs x 2 virtual-virtual VFOs each). Of course, it is up to you to decide if it is worth it ans if it is useful at all.
But it will be useful to have a menu setting to tweak the interval on which Dual Watch probes the two VFOs (the place in the code i referenced above).

I'll have a think about it, bare with me.
The dual-watch scan rate is set by "dual_watch_count_toggle_10ms", I've been moving all the time settings to the top of misc.c.
The toggle time is currently 100ms between the two vfo's.

The dual-watch scan rate is set by "dual_watch_count_toggle_10ms", I've been moving all the time settings to the top of misc.c.
The toggle time is currently 100ms between the two vfo's.

Awesome! Thank you for making it easily adjustable.


Fix tx_successfully_send message

Why open source a project when there is no interest in public contribution?

@Turden why did you close this? Seems like a good contribution


Facebook API v7 FIX

If you have any problem with the new version of the facebook API just make this update to the file FacebookPages.pq
Line 13:

> scopes = {
> 	"read_insights", "pages_manage_ads", "pages_manage_metadata", "pages_read_engagement", "pages_read_user_content", "pages_manage_posts", "pages_manage_engagement"
> };

And that's it, your're ready to go.

Hello @imermetal, would you be available for help debugging?


How to set up the file metadata.json not to be generated or requested?

I generated the online website just for my own convenience for cross-platform reading, so I don’t need some SEO-related files. Can you add a setting to cancel the generation of these files to make website access faster?

The metadata will actually be used in the website itself in future versions and it is also used to enable incremental export. It actually has nothing to do with SEO. Is there a reason you don't want it?

Because I have a lot of documents, the metadata.json file is relatively large, and the request for metadata.json is not asynchronous, so when the website is opened for the first time or when the cache fails, the page will freeze until the request is completed. I tried to remove the metadata.json request myself and found that it had no effect, so I thought it was used to optimize SEO. If it has other uses in the future, can you try to request the file asynchronously?

The metadata will actually be used in the website itself in future versions and it is also used to enable incremental export. It actually has nothing to do with SEO. Is there a reason you don't want it?

Because I have a lot of documents, the metadata.json file is relatively large, and the request for metadata.json is not asynchronous, so when the website is opened for the first time or when the cache fails, the page will freeze until the request is completed. I tried to remove the metadata.json request myself and found that it had no effect, so I thought it was used to optimize SEO. If it has other uses in the future, can you try to request the file asynchronously?

It is already asynchronous, however the page needs to wait for that data before it can function properly. What I can do however, is separate the crucial data into a separate file that should be much smaller. I think that is a better way to do it anyway, thank you for the feedback.

Well, also I should be able to display the current page as well without that data. I only need that data to load embedded pages, and to allow navigation.

Well, also I should be able to display the current page as well without that data. I only need that data to load embedded pages, and to allow navigation.

It would be great if it could be implemented. Also I really appreciate your plugin, it's one of the most important reasons why I use Obsidian

Hey, can you test if this works any better in the latest beta version? I have exported extremely large vaults with it and have not experianced any loading time issues from it.


Passing LatLng object to directions.

Google maps API allows one to pass either Address, PlaceID or a LatLng object as source/destination of travel. But this isn't yet possible with this plugin:
  <directions
    draggable="true"
    travel-mode="DRIVING"
    origin="{{latlongorigin}}"
    destination="Chennai">
  </directions>

is parsed to :
  <directions draggable="true" travel-mode="DRIVING" origin="  {'lat':41.850033,'lng':-87.65005229999997}" destination="Chennai">

where $scope.latlongorigin = new google.maps.LatLng(41.850033, -87.6500523);

Please add a plunker example for this

Closing for inactivity

I just add the same problem. If you have fixed it can you please help me @shubhamsizzles
Thank you very much!


Jellyfin internal subtitles are not detected

What is the issue
Jellyfin internal subtitles are not detected
What do you do to make it happen
On both Jellyfin 10.9.11 and 10.10.3, load a video with internal subtitles (automatically loaded from the .srt file which is in the same directory as the video file). Confirm that the expected subtitle is being used from the CC menu, and if not, select it. I have "Auto-load detected subtitles", so the internal subtitles should be loaded automatically. From there:

asbplayer's subtitle view won't start up.
the side panel shows "Load subtitles onto a video to start using asbplayer."
Clicking "Load Subtitles" from the asbplayer right-click menu opens the "Select Subtitles" modal with all three "Subtitle Track" dropdowns showing "Empty" and containing no options.
Jellyfin plays with regular .srt subtitles showing, not asbplayer's.

Loading external subtitles with drag-and-drop does still work.
What browser?

Chrome 131.0.6778.140
Brave 1.71.121
All other extensions disabled, as well as Brave Shields

Is it happening with local video or streaming video?
Streaming
If the issue is happening with streaming video, what version of the extension are you using?
v1.6.1
If the issue is happening with streaming video, on what website(s) is the issue happening?
Jellyfin

The reason I couldn't find pages/emby-jellyfin-page.ts was because it wasn't actually being loaded at all since my Jellyfin is hosted at myexampleserver.com/jellyfin/, which doesn't match against the rules in pages.json. Changing the rule in pages.json to match on the path instead fixed that issue, but pages/emby-jellyfin-page.ts doesn't handle the basepath so it tries to download subs from myexampleserver.com/Videos/... and not myexampleserver.com/jellyfin/Videos/...
I'll rename the issue.


TerminalOpen event

Current implementation provide way to set options to the :terminal buffer using BufWinEnter.
au BufWinEnter * if &buftype == 'terminal' | setlocal bufhidden=hide | endif

But this is bits bothersome. How about add TerminalOpen event which raise only when open :terminal?

I think TermOpen name is better.
Because, TermChanged and TermResponse events are already exists.

Shougo wrote:

I think TermOpen name is better.
Because, TermChanged and TermResponse events are already exists.

Those events relate to 'term', so it's actualy better to avoid prefixing
with "Term" and use "Terminal".
--
From "know your smileys":
:-)-O	Smiling doctor with stethoscope
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Neovim also uses TermOpen for this event, so consistency would be nice.

James McCoy wrote:

Neovim also uses TermOpen for this event, so consistency would be nice.

All terminal window stuff is new, it's best to make it as clear now, we
can't change it later.  We already have confusion between "term",
"terminal" and "terminal window".  Since the option is called 'term', we
can associate things with that option name.  But the new terminal window
feature is clearly called "terminal" and not "term", so let's use that
name.  The shortening can only cause confusion.
--
In order for something to become clean, something else must become dirty;
but you can get everything dirty without getting anything clean.
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Still prefer TermOpen，as it is already exist in neovim.

“Term” is a recognized shortcut for “terminal”, no matter whether you spell it short or long if something like TermFocusLost (i.e. FocusLost controlled by terminal escape sequences; only the example) is implemented to fire on events caused by outside terminal that will cause confusion. If it is not, no confusion in TermOpen. Confusion could only be avoided by adding adjective, not by avoiding shortening.
And do not forget that :terminal can be shortened to :term, and you did exactly that in documentation as well (:h terminal-use, :h term++close), in &termkey and &termsize option names, in all associated function names, in plugin name (termdebug). If you want to stay consistent on that, do rename everything I mentioned.

yeah, the first time I know the key word term is in vim8, before this, the terminal is :terminal and :te in neovim, then vim8 add a new command :term instead of using :te.

@wsdjeg Both Vim and Neovim added command :terminal. But Neovim removed :tearoff as well, so minimal shortcut in Neovim is :te and minimal shortcut in Vim is :ter.

oh, I got it, sorry for my previous comment


How to setup creating key in resource?

Hi all,
can you tell how can I setup key to: Namespace+ClassName+Property...
Thanks.

I assume you are talking about "move to resource", right?
Where would you get property from?

@svatauli: any more specific info what this is about?

Closed - no clue what this is about, and no more info provided.


⚠ Jenkins Plugins Artifactory has degraded performance

In d7b075a, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1220 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 32cb88f after 1 hour, 6 minutes.


Add "terraform show" to rover command

hi @naeemdhby,  can you add syntax examples please? I always get
Terraform init return code 0
calling show
@calling show
running terraform show with -plan /home/vscode/.terraform.cache/contoso-sandpit/rover_jobs/20220727025455088189884/tfstates/level1/tfstate/d1s146-ddd1we.tfplan
 -TF_VAR_workspace: tfstate
 -state: /home/vscode/.terraform.cache/contoso-sandpit/rover_jobs/20220727025826081804675/tfstates/level0/tfstate/examples.tfstate
 -plan:  /home/vscode/.terraform.cache/contoso-sandpit/rover_jobs/20220727025826081804675/tfstates/level0/tfstate/examples.tfplan
null
Terraform show return code: 0
@calling clean_up_variables
cleanup variables
clean_up backend_files

Hi @arnaudlh , rover show command will read tfstate file output, so first need to do rover apply, and then do rover show.
@onlyillusion I think would be a good idea to rover show also read tfplan file as well, for example, have a condition to read plan file if passed on as argument, otherwise read from tfstate file.

Hey, we need this parameter to create json files out off the tfplan files for infracost. it also a good option to read tfplan file as well.

I have modified your PR to make it working.


Extra (parameter) results not present in per-message metadata response

What is the current behavior?
It looks like the extra field comes back within the final metadata response but not in the per-message response
Steps to reproduce
https://github.com/deepgram/deepgram-python-sdk/tree/main/examples/streaming/microphone
Expected behavior
Using raw websockets the metadata includes the extra parameter.  "extra" is included already in the final response.
"metadata": {"extra": {"example": "param"}, "request_id": "0bbe9f92-5b42-44bc-80cc-7c8f8c891e3f", "model_info": {"name": "general", "version": "2024-01-26.8851", "arch": "base"}, "model_uuid": "1ed36bac-f71c-4f3f-a31f-02fd6525c489"}

PR merged. Will be cutting a release shortly. Closing issue.


pscale: 0.89.0 -> 0.90.0

Description of changes
https://github.com/planetscale/cli/releases/tag/v0.90.0
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 165916 run on x86_64-linux 1

1 package built:

pscale

Perhaps we should wrap the executable with that variable?

Yes, please. You can't do much wrong compiling a go binary.

https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/go.section.md#ldflags-var-go-ldflags

@Kranzes @SuperSandro2000 thanks for the review. I've updated it with ldflags to populate that information :)


GPII-2802 - Add permissions section to gcp/README

This section describes how the permissions are set in GCP

lgtm. We need to make sure this stays up to date re. the work @natarajaya's doing on IAM permissions.

LGTM

LGTM


Would Keras support unique function?

Jax unique: https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.unique.html
TF unique: https://www.tensorflow.org/api_docs/python/tf/unique

Hi @cqwangding -
Keras doesn't have unique function. As keras is high level API on the top of tensorflow,jax,numpy,torch.
And this unique function is for data manipulation which can be done before keras models.
So all these keras backends(tensorflow,jax,numpy,torch) has seperate unique functions.
tensorflow: tf.unique
jax: jax.numpy.unique
torch: torch.unique
numpy: numpy.unique

@mehtamansi29 This feature request is from AdBrain. They may use it in the model instead of the data pipeline. Not exactly sure about which model they are using though.

I believe @hertschuh is working on a unique op. Though he is out for two weeks, so we can expect more action on this when he is back!

Per offline discussion, this is no longer needed.


Removed deviation

Hi,
I have removed deviation "route_policy_under_afi_unsupported" from metadata.textproto files.

Pull Request Functional Test Report for #2206 / d8ea3cd2e14c85f0fe011992d41b52cc633fe51e
Virtual Devices

Device
Test
Test Documentation
Job
Raw Log

Arista cEOS

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Cisco 8000E

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Cisco XRd

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Juniper cPTX

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Nokia SR Linux

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Openconfig Lemming

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Hardware Devices

Device
Test
Test Documentation
Raw Log

Arista 7808

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Cisco 8808

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Juniper PTX10008

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Nokia 7250 IXR-10e

RT-1.10: BGP Keepalive and HoldTimer Configuration TestRT-1.2: BGP Policy & Route InstallationRT-1.5: BGP Prefix LimitRT-1.12: BGP always compare MEDRT-1.3: BGP Route PropagationRT-1.7: Local BGP Test

Help

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 48.7%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1536

Relevant Lines:
3154

💛  - Coveralls


changed 5am.is to petercammeraat.net

Thank you, @petercammeraat! Since I only use GitHub as a mirror I added the changes manually 😊
Change was made here:
https://git.sr.ht/~timharek/timharek.no/commit/2c7f07c21bb2dedaa4024914fa8b687a8d2f9372


Unmatched "

Hello,
I am having an Unmached " on a very simple helloworld script. My configs:
{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "bashdb",
            "request": "launch",
            "name": "Bash-Debug (type in script name)",
            "cwd": "${workspaceFolder}",
            "program": "${command:AskForScriptName}",
            "args": []
        },
        {
            "type": "bashdb",
            "request": "launch",
            "name": "Bash-Debug (select script from list of sh files)",
            "cwd": "${workspaceFolder}",
            "program": "${command:SelectScriptName}",
            "args": []
        },
        {
            "type": "bashdb",
            "request": "launch",
            "name": "Bash-Debug (hardcoded script name)",
            "cwd": "${workspaceFolder}",
            "program": "${workspaceFolder}/path/to/script.sh",
            "args": []
        },
        {
            "type": "bashdb",
            "request": "launch",
            "name": "Bash-Debug (simplest configuration)",
            "program": "${file}"
        }
    ]
}

Configs are ok, need more info:

script content
OS version
debug output

OS: Ubuntu 16.04lts
Script:
#!/bin/bash

echo $PATH

debug out:
cd . ; bash -c "cd \"/home/alasec01/Projects/Meerkat-Reports/scripts\"; while [[ ! -p \"/tmp/vscode-bash-debug-fifo-17200\" ]]; do sleep0.25; done; \"bash\" \"/home/alasec01/.vscode/extensions/rogalmic.bash-debug-0.3.3/bashdb_dir/bashdb\" --quiet --tty \"/tmp/vscode-bash-debug-fifo-17200\" --tty_in \"/tmp/vscode-bash-debug-fifo-17200_in\" --library \"/home/alasec01/.vscode/extensions/rogalmic.bash-debug-0.3.3/bashdb_dir\" -- \"/home/alasec01/Projects/Meerkat-Reports/scripts/tmp.sh\" "
Unmatched ".

Can you check your vscode's terminal interpreter?
echo "$SHELL"

/bin/tcsh

So that's the reason, for this vscode workspace you need to set it to bash. Unfortunately last time I checked vscode api did not allow to check which interpreter is in use, so only way to make it work is to assume bash on Linux and mac + cmd on windows.

Thanks

Sorry for the late comment, even if the issue is closed.
What was the concrete solution to this?
I have set bash as the default terminal in my workspace, yet I still get this same error every time I launch the debugger.
Thanks!


Maybe after you get back it would make more sense to implement these 50%
https://repository.genmymodel.com/devshgraphicsprogramming/VulkanComputeAndRenderPassIndependentPipelines
https://repository.genmymodel.com/devshgraphicsprogramming/VulkanRenderpassIndependentPipelines
This means an IRenderPassIndependentPipeline<> and IComputePipeline<> without any IPipelineLayout<> objects implemented yet (that's just to much).
In place of IPipelineLayout<> you can use the current  SMaterialLayer<>
IDriverPipelineCache is a cache for helping to speed up the creation of new compute pipelines (separate from the runtime per-thread caching on OpenGL), there's an actual Vulkan API object for that, but with OpenGL we could put that to some use, like storing introspection or compiled SPIR-V for parsed GLSL + extension list. For now it can be an empty useless object.
Then at-least we:

disable/enable blend per FBO attachment
finally support all blending modes and factors on any FBO attachment separately
separate color mask per attachment
support all the Render/Raster states that vulkan advertises (lots of multisampling state)
finally support stencil operations properly (can rewrite OITZoo without bare OpenGL)
support the GL_PATCH primitive type without fucking guessing from whether we're using a tessellation shader

This changes our asset hierarchy a bit, In the follwing way.
~ I<>MeshBuffer -> I<>MeshDataFormat -> I<>Buffer ~
I<>MeshBuffer -> I<>RenderpassIndependentPipeline & I<>Buffer -> I<>SpecializedShader & I<>PipelineLayout/SMaterialLayer
So I<>MeshBuffer takes over the I<>Buffer vertex input bindings (so now vertex and index are stored in the same place) while I<>RenderpassIndependentPipeline replaces the remaining functionality of I<>MeshDataFormatDesc.
I<>RenderpassIndependentPipeline also replaces everything except for SMaterial::TextureLayer[] which is now moved outside as a member of I<>MeshBuffer instead.
P.S. Adding support for Compute shaders might be super easy now, just an IVideoDriver::setComputePipeline + IVideoDriver::dispatchCompute{Indirect}()
P.P.S.  In Vulkan Compute and Graphics pipelines can be set separately, we'll try to emulate the same with OpenGL.

anything fucked up after the merge?

nah, not yet at least

@Crisspl you need a IPipelineLayout::isCompatibleWith(const IPipelineLayout*)
its needed for a lot of things, such as deciding if descriptor sets need to be unbound or push-constants are nullified (in IRR_DEBUG we can push an all 0s range on the new pipeline when this happens, and do nothing on release builds).

@Crisspl you may want to read (but not apply fully) the ideas in #25 and #59

@Crisspl  check if this is true in the Vulkan spec

There might be something that says that if your pushconstants are incompatible, your descriptor sets are also invalidated

yes, compatibility for descriptor sets requires also compatibility for push constants. And compatibility for push constants requires exactly the same push constant ranges

Observation: lack of assumption that vertex buf binding = attrib number aka location makes hash for VAOs more complex and less probable that fitting VAO will be found in cache (+ more VAO objects needed to be created).

Observation: lack of assumption that vertex buf binding = attrib number aka location makes hash for VAOs more complex and less probable that fitting VAO will be found in cache (+ more VAO objects needed to be created).

How big is the hash with the above @Crisspl  ?

Extras to resolve:
https://github.com/buildaworldnet/IrrlichtBAW/pull/358#commitcomment-34799198

what about glCreateSamplers? For some reason until now we were using glGenSamplers and somehow it was working, not why. I'm going to add COpenGLExtensionHandler but i see there's this line
static PFNGLCREATESAMPLERSPROC pGlCreateSamplers; //NULL

seems kinda mysterious.

what about glCreateSamplers? For some reason until now we were using glGenSamplers and somehow it was working, not why. I'm going to add COpenGLExtensionHandler but i see there's this line

If you don't have EXT nor ARB DSA, then use Gen samplers, if you have DSA then create.
You can make an
extGlCreateSamplers

that wraps around the DSA thing and emulates it (like the other create/gen wrappers)

we have to know what (thing/type/...) should go to driver::draw*Indirect() in vao parameter. As for now it was IMeshDataFormatDesc<video::IGPUBuffer>* but obviously it cant be like this anymore

also: look at a few of my last commits and look for comments with TODO. i have some questions regarding driver's member ResetRenderStates

we have to know what (thing/type/...) should go to driver::drawIndirect() in vao parameter. As for now it was IMeshDataFormatDescvideo::IGPUBuffer but obviously it cant be like this anymore

ok the drawcall cannt set the VAO/Pipeline implicitly inside the draw anymore, we need a bindPipeline and bindDescriptorSets and pushConstants just like we used to have setMaterial that is separate from draw

also: look at a few of my last commits and look for comments with TODO. i have some questions regarding driver's member ResetRenderStates

Could you list them with links?

ok the drawcall cannt set the VAO/Pipeline implicitly inside the draw anymore, we need a bindPipeline

so meshbuffer shouldnt hold graphics pipeline object?

Could you list them with links?

https://github.com/buildaworldnet/IrrlichtBAW/pull/358/files#diff-e542fbce317985be1bfa2e51a7fe1a7eR1042
https://github.com/buildaworldnet/IrrlichtBAW/pull/358/files#diff-e542fbce317985be1bfa2e51a7fe1a7eR2703

so meshbuffer shouldnt hold graphics pipeline object?

yes it should, you can use it inside the draw the validate the same one has been bound as required

yes it should, you can use it inside the draw the validate the same one has been bound as required

O_O oh ok

@Crisspl does our raster parameter struct have a "rasterizerDiscard` boolean?

does our raster parameter struct have a rasterizerDiscard boolean?

yes

@Crisspl there's a bunch of files in
https://github.com/Crisspl/IrrlichtBAW/blob/shaders/source/Irrlicht
That have last been modified over a year ago, they're probably as good as dead now (i.e. COpenGL*Renderer)

@Crisspl also the following function in IVideoDriver will give you a headache (abstract non defined)
virtual void removeAllTextureBufferObjects() =0;
you can kill it

To make dynamic offsets work without copying of arrays, i'd have to modify precomputed (in COpenGLDescriptorSet) multibind params in-place.
(this obviously requires COpenGLDescriptorSet::getMultibindParams() to return params struct as non-const ref; this implies that GL state tracker structs have to hold descriptor sets as non-const [smart] ptrs; this implies that driver::bindDescriptorSets() has to take descriptor sets as non-const ptrs as well)
Ofc after modification and running multibind calls i'd have to bring params to previous values. But real implications are that concurrent access to precomputed multibind params in COpenGLDescriptorSet is out of question.
So.. either that or we have to copy arrays of offsets in order to apply dynamic offsets. I personally think that copying is much better here (copying has to happen only if descriptor set was bound with any dynamic offsets, only offsets arrays are copied, and ofc it concerns UBOs and SSBOs only)

So.. either that or we have to copy arrays of offsets in order to apply dynamic offsets. I personally think that copying is much better here (copying has to happen only if descriptor set was bound with any dynamic offsets, only offsets arrays are copied, and ofc it concerns UBOs and SSBOs only)

This small amount of copying makes sense, but only in COpenGLDriver and on-stack (91+96 values on stack, current OpenGL maximums for SSBO and UBO counts).

Done everything except:

https://github.com/buildaworldnet/IrrlichtBAW/pull/358#discussion_r332878486 (iterator pair...)
clearScreen() GL state changes (related: https://github.com/buildaworldnet/IrrlichtBAW/pull/358#discussion_r328449492 and https://github.com/buildaworldnet/IrrlichtBAW/pull/358#discussion_r333882980)
get rid of tex targets and img formats in COpenGLDescSet multibind params struct
https://github.com/buildaworldnet/IrrlichtBAW/pull/358#discussion_r323703046

to me it's merge-ready except this discussion

we need to put IBuffer back into irr::core

we need to put IBuffer back into irr::core

there are 2 IBuffers now: asset::IBuffer and core::IBuffer decided here https://github.com/buildaworldnet/IrrlichtBAW/pull/358#discussion_r335101656 where asset::Ibuffer is answer for "ICPU and IGPU Buffer need a common base on top of core::IBuffer"

there are 2 IBuffers now: asset::IBuffer and core::IBuffer decided here #358 (comment) where asset::Ibuffer is answer for "ICPU and IGPU Buffer need a common base on top of core::IBuffer"

Ok resolved.

to me it's merge-ready except this discussion

Your input attachment query is not querying all the extensions
pipeline caching mess
compute pipelines
pasting of enabled GLSL extensions

to find the conversations scross up and press "Load More` until all items are not hidden anymore

it's ready

maybe try closing this PR and opening another one so that comments in this one are preserved. (If you're that retargeting will cause all the comments to disappear). idk how to dump comments to some usable form (not screenshot; i mean theres chrome extension to "screenshot" whole page whatever long wouldnt it be but it wont let us ctrl+f later)

There are two very old unresolved things:

using ifdef instead of commenting out old code so that I can find it later
@AnastaZIuk documentation update

Obviously need to clear the latest comments and we can merge.
But then we immediately open another PR for compute pipelines and making this branch finally build.

@Crisspl merge and correct https://github.com/Crisspl/IrrlichtBAW/pull/15
Then I will hit the merge button.


Add an FAQ section to README

There is now one: https://github.com/NVIDIA/JAX-Toolbox#faq-frequently-asked-questions
Did you wanted some specific Q&A in it? if not, we can close this issue.


Fix CPU usage issue

Problem
I have 2 x Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
64Gb DDR4,
NVMe 1Tb,
800Mbit Bandwidth
It's been running good without skips on 2.0.5, but after upgrade to 2.0.6 it doesn't work at all. Rebuilting storage and then crash.

I went back to 2.0.5, now everything works fine, but during the day it didn't work - solana catchup didn't go.
please fix 2.0.6, DO NOT RELEASE 2.0.6 to master as "minimum required", without fixing CPU issues.

Closing this issue out because:

We do validator support in our Discord (https://discord.com/invite/solana)
There is insufficient information in this post (which cluster / nature of the crash / ...)
Even for testnet, your node is under provisioned


add new features for the JSON

allow a route in the API to download the JSON
switch so the name of the concept mapping (e.g. FEMALE) is use in the json name

@calmacx could you just add an item to the changelog particularly as this is a compatibility change. Thanks


Remove public folder

When we run ember build for production environments a ember-social folder is created in dist/ containing the crossdomain and robot.txt files which is unnecessary and undesirable at least as far as I know.
Just removing the public folder fixes it and should not cause any problems.

Thanks!


chore: ignore docs path explicitly

Adding skip actions to allow required checks to pass on the ignored paths
Done according to:
https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks

Checklist

[ ] run make test
[ ] PR title must follow conventional-commit-standard
[ ] tests and/or benchmarks are included
[ ] documentation is changed or added
[ ] commit message and code follows the Code of conduct

I think this should have been done as a single workflow. There was no need to add two "dummy" ones.


Update binaries

Generated by developer/images/dependencies-update/hack/bin/update.sh

/test pipeline-service-update-dependencies


Bar chart renders half of the data only

The bar chart renders only half the data. renders only even number indexed. (0, 2, 4, 6).
Is this a bug ?
Any advice to the problem

Can you post an example of the data being passed in so this issue can be recreated?

I had the same issue but I managed to fix it with this option:
options={{ maintainAspectRatio: false }}
e.g.
<Line data={this.state} options={{ maintainAspectRatio: false }} />;


Add link to Canopsis integration in docs

Hello,
Please find in this pull request some informations about Canopsis-Prometheus integration.
Canopsis is an Open Source ( french project https://github.com/capensis/canopsis ) Hypervisor Solution which collect information about several sources ( through AMQP bus or HTTP API ) and aggregate them into a single monitoring interface. This can be usefull to make some correlation rules, global alarm list or global weather of your IT.
Request you to please include a link to the Canopsis-Prometheus connector documentation in your Integrations page.
Thanks

Thanks!


Enable loading XSF files from FHI-AIMS

The FHI-AIMS code offers to output the grid data either as CUBE or as XSF files. However, the structure of data in those XSF files is somewhat different from what one gets from VASP by applying the v2xsf tool on a LOCPOT or CHGCAR file. The loadXSF and loadXSFGeom functions need to be modified in order to read the FHI-AIMS data correctly. Essentially, the grid data in the FHI-AIMS generated XSF files, including the grid dimensions and the ordering of the data points, are the same as they would be in the corresponding CUBE files (except that lengths are fortunately still expressed in angstrom, not bohr, in XSF). The consequences can be summarized into the following three points:

Even for periodic systems, the grid dimensions need not be  the same as the dimensions of the periodic cell. The default behavior for periodic systems, if you do not specify anything to the contrary in the control.in file, is still such that the grid covers the whole unit cell. However, a grid covering the unit cell means something slightly different for FHI-AIMS compared to what it means for VASP. There are no repeated data points at the unit cell boundary in FHI-AIMS like there would be in the XSF output from VASP. The first and the last point along an axis are not identical. The logic of FHI-AIMS is that the positions of nodes that form the grid are centers of the corresponding "voxels", which implies that the first and last point (node) along a particular axis do not in fact lie right on the cell boundary but they are shifted by half the voxel size (=half the grid step). Let me illustrate this by a concrete example: Suppose we have a 10x10x10 cell, which we want to sample with a grid in which the neighboring points are separated by the step of 0.1. The corresponding grid we expect to get from VASP would have 101x101x101 points, the origin would be at (0,0,0) and the grid vectors (pointing from the grid origin to the final points along the corresponding axes) would be  10   0   0
   0  10   0
   0   0  10

The first and the one-hundred-first points are duplicates of each other in any of the three main directions.
In XSF from FHI-AIMS, in contrast, there would be just 100x100x100 data points. The origin would be at (0.05, 0.05, 0.05) and the grid vectors would be shorter by one "voxel" compared to VASP, like this:  9.9  0.0  0.0
  0.0  9.9  0.0
  0.0  0.0  9.9

Well, actually, the grid vectors from FHI-AIMS would be  0.0  0.0  9.9
  0.0  9.9  0.0
  9.9  0.0  0.0

as I will explain in the next point. Anyway, the control.in file of FHI-AIMS allows a fine-tuned control over the grid size and position, by using the cube origin and cube edge keywords. So, in principle, the user could always extract the same grid from FHI-AIMS as it would have been obtained from VASP. But it would be rather awkward. In particular, the cube origin in control.in does not really mean origin. Instead, it specifies the center of the grid. So it would have to be something like half the lattice vector in every direction minus half the grid step in every direction in order to obtain the desired result. I want to avoid having to do this when using FHI-AIMS. It should be possible as the XSF file contains both the information on the lattice vectors (as PRIMVEC) as well as the independent information on the grid dimensions. The lattice vectors would still be  10.0     0.0    0.0
   0.0    10.0    0.0
   0.0     0.0   10.0

even from FHI-AIMS, but the grid vectors would differ from the lattice vectors and the origin would be non-zero. The loading functions in ppafm/io.py need to interpret this information correctly.

The arrangement of data on the grid as for the order of the axes is the same in the FHI-AIMS-generated XSF files as it is in CUBE files. Consequently, the first and third grid axes are swapped with respect to the lattice vectors in periodic systems. Likewise, these grid axis are also swapped with respect to the Cartesian axes in non-periodic systems. Again, this could in principle be recognized by the loading function(s) since the grid vectors are recorded in the XSF file and can be compared to the lattice vectors in the same file.
Finally, the Hartree (electrostatic) potential is given in the Ha (rather than eV) units by FHI-AIMS. This is a point were I got a bit desperate because, in contrast to the issues above, there is no record inside the XSF file to indicate in what units the data are supposed to be expressed.  After having  figured out how to solve the other issues, I really did not want to introduce some new keyword argument to the PP-AFM code that would explicitly specify whether the input is from FHI-AIMS or from VASP. So, my proposed solution: Whenever the loadXSF functions detects that the first and third grid vectors have been swapped, a matter which has to be decided anyway, assume that the data originate from FHI-AIMS and, if moreover the data being read are expected to be a potential (rather than density), do the conversion from Ha to eV.

Hi Martin,
Just a quick comment! Do not use XSF from AIMS. Never, ever. There is a mess in the units, in the cell, as you are pointing out, and so on. I have once looked into it, it does not make sense to work with it for us, it is a not-necessary extra work. We should just say it in the manual, that in FHI-aims one should use only cube files.
Ondrej
From: Martin Ondráček @.>
Sent: tiistai 24. lokakuuta 2023 17.56
To: Probe-Particle/ppafm @.>
Cc: Subscribed @.***>
Subject: [Probe-Particle/ppafm] Enable loading XSF files from FHI-AIMS (Issue #217)
The FHI-AIMS code offers to output the grid data either as CUBE or as XSF files. However, the structure of data in those XSF files is somewhat different from what one gets from VASP by applying the v2xsf tool on a LOCPOT or CHGCAR file. The loadXSF and loadXSFGeom functions need to be modified in order to read the FHI-AIMS data correctly. Essentially, the grid data in the FHI-AIMS generated XSF files, including the grid dimensions and the ordering of the data points, are the same as they would be in the corresponding CUBE files (except that lengths are fortunately still expressed in angstrom, not bohr, in XSF). The consequences can be summarized into the following three points:

Even for periodic systems, the grid dimensions need not be the same as the dimensions of the periodic cell. The default behavior for periodic systems, if you do not specify anything to the contrary in the control.in file, is still such that the grid covers the whole unit cell. However, a grid covering the unit cell means something slightly different for FHI-AIMS compared to what it means for VASP. There are no repeated data points at the unit cell boundary in FHI-AIMS like there would be in the XSF output from VASP. The first and the last point along an axis are not identical. The logic of FHI-AIMS is that the positions of nodes that form the grid are centers of the corresponding "voxels", which implies that the first and last point (node) along a particular axis do not in fact lie right on the cell boundary but they are shifted by half the voxel size (=half the grid step). Let me illustrate this by a concrete example: Suppose we have a 10x10x10 cell, which we want to sample with a grid in which the neighboring points are separated by the step of 0.1. The corresponding grid we expect to get from VASP would have 101x101x101 points, the origin would be at (0,0,0) and the grid vectors (pointing from the grid origin to the final points along the corresponding axes) would be

      10   0   0

       0  10   0

       0   0  10

The first and the one-hundred-first points are duplicates of each other in any of the three main directions.
In XSF from FHI-AIMS, in contrast, there would be just 100x100x100 data points. The origin would be at (0.05, 0.05, 0.05) and the grid vectors would be shorter by one "voxel" compared to VASP, like this:
9.9  0.0  0.0
0.0  9.9  0.0
0.0  0.0  9.9
Well, actually, the grid vectors from FHI-AIMS would be
0.0  0.0  9.9
0.0  9.9  0.0
9.9  0.0  0.0
as I will explain in the next point. Anyway, the control.in file of FHI-AIMS allows a fine-tuned control over the grid size and position, by using the cube origin and cube edge keywords. So, in principle, the user could always extract the same grid from FHI-AIMS as it would have been obtained from VASP. But it would be rather awkward. In particular, the cube origin in control.in does not really mean origin. Instead, it specifies the center of the grid. So it would have to be something like half the lattice vector in every direction minus half the grid step in every direction in order to obtain the desired result. I want to avoid having to do this when using FHI-AIMS. It should be possible as the XSF file contains both the information on the lattice vectors (as PRIMVEC) as well as the independent information on the grid dimensions. The lattice vectors would still be
10.0     0.0    0.0
0.0    10.0    0.0
0.0     0.0   10.0
even from FHI-AIMS, but the grid vectors would differ from the lattice vectors and the origin would be non-zero. The loading functions in ppafm/io.py need to interpret this information correctly.

The arrangement of data on the grid as for the order of the axes is the same in the FHI-AIMS-generated XSF files as it is in CUBE files. Consequently, the first and third grid axes are swapped with respect to the lattice vectors in periodic systems. Likewise, these grid axis are also swapped with respect to the Cartesian axes in non-periodic systems. Again, this could in principle be recognized by the loading function(s) since the grid vectors are recorded in the XSF file and can be compared to the lattice vectors in the same file.
Finally, the Hartree (electrostatic) potential is given in the Ha (rather than eV) units by FHI-AIMS. This is a point were I got a bit desperate because, in contrast to the issues above, there is no record inside the XSF file to indicate in what units the data are supposed to be expressed. After having figured out how to solve the other issues, I really did not want to introduce some new keyword argument to the PP-AFM code that would explicitly specify whether the input is from FHI-AIMS or from VASP. So, my proposed solution: Whenever the loadXSF functions detects that the first and third grid vectors have been swapped, a matter which has to be decided anyway, assume that the data originate from FHI-AIMS and, if moreover the data being read are expected to be a potential (rather than density), do the conversion from Ha to eV.

Reply to this email directly, view it on GitHubhttps://github.com/Probe-Particle/ppafm/issues/217, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFZAFI2O266BHY327JTRYLYA7JI5AVCNFSM6AAAAAA6N244KCVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE2TSNBVHA2DSMA.
You are receiving this because you are subscribed to this thread.Message ID<EMAIL_ADDRESS>

Hi, Ondřej,
I have edited out the copy of my own post from yours in this comment section to keep it more concise. I hope this alteration does not break anything.
As for the substance: Well, avoiding the XSF from FHI-AIMS would surely be a solution too, but...
The XSF from AIMS is not any worse than the CUBE file as long as you know it comes from AIMS. In other words, the only advantage of CUBE is in fact that we can always assume it comes from FHI-AIMS and handle it accordingly. Other than that, there is not that much difference and in fact the main problem of XSF from AIMS is that it is too similar to CUBE (the exactly same grid as in CUBE, with grid point positions and ordering untypical for the XSF format).
There is at least one issue I need to solve because of FHI-AIMS, irrespective whether I am going to use XSF or CUBE files. I do not like the origin of the grid-covered space being forced to the origin of the Cartesian system. As it is now, I have to make sure that the origin of the grid in the CUBE file is always at zero, which means always putting the center of the unit cell to the cube origin line in the control.in, which is a pain.

Hi,
“””
The XSF from AIMS is not any worse than the CUBE file as long as you know it comes from AIMS. In other words, the only advantage of CUBE is in fact that we can always assume it comes from FHI-AIMS and handle it accordingly. Other than that, there is not that much difference and in fact the main problem of XSF from AIMS is that it is too similar to CUBE (the exactly same grid as in CUBE, with grid point positions and ordering untypical for the XSF format).
“””
I see the main problem in the units conversion that is why I would suggest to omit it. Especially when CUBE is like a nature format for AIMS.
My point was to save your time, to not try solve all the problems on the world 😊.
Regards,
Ondrej
From: Martin Ondráček @.>
Sent: tiistai 24. lokakuuta 2023 18.32
To: Probe-Particle/ppafm @.>
Cc: Krejci Ondrej @.>; Comment @.>
Subject: Re: [Probe-Particle/ppafm] Enable loading XSF files from FHI-AIMS (Issue #217)
Hi, Ondřej,
I have edited out the copy of my own post from yours in this comment section to keep it more concise. I hope this alteration does not break anything.
As for the substance: Well, avoiding the XSF from FHI-AIMS would surely be a solution too, but...
The XSF from AIMS is not any worse than the CUBE file as long as you know it comes from AIMS. In other words, the only advantage of CUBE is in fact that we can always assume it comes from FHI-AIMS and handle it accordingly. Other than that, there is not that much difference and in fact the main problem of XSF from AIMS is that it is too similar to CUBE (the exactly same grid as in CUBE, with grid point positions and ordering untypical for the XSF format).
There is at least one issue I need to solve because of FHI-AIMS, irrespective whether I am going to use XSF or CUBE files. I do not like the origin of the grid-covered space being forced to the origin of the Cartesian system. As it is now, I have to make sure that the origin of the grid in the CUBE file is always at zero, which means always putting the center of the unit cell to the cube origin line in the control.in, which is a pain.
—
Reply to this email directly, view it on GitHubhttps://github.com/Probe-Particle/ppafm/issues/217#issuecomment-1777493372, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFZAFPH4GRVE5TYIMJ736DYA7NOVAVCNFSM6AAAAAA6N244KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZXGQ4TGMZXGI.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>

Hi everybody. This is just to announce that I am going to finish this work after all. The reasons are that:

The bulk of the work is already done, part of it (regarding the XSF input and output itself) in my xsf-and-cube branch, another part (regarding the grid origin) now submitted as PR #229 (needed to fix an other issue anyway), so what I mainly have to do now is just merging and testing the stuff.
I want to use FHI-AIMS, whether through XSF or CUBE, and neither actually worked as it should have until recently. Resolving the issue with the grid origin was likely the main thing needed to make CUBE work while much additional work was needed for XSF, but I did not know that before I actually did the work. Now that the work is mostly done, why throw it out of window.
Having the two input options available, both CUBE and XSF, is a kind of safety feature. In the future, I want to avoid the frustration I have experienced in my attempts to use PP-AFM with FHI-AIMS. The experience went something like: "I have troubles using the XSF files from FHI-AIMS in PP-AFM." "Well, XSF from AIMS sucks, why don't you use CUBE?". "Handling coordinates  from CUBE is a total mess, I can't figure out how the scan region relates to atom positions..." "Okay, CUBE is not that well tested in PP-AFM, why don't you use XSF instead?" And so on, loop to infinity.
So while I am going to stick to CUBE files in my everyday work with FHI-AIMS, once anything breaks down, I want to have the option to test CUBE against XSF and see whether the problems are dependent on the input format or not.
Maintaining the ability to read XSF from various sources will force us to handle this format properly, including, for instance, handling the seemingly unrelated issue of grid dimensions of the OutFz.xsf files which I discuss in #198.

A possible argument against implementing this feature: The XSF from FHI-AIMS is indeed rather idiosyncratic. While my idea was that our functions that implement the XSF input should be as general as possible, there will nevertheless have to be a point in the code where it decides "is this XSF from AIMS or not?" and handles the "Yes, from AIMS" branch separately. This is so for two reasons:

The data ordering (ZYX vs XYZ or, if you want, F-contiguous vs C-contiguous) and
the eV vs hartree units for potentials.

The necessity of making such decision is potentially dangerous: If a XSF file were misidentified as created by FHI-AIMS even though it actually originated from elsewhere, that would cause troubles. But I believe I can implement the FHI-AIMS vs non FHI-AIMS input recognition safely.


Fix cmake install target

Changes from #1325 caused the install files to be wrong. I found a simpler way to handle it.

Why was it not detected by the install test app? how can we improve the test?

Why was it not detected by the install test app? how can we improve the test?

It's not detected because the layout is proper, but there were extra files being added. I'm not sure how easy it would be to test.


Improve performance of rendering creature heads

Based on the performance investigation documented in #767, creature heads are potentially a performance bottleneck.
We could render these heads more efficiently, in a way that does not require 30 nodes per creature. At a minimum, we should at least prototype a creature with these improvements and see how what impact this could have on performance.
-create a creature which uses a conventional sprite sheet for their head
-create a very rough prototype (just a screenshot or a flat garbage sprite is fine)
-stress test this creature variant to ensure the performance gains are there
-make it look good
 -create 'screenshot' of facing forward
 -create 'screenshot' of facing away
 -blinking animation
 -eating animation
 -idle animations
 -emotes
-stress test this creature variant again to ensure the performance gains remain

There are many bottlenecks in rendering creatures, but their head sprites seem to not carry a very significant impact. Downgrading to priority 4.

A contributor "pixaeron" created a patch for v0.2820 which replaced the creature shaders with four modulated images. It introduced other problems so was never merged, but it's worth keeping in case we attempt something similar in the future.
4-image-performance-tweak.patch.txt


Backspacing the entry field causes stutter/delay

Microsoft PowerToys version
0.56.2
Running as admin

[X] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
Simply hold backspace with text in PowerToys Run.
✔ Expected Behavior
A smooth removal of characters without lag or interruption.
Ideally, while the backspace key is being held down, the search query should not update.
❌ Actual Behavior
Every time a character is removed, the results update. This causes lag when trying to remove sections of the search query.
Here is a short video showing how slow removing text can be while simply holding backspace.
https://user-images.githubusercontent.com/49583515/157976831-66518fbc-2b24-4662-b06a-7907d2e72236.mp4
And this is actually not a bad example. Searching for several files with similar names causes an even larger delay as the search updates every time a character is removed.
Contrast this with something like Spotlight for macOS which is fast and smooth while holding backspace, only updating the search results when input stops.
Other Software
No response

So, this is more like an enhancement to only do search after waiting for some time to check if the user stopped typing.

Why does it only have to do search after waiting?
The Start menu search and Spotlight both search continuously (as is evident from the updating results as I keep holding Backspace) while being completely smooth.
I think this should be an enhancement to just be smoother without losing functionality.

One of the motivations for PowerToys Run is to have stable results. So, that when we click a result, it won't suddenly change to another result because another search was just finished and the results changed.

How is that relevant to the stuttering problem?
That sounds like something that should also be tackled, but, again, not at the expense of performance or functionality.
My point with the comment above is that it shouldn't be stuttering even with the current approach of continuous updating because other similar tools don't suffer suffer this.
The very top of the front page for Run reads:

a quick launcher for power users that contains some additional features without sacrificing performance.

That is anything but my experience.
Backspacing causes stutters.
Starting a query causes even bigger stutters (which I assume has the same root cause as backspacing).
Overall it's just annoying to use due to the constant lag so I find myself going for the Start menu instead.

It's possible that some of the PowerToys Run plugins might be taking longer than usual to do searches. You can try to disable the plugins 1 by 1 in Settings to find out which one it might be.

I've disabled everything, and it's still stuttering a little when holding Backspace or typing very quickly.
That said, now I have to type quite a bit quicker than I naturally can to notice it.
But after doing what you suggested and reenabling the plugins one-by-one, I've come to a conclusion that Windows Search has the biggest impact on performance by far.
It's the main culprit in the big startup stutter and the frequent stuttering while typing and holding Backspace - especially when it comes across a result(s).
Incidentally it's the plugin I'm the most interested in using..
The Programs plugin also increases stuttering a little.

I believe this is now there with the input smoothing.  /needinfo


Allow specifying field names for operators of range field types

Is your feature request related to a problem? Please describe.
For example, currently for integer_range field one has to provide such a value object:
"expected_attendees" : {
    "gte" : 10,
    "lt" : 20
}

This is problematic if my range field is not structured like this in original data source. My use case is that my incoming documents have min_price and max_price fields and I would like to index them to a range field. But currently this does not seem to be possible without modifying the structure of the document before indexing. This is a problem for me because I am using OpenSearch both as search engine and data store itself. So I would like to retain the original data structure and be able to index the range field.
Describe the solution you'd like
I would propose that one could define for the field names for gte, lt, etc., operators. So that I could provide also:
"expected_attendees" : {
    "min" : 10,
    "max" : 20
}

Ideally, I could also have additional fields in the object, which would be ignored, but could be specified as fields for other indices, e.g.:
{
  "name": "foobar",
  "min_price": 10,
  "max_price": 20
}

Describe alternatives you've considered
I could change input document structure before I index it, but then I would have to store original data additionally (or reverse transformation when reading source documents from search).

My intuition is that transforming your data upon indexing here is the right way to go, as opposed to adding the flexibility to handle any input structure. Adding additional mappings for your field names will work for your use case but this seems like a bit of a rabbit hole to go down to handle any arbitrary input structure. Storing the original data additionally (as you proposed) I think is generally the way to preserve the input structure.
@dblock Do you have any thoughts here?

I am not sure if there is really a rabbit hole here. Currently, there is just implicit expected_attendees.gte field name expected, and not sure why it could not be expected_attendees.min or expected_attendees_min even? Once that is configurable, there is no other structures to support, I cannot imagine one at least, so not sure what would be the other things deeper in the rabbit hole if we support that?

If I understand this correctly the first part of the proposal is to be able to specify min and max vs. range operators. Is this just an alias where min=gte and max=lte? I am guessing those were chosen to allow lt and ge and multiple names to accomplish the same thing doesn't seem common, but I could be wrong - any known examples?
Is the second proposal is to be able to transform 2 fields into a range?
PUT items
{
  "mappings": {
    "properties": {
      "price_range": {
        "type": "integer_range",
        "lte": "min_price",
        "gte": "max_price"
      },
    }
  }
}

PUT items/_doc/1?refresh
{
  "min_price" : 10,
  "max_price": 20
}

GET items/_search
{
  "query" : {
    "term" : {
      "price_range" : {
        "value": 12
      }
    }
  }
}

...
"min_price" : 10,
"max_price" : 20

Agree with @andrross that this can be accomplished with a data transform, so maybe we are asking OpenSearch to do too much? Other than all-in-one convenience, what would be the advantages (e.g. significant performance benefit or query capability) of implementing this in the engine?

Is this just an alias where min=gte and max=lte?

Yes.

     "price_range": {
       "type": "integer_range",
       "lte": "min_price",
       "gte": "max_price"
     }

This looks great syntax. Exactly that.

this can be accomplished with a data transform

I mean, many things can be accomplished by a transform, but OpenSearch still supports it, for example, everything about data formatting and parsing. It could just accept millisecond values (what it uses internally) and asks users to parse their dates before ingest manually, but it does support formats for parsing instead. And this to me here is similar, a format of how a range is described. OpenSearch could support different formats of range structure.

I am not sure what the conventions are around aliases, do you (or does anyone) know of a precedent for introducing aliases for functions?
If you want to try implementing this, see if you can accomplish the transform with a plugin. I do suspect that the more generic transform feature for all *_range types is going to be very far from trivial, but always welcome PRs to look at suggested implementations.

Curious how others feel about this type of ingestion features, maybe @macrakis?

OK, let's first see what others think before I attempt a PR or something. Just to be clear, are you suggesting to add a PR with an official plugin? Or PR to the core OR make my own local plugin?

I think either works for a prototype. If you develop your own plugin then you get a ton of freedom to innovate in whichever way you want! We can always adopt it in the official project/distribution over time if it becomes popular.

I was looking into existing plugins and I do not see any which would be doing any transformations? Is there any plugin API hook you would suggest for me to look into as possibly relevant to this task?

I don't know this area well, maybe someone else can suggest something?


Fix for incorrect apt repo key value.

The key 'F7B8CEA6056E8E56' is incorrect which results in the module adding this key on every run. Correct key is: '056E8E56' as taken from http://www.rabbitmq.com/signatures.html

I don't think the key is incorrect. You can see the complete value if you query apt-key:
# apt-key adv --list-keys --with-colons --fingerprint | grep -i rabbit
pub:-:1024:17:F7B8CEA6056E8E56:2007-07-06:::-:RabbitMQ Release Signing Key <info@rabbitmq.com>::scESC:

This is more likely a bug in the apt module. What version of the apt module are you using? If you don't mind filing a ticket on https://tickets.puppetlabs.com with more details we can try to get this fixed.

I agree, the key is not incorrect. I have filed a bug with some more details as per request.
Bug report: https://tickets.puppetlabs.com/browse/MODULES-1786


Existence of Pandoc is checked locally

So in remote case if pandoc is missing locally we won't let user publish. On the other hand, if pandoc is missing on the server, user may get error messages in REPL.

Same for pdflatex

https://github.com/Microsoft/RTVS/commit/3636d3f9a2b4e9714f0760a4e667299b643664e3


THOUGHT: Scaling of scenario from the user's perspective

With regards to the local datasets: in this stage, I think It's worthwhile thinking about an approach that starts from the (current) user's perspective.
What if the user can input all the necessary data to set the present for a local dataset through the user's interface?
Prefiled database
Since this is all based on real Dutch provinces, municipalities, neighbourhoods etc, we can fill a database with region information. (e.g. in the neighbourhood of Paddepoel there are 504 households). This will help the user enormously and radiates confidence.
Steps for the user to complete in user's interface?
Just my first thoughts:

What do you want to model? -> Select [provincie, gemeente, wijk, buurt, custom]
User selects e.g. 'buurt'
(Unless custom): which buurt do you like to model?
User fills in e.g. 'Paddepoel'
And all the inputs are pre-filled.

some of them will be taken from the database (e.g. number of households, number of cars per household)
some of them will be taken from the Dutch average (e.g. technology share)

User can adjust inputs

Setting present technology mixes though etmodel's GUI?
I think one of the challenges is the technology mix: it is a group of sliders. Perhaps the current GUI can be of help, in a way that @antw suggested already before.
Disadvantages

No version control for obvious reasons
This still does not deal with the fact that local scenarios should not be based relatively on the dutch dataset and does not exclude input statements per se.

cc @DorinevanderVlies @antw @jorisberkhout @AlexanderWirtz

We're currently documenting our ideas and thought in here. This idea hasn't occurred to us.

What if the user can input all the necessary data to set the present for a local dataset through the user's interface?

I'm currently thinking that that's not where you want the user to do this in all honesty. I already proposed an idea in the documentation to create a completely seperate application which can take numbers [? vague statement alert] and output an ETsource dataset (a bit like a user-friendly ETdataset?). Ofcourse this is an ambitious time-consuming project I can imagine. But it might be worth doing though, although I'm not sure what it will look like because I have not enough knowledge yet about the current way a dataset is created.

A prefilled database would definitely be nice, but it is probably more of a second version than an initial version. So I suggest we should keep it in mind but design something now that also works without it.
For whom are we designing this? For our own research about provinces, municipalities etc in order to generate etm data for them analogous to national data? Or for users to easily scale down and adjust a national dataset to their local circumstances?
For the former we probably need to have version control and do not need the GUI. For the latter a GUI is almost a necessity... but version control is also still nice - at least some change tracking when the user modifies his local scenario.


[Onyx-676] Update beam version into 2.2.0

Closes #676.
This PR:

Updates beam version into 2.2.0
Temporarily disables some part of test that doesn't run correctly with beam 2.2.0

Closing since there's no plan to handle the remaining issues now.


Trigger prod CLI release pipeline for v0.9.1

Issue #, if available:
Description of changes:
Trigger prod CLI release pipeline for v0.9.1
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


SpringCloud Fingley release breaks SpringCloudCommandRouter

SpringCloud Fingley release breaks SpringCloudCommandRouter
This call is broken (Doesn't exist on DiscoveryClient anymore).
this.discoveryClient.getLocalServiceInstance();
How to reproduce.
A project with this version of cloud and Axon 3.1 .2
spring-cloud-commons-2.0.0.M5.jar
Fix:
As we wire in the CommandRouter, we will make an own implementation which works around the issue. (two methods make the call and can be extended). If time permits we will also provide a patch.

Will update spring-cloud-commons version to 1.3.1.RELEASE, as in that version the DiscoveryClient#getLocalServiceInstance() is already deprecated as pre-work to spring-cloud-commons/2

Thanks, our patch is working by injecting this bean:
SimpleDiscoveryProperties simpleDiscoveryProperties;
Also I noticed a but in the HTTP backup router constructor signature.
Please notice, the default value is after the closing curly brace.
which will yield : /message-routing-information:/message-routing-information.
@Autowired
   public SpringCloudHttpBackupCommandRouter(DiscoveryClient discoveryClient,
                                             RoutingStrategy routingStrategy,
                                             RestTemplate restTemplate,
                                             @Value("${axon.distributed.spring-cloud.fallback-url}:/message-routing-information") String messageRoutingInformationEndpoint) {
       this(discoveryClient,

Thanks for sharing @dzonekl! Just fixed the @Value default value typo.


Swift version?

I read, that there might be an upcoming Swift version? Maybe a ClosuresKit? Does someone know where it can be found?

Just noticed your comment. Feel free to use the name ;-)

https://github.com/britzlieg/ClosuresKit

Just came across some code using a react pattern on uitextfield.  - https://github.com/Ramotion/reel-search - what's the best way to hook into delegate methods with swift?
let tf = UIView()
tf.onTap = {}
fyi - http://objc2swift.yahoo-labs.jp/

this seems to be the best replacement thus far for blocks in swift.
https://github.com/collinhundley/APTargets


adding semi-colon to end of location_custom_cfg_append options.

This already exists on the prepend, just missing on the append.

@kwolf thanks for the contribution, but you should be aware that this is by design.
If you want the versions with a trailing semi-colon you should use location_cfg_append instead.

Sorry about that.  Thanks for pointing out the other option!


Get "Input string was not in the correct format" error for Azure image classification when Azure's regieon format is Poland

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog): 16.13.2
Visual Studio Version

To reproduce

set system region format to russia/Poland (whatever decimal separator is ',')
run azure image classification

The bug is not repro on the latest main branch: <IP_ADDRESS>6501.
After change the system region format to Russia/Poland like below, the Azure image classification can success to train, evaluate, consume and run the generated projects. There is no error in whole progress.


Zero Speed Determination Algorithm

Need to come up with a zero speed determination algorithm.

Working on this.

Issue resolved in commit 3057191.


chore(deps): add auto-approve action for dependabot PRs

:tada: This PR is included in version 1.4.12 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Detect repeated axis in reduction

Fix #4486.

/test

run jenkins please

Jenkins CI test (for commit 8e81a97efe3036d24c2acb5abde2bbbf50a4562c, target branch master) succeeded!

/test mini

Updated the branch because it must have #6034 to run tests.

Jenkins CI test (for commit 08b9255006bb81fe8838c42a275272aae19c28c5, target branch master) succeeded!