ROS 2 interoperability issues

I'm unable to run https://github.com/BerkeleyAutomation/FogROS2/blob/main/fogros2_examples/launch/talker.launch.py successfully and obtain basic interoperability. Neither between edge-cloud, nor between edge-edge.
From my inspection, VPN works just fine but there's something wrong in the setup. ROS 2 CLI daemons aren't able to see simple Nodes launched (manually) across machines in the VPN.
This deserves further attention.

@vmayoral can you try again on the humble branch?  I've made a few changes and I'm now able to run the talker example without issue.  If you do run into an issue the ssh changes I made should make it easier to find the root cause.

@jeffi, gave it a try again. Failed twice with humble branch:

Attempt 1 :x:
Attempt 2 ❌

Here're a few additional observations:

👍 there's definitely an improvement with the prompt, so @jeffi's changes definitely added good.
I think we should polish the 101 experience (the talker launchfile I demonstrate above) before exposing this to others.
humble branch is inconsistent from a ROS 2 perspective, it should upon rolling in here https://github.com/BerkeleyAutomation/FogROS2/blob/humble/Dockerfile#L1, not on top of Galactic (Humble is branching out Rolling). I believe @KDharmarajanDev took care of this in main.
we're missing in humble branch many of the changes introduced in main (sorry I sent them there, I just didn't know that humble was the actual "good" one, since generally, development happens in main and then you branch things out when ready, as appropriate). I think it'd be nice to cherry-pick many of the changes introduced in main regarding the README polishing or just rebase things together.
I think it'd be nice to default to whatever development repo we're sticking with (right now, default is main).

We branched humble since it will undergo many changes that are inconsistent with the paper.  We branched recently, so it should be straightforward to merge everything onto the humble branch.  Anything on main that is inconsistent with the paper, we may need to back out.  Sorry for not being clear about this earlier.
What are the steps you're taking to reproduce the issues you see?  Also are you using an EC2 instance near you?  Or us-west-1?

Been using us-west-1. This is known to work in the past and I would expect
bigger latencies, but still interoperability.
Do you guys get issues with machines in Europe?
El El mié, 13 abr 2022 a las 18:44, Jeff Ichnowski @.***>
escribió:

We branched humble since it will undergo many changes that are
inconsistent with the paper. We branched recently, so it should be
straightforward to merge everything onto the humble branch. Anything on
main that is inconsistent with the paper, we may need to back out. Sorry
for not being clear about this earlier.
What are the steps you're taking to reproduce the issues you see? Also are
you using an EC2 instance near you? Or us-west-1?
—
Reply to this email directly, view it on GitHub
https://github.com/BerkeleyAutomation/FogROS2/issues/19#issuecomment-1098267575,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKPYDVWCAIRVOKTBD5JBOTVE32VNANCNFSM5TASODDQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think some DDS changes have occurred. So it might be the issue.  Can you try with a closer instance?  That’s best practice anyways.  We’ll try with EU servers too

On Apr 13, 2022, at 10:41 AM, Víctor Mayoral Vilches @.***> wrote:

Been using us-west-1. This is known to work in the past and I would expect
bigger latencies, but still interoperability.
Do you guys get issues with machines in Europe?
El El mié, 13 abr 2022 a las 18:44, Jeff Ichnowski @.***>
escribió:

We branched humble since it will undergo many changes that are
inconsistent with the paper. We branched recently, so it should be
straightforward to merge everything onto the humble branch. Anything on
main that is inconsistent with the paper, we may need to back out. Sorry
for not being clear about this earlier.
What are the steps you're taking to reproduce the issues you see? Also are
you using an EC2 instance near you? Or us-west-1?
—
Reply to this email directly, view it on GitHub
https://github.com/BerkeleyAutomation/FogROS2/issues/19#issuecomment-1098267575,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKPYDVWCAIRVOKTBD5JBOTVE32VNANCNFSM5TASODDQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

I just ran the humble branch with docker, and it works fine on my end

I repeated the steps that @vmayoral mentioned. I had the same problem whenever I start fogros container the first time.
Then my theory is that if the VPN's wg0 interface exists, but /etc/wireguard/wg0 file does not exist(because the container is new), then the interface is not deleted and restarted. Then the setup does not succeed.

(Just getting the issue up to date from conversations over Slack and email)
The issue appears to stem from ROS_DOMAIN_ID.  When we set it on the robot, the setting is not transmitted to the cloud instance.  The result is that robot and cloud were talking on different port ranges, and thus never to each other.  The workaround for now is to leave ROS_DOMAIN_ID unset on the robot (to match the cloud).  The fix will be to match the  ROS_DOMAIN_ID setting on the cloud.

Tried latest changes in humble in a clean, Docker-bases setup and failed ❌ :
Error dump
root@xilinx:/home/root/fog_ws# ros2 launch fogros2_examples talker.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2022-04-16-06-52-48-253448-xilinx-69
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [root]: init
using ROS workspace:  /home/root/fog_ws
creating EC2 instance
security group id is ['sg-0e92c69af4995c85c']
i-04e0451b9131a77c3
EC2 instance is created with ip address: <IP_ADDRESS>
Hit:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:4 http://repo.ros2.org/ubuntu/main focal InRelease [4,676 B]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:6 http://packages.ros.org/ros2/ubuntu focal InRelease [4,676 B]
Get:7 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,712 kB]
Get:8 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [320 kB]
Get:9 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB]
Get:10 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [917 kB]
Get:11 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [131 kB]
Get:12 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B]
Get:13 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [918 kB]
Get:14 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [205 kB]
Get:15 http://repo.ros2.org/ubuntu/main focal/main arm64 Packages [987 kB]
Get:16 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.6 kB]
Get:17 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB]
Get:18 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,336 B]
Get:19 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B]
Get:20 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB]
Get:21 http://repo.ros2.org/ubuntu/main focal/main amd64 Packages [1,012 kB]
Get:22 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB]
Get:23 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB]
Get:24 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB]
Get:25 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [808 B]
Get:26 http://packages.ros.org/ros2/ubuntu focal/main amd64 Packages [1,012 kB]
Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,385 kB]
Get:28 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [240 kB]
Get:29 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.0 kB]
Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [859 kB]
Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB]
Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B]
Get:33 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [698 kB]
Get:34 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [124 kB]
Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.3 kB]
Fetched 11.2 MB in 2s (4,975 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
  locales
1 upgraded, 0 newly installed, 0 to remove and 173 not upgraded.
Need to get 3,871 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 locales all 2.31-0ubuntu9.7 [3,871 kB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 3,871 kB in 0s (15.4 MB/s)
(Reading database ... 172136 files and directories currently installed.)
Preparing to unpack .../locales_2.31-0ubuntu9.7_all.deb ...
Unpacking locales (2.31-0ubuntu9.7) over (2.31-0ubuntu9.2) ...
Setting up locales (2.31-0ubuntu9.7) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Generating locales (this might take a while)...
  en_US.ISO-8859-1... done
  en_US.UTF-8... done
Generation complete.
Processing triggers for man-db (2.9.1-1) ...
Generating locales (this might take a while)...
  en_US.ISO-8859-1... done
  en_US.UTF-8... done
Generation complete.
Reading package lists...
Building dependency tree...
Reading state information...
software-properties-common is already the newest version (<IP_ADDRESS>).
0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded.
'universe' distribution component is already enabled for all sources.
Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease
Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Fetched 222 kB in 1s (425 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
lsb-release is already the newest version (11.1.0ubuntu2).
curl is already the newest version (7.68.0-1ubuntu2.7).
gnupg is already the newest version (2.2.19-3ubuntu2.1).
0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded.
Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease
Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Fetched 222 kB in 1s (423 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ros-rolling-desktop is already the newest version (0.9.3-1focal.20220204.201458).
0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  python3-bson python3-bson-ext python3-tornado ros-rolling-rosapi
  ros-rolling-rosapi-msgs ros-rolling-rosbridge-library
  ros-rolling-rosbridge-msgs ros-rolling-rosbridge-server
Suggested packages:
  python3-pycurl python-tornado-doc
The following NEW packages will be installed:
  python3-bson python3-bson-ext python3-tornado ros-rolling-rosapi
  ros-rolling-rosapi-msgs ros-rolling-rosbridge-library
  ros-rolling-rosbridge-msgs ros-rolling-rosbridge-server
  ros-rolling-rosbridge-suite
0 upgraded, 9 newly installed, 0 to remove and 173 not upgraded.
Need to get 743 kB of archives.
After this operation, 8,006 kB of additional disk space will be used.
Get:1 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosapi-msgs amd64 1.1.2-1focal.20220121.214852 [249 kB]
Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-bson amd64 3.10.1-0ubuntu2 [34.8 kB]
Get:3 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-library amd64 1.1.2-1focal.20220122.073542 [44.2 kB]
Get:4 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosapi amd64 1.1.2-1focal.20220126.191225 [18.6 kB]
Get:5 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-msgs amd64 1.1.2-1focal.20220121.213906 [43.0 kB]
Get:6 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-server amd64 1.1.2-1focal.20220126.193440 [19.8 kB]
Get:7 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-suite amd64 1.1.2-1focal.20220126.193929 [6,600 B]
Get:8 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-bson-ext amd64 3.10.1-0ubuntu2 [22.3 kB]
Get:9 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 python3-tornado amd64 6.0.3+really5.1.1-3 [304 kB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 743 kB in 0s (6,784 kB/s)
Selecting previously unselected package python3-bson.
(Reading database ... 172136 files and directories currently installed.)
Preparing to unpack .../0-python3-bson_3.10.1-0ubuntu2_amd64.deb ...
Unpacking python3-bson (3.10.1-0ubuntu2) ...
Selecting previously unselected package python3-bson-ext.
Preparing to unpack .../1-python3-bson-ext_3.10.1-0ubuntu2_amd64.deb ...
Unpacking python3-bson-ext (3.10.1-0ubuntu2) ...
Selecting previously unselected package python3-tornado.
Preparing to unpack .../2-python3-tornado_6.0.3+really5.1.1-3_amd64.deb ...
Unpacking python3-tornado (6.0.3+really5.1.1-3) ...
Selecting previously unselected package ros-rolling-rosapi-msgs.
Preparing to unpack .../3-ros-rolling-rosapi-msgs_1.1.2-1focal.20220121.214852_amd64.deb ...
Unpacking ros-rolling-rosapi-msgs (1.1.2-1focal.20220121.214852) ...
Selecting previously unselected package ros-rolling-rosbridge-library.
Preparing to unpack .../4-ros-rolling-rosbridge-library_1.1.2-1focal.20220122.073542_amd64.deb ...
Unpacking ros-rolling-rosbridge-library (1.1.2-1focal.20220122.073542) ...
Selecting previously unselected package ros-rolling-rosapi.
Preparing to unpack .../5-ros-rolling-rosapi_1.1.2-1focal.20220126.191225_amd64.deb ...
Unpacking ros-rolling-rosapi (1.1.2-1focal.20220126.191225) ...
Selecting previously unselected package ros-rolling-rosbridge-msgs.
Preparing to unpack .../6-ros-rolling-rosbridge-msgs_1.1.2-1focal.20220121.213906_amd64.deb ...
Unpacking ros-rolling-rosbridge-msgs (1.1.2-1focal.20220121.213906) ...
Selecting previously unselected package ros-rolling-rosbridge-server.
Preparing to unpack .../7-ros-rolling-rosbridge-server_1.1.2-1focal.20220126.193440_amd64.deb ...
Unpacking ros-rolling-rosbridge-server (1.1.2-1focal.20220126.193440) ...
Selecting previously unselected package ros-rolling-rosbridge-suite.
Preparing to unpack .../8-ros-rolling-rosbridge-suite_1.1.2-1focal.20220126.193929_amd64.deb ...
Unpacking ros-rolling-rosbridge-suite (1.1.2-1focal.20220126.193929) ...
Setting up python3-tornado (6.0.3+really5.1.1-3) ...
Setting up ros-rolling-rosapi-msgs (1.1.2-1focal.20220121.214852) ...
Setting up python3-bson (3.10.1-0ubuntu2) ...
Setting up python3-bson-ext (3.10.1-0ubuntu2) ...
Setting up ros-rolling-rosbridge-msgs (1.1.2-1focal.20220121.213906) ...
Setting up ros-rolling-rosbridge-library (1.1.2-1focal.20220122.073542) ...
Setting up ros-rolling-rosapi (1.1.2-1focal.20220126.191225) ...
Setting up ros-rolling-rosbridge-server (1.1.2-1focal.20220126.193440) ...
Setting up ros-rolling-rosbridge-suite (1.1.2-1focal.20220126.193929) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
ssh -o StrictHostKeyChecking=no -i /tmp/fogros/272/FogROSKEYAWS167.pem<EMAIL_ADDRESS>"source /opt/ros/rolling/setup.bash && ros2 launch rosbridge_server rosbridge_websocket_launch.xml &"
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease
Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Fetched 222 kB in 1s (428 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  python3-colcon-installed-package-information python3-colcon-override-check
The following NEW packages will be installed:
  python3-colcon-installed-package-information python3-colcon-override-check
The following packages will be upgraded:
  python3-colcon-common-extensions
1 upgraded, 2 newly installed, 0 to remove and 172 not upgraded.
Need to get 15.5 kB of archives.
After this operation, 74.8 kB of additional disk space will be used.
Get:1 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-common-extensions all 0.3.0-1 [2,824 B]
Get:2 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-installed-package-information all 0.0.1-1 [7,512 B]
Get:3 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-override-check all 0.0.1-1 [5,172 B]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 15.5 kB in 0s (1,363 kB/s)
(Reading database ... 173133 files and directories currently installed.)
Preparing to unpack .../python3-colcon-common-extensions_0.3.0-1_all.deb ...
Unpacking python3-colcon-common-extensions (0.3.0-1) over (0.2.1-1) ...
Selecting previously unselected package python3-colcon-installed-package-information.
Preparing to unpack .../python3-colcon-installed-package-information_0.0.1-1_all.deb ...
Unpacking python3-colcon-installed-package-information (0.0.1-1) ...
Selecting previously unselected package python3-colcon-override-check.
Preparing to unpack .../python3-colcon-override-check_0.0.1-1_all.deb ...
Unpacking python3-colcon-override-check (0.0.1-1) ...
Setting up python3-colcon-common-extensions (0.3.0-1) ...
Setting up python3-colcon-installed-package-information (0.0.1-1) ...
Setting up python3-colcon-override-check (0.0.1-1) ...
[INFO] [launch]: All log files can be found below /home/ubuntu/.ros/log/2022-04-16-06-56-37-215579-ip-172-31-31-128-5219
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [rosbridge_websocket-1]: process started with pid [5484]
[INFO] [rosapi_node-2]: process started with pid [5486]
Reading package lists...
Building dependency tree...
Reading state information...
unzip is already the newest version (6.0-25ubuntu1).
wireguard is already the newest version (1.0.20200513-1~20.04.2).
The following additional packages will be installed:
  bridge-utils containerd pigz runc ubuntu-fan
Suggested packages:
  ifupdown aufs-tools cgroupfs-mount | cgroup-lite debootstrap docker-doc
  rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  bridge-utils containerd docker.io pigz runc ubuntu-fan
0 upgraded, 6 newly installed, 0 to remove and 172 not upgraded.
Need to get 74.2 MB of archives.
After this operation, 360 MB of additional disk space will be used.
Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 bridge-utils amd64 1.6-2ubuntu1 [30.5 kB]
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 runc amd64 1.0.1-0ubuntu2~20.04.1 [4,155 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 containerd amd64 1.5.5-0ubuntu3~20.04.2 [33.0 MB]
Get:5 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.7-0ubuntu5~20.04.2 [36.9 MB]
Get:6 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 ubuntu-fan all 0.12.13 [34.5 kB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
[rosapi_node-2] [WARN] [1650092206.223525877] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
Fetched 74.2 MB in 4s (20.7 MB/s)
[rosbridge_websocket-1] [INFO] [1650092206.311298513] [rosbridge_websocket]: Rosbridge WebSocket server started on port 9090
Selecting previously unselected package pigz.
(Reading database ... 173168 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../1-bridge-utils_1.6-2ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.6-2ubuntu1) ...
Selecting previously unselected package runc.
Preparing to unpack .../2-runc_1.0.1-0ubuntu2~20.04.1_amd64.deb ...
Unpacking runc (1.0.1-0ubuntu2~20.04.1) ...
Selecting previously unselected package containerd.
Preparing to unpack .../3-containerd_1.5.5-0ubuntu3~20.04.2_amd64.deb ...
Unpacking containerd (1.5.5-0ubuntu3~20.04.2) ...
Selecting previously unselected package docker.io.
Preparing to unpack .../4-docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb ...
Unpacking docker.io (20.10.7-0ubuntu5~20.04.2) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../5-ubuntu-fan_0.12.13_all.deb ...
Unpacking ubuntu-fan (0.12.13) ...
Setting up runc (1.0.1-0ubuntu2~20.04.1) ...
Setting up bridge-utils (1.6-2ubuntu1) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Setting up pigz (2.4-1) ...
Setting up containerd (1.5.5-0ubuntu3~20.04.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up ubuntu-fan (0.12.13) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service.
Setting up docker.io (20.10.7-0ubuntu5~20.04.2) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Adding group `docker' (GID 131) ...
Done.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for systemd (245.4-4ubuntu3.13) ...
Processing triggers for man-db (2.9.1-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
python3-pip is already the newest version (20.0.2-5ubuntu1.6).
0 upgraded, 0 newly installed, 0 to remove and 172 not upgraded.
Requirement already satisfied: wgconfig in /usr/local/lib/python3.8/dist-packages (0.2.2)
Requirement already satisfied: boto3 in /usr/local/lib/python3.8/dist-packages (1.21.6)
Requirement already satisfied: paramiko in /usr/local/lib/python3.8/dist-packages (2.9.2)
Requirement already satisfied: scp in /usr/local/lib/python3.8/dist-packages (0.14.4)
Requirement already satisfied: botocore<1.25.0,>=1.24.6 in /usr/local/lib/python3.8/dist-packages (from boto3) (1.24.6)
Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /usr/local/lib/python3.8/dist-packages (from boto3) (0.5.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from boto3) (0.10.0)
Requirement already satisfied: pynacl>=1.0.1 in /usr/lib/python3/dist-packages (from paramiko) (1.3.0)
Requirement already satisfied: bcrypt>=3.1.3 in /usr/local/lib/python3.8/dist-packages (from paramiko) (3.2.0)
Requirement already satisfied: cryptography>=2.5 in /usr/lib/python3/dist-packages (from paramiko) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.25.0,>=1.24.6->boto3) (1.25.8)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore<1.25.0,>=1.24.6->boto3) (2.7.3)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3/dist-packages (from bcrypt>=3.1.3->paramiko) (1.14.0)
Requirement already satisfied: cffi>=1.1 in /usr/local/lib/python3.8/dist-packages (from bcrypt>=3.1.3->paramiko) (1.15.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko) (2.21)
/home/root fog_ws
removing old workspace
mv: cannot move 'fog_ws' to a subdirectory of itself, 'fog_ws/fog_ws'
successfully extracted new workspace
[ERROR] [launch]: Caught exception in launch (see debug for traceback): No module named 'fogros2.actions'

I'll be taking a few days off but will re-engage with the discussion next week!

Superseded by #36


AMI doesn't start ES?

Saw this in the AMI build log:
[18:26:57][Step 1/1]     amazon-ebs: ### NOT starting elasticsearch by default on bootup, please execute
[18:26:57][Step 1/1]     amazon-ebs: sudo update-rc.d elasticsearch defaults 95 10

Suspect we should do something about that.

Fixed in #45. See https://github.com/guardian/elk-stack/blob/master/cloudformation/ELK_Stack_Multi_AZ_in_Private_VPC.json#L407


ed25519::Pair::from_standard_components returns unexpected Err when derivation path contains a "Soft" junction

When a the derive path passed to ed25519::Pair::from_standard_components contains a soft junction, the function returns an error. The function does not document the error case, leaving the user to discover it at runtime.
The behavior is different from sr25519::Pair::from_standard_components, which does allow Soft junctions.
Suggested fix: Either change ed25519::Pair to allow soft junctions (like sr25519::Pair does), or document the error case as recommended in the excellent rust api guidelines.

cc @burdges

Just fyi, there are several poor designs for soft derivations on Ed25519 around:
https://forum.web3.foundation/t/key-recovery-attack-on-bip32-ed25519/44
I implemented a reasonable design in https://github.com/w3f/hd-ed25519 but never really polished it.  And Tor has another good. design.  I'd prefer if we do not rush the hd-ed25519 crate into production but if we actually need it anywhere then we can clean it up
@jacogr might know if we should do this sooner or if it can wait.

Soft paths are not implemented for ed25519 - specifically for the reasons  mentioned above, i.e. not to rush a half-baked version. It is intentional that the code returns an error here (unlike sr25519), i.e.
https://github.com/paritytech/substrate/blob/4758636176b2949dd2f7921c85c840b08d99b0eb/core/primitives/src/ed25519.rs#L408
As it stands, it is up to the callers (i.e. be it subkey or a wallet or a key generator), to not allow the soft paths in ed25519, while allowing hard. If a solid solution becomes available, we should adapt, but it is not something that probably should be rushed into production tomorrow.


Comparer/ComparerCollection isel

I thought this was something different when reading the title of the PR. The isel() method that had in mind would wrap the corresponding xarray.Dataset.isel method (or like pandas iloc).
Doesn't the sel() method already do what you have in this PR?


potential changes to the model for consideration

Sorry for combining a bunch of things in 1 issue, but maybe this way it's easier to ignore.
I intended to spend some time in the past year exploring a bunch of issues that might be worth considering for the 2020 hake assessment but have only found time to do so very recently.
Three issues that I thought would be worth consideration by the JTC are

semi-parametric selectivity (recurring sensitivity in the past 2 years)
the use of zero-centered vs. simple deviation parameters for recruitment
putting a prior on the Dirichlet-Multinomial parameters to aid MCMC convergence

Here's a little bit more detail on what I've learned so far about each of these. I've uploaded some models with 24-million mcmc chains to /Hake JTC 2020/models/explorations_from_Ian/ on Google Drive. I put some comparison plots up there too.
1. Semi-parametric selectivity
I tried estimating the sigma_s parameter as well as fixing to 0.7 and 1.0. The estimated value was 0.77. In all cases, the recent recruitments seem implausible with the stock increasing very rapidly at the end of the time series. It would take some work to understand why the different parameterization of selectivity would be causing this, so in spite of the benefits of mcmc convergence (the diagnostics I looked at are indeed better for these semi-parametric selectivity models), I would be cautious about pursuing this option right now.
2. zero-centered vs simple deviations for recruitment
See discussion here: https://github.com/admb-project/admb/issues/107 about a bug in ADMB that hasn't yet been fixed. Cole is a smart guy and I think I'm finally convinced me that rather than try to fix ADMB, we should stop using zero-centered devs. Switching to "simple deviations" just requires changing line 83 of the control file to 2 #do_recdev:  0=none; 1=devvector; 2=simple deviations.
The impact on the model is not much (B0 goes down a bit while the rest of the time series is almost the same, so relative spawning biomass goes up a bit). Ultimately, the justification for the zero-centering of the main period of recruitment devs isn't that strong and going to simple devs seems better than using results that include a known bug (even if the impact of the bug from what I can tell is even more trivial than the change in structure to simple devs).
3. Prior on the Dirichlet-Multinomial parameter
The transformation of the log(Theta) parameter described under 2.4.4 Age composition likelihood in the 2019 hake assessment means that a uniform prior on log(Theta) implies a U-shaped distribution for the data weights. Inspired by Jim Thorson, I've been exploring a beta prior which would result in a flatter transformed distribution of the weights but haven't come up with the answer yet. I think the impact would be minimal on both fishery and survey weighting in the MLE context, but the prior would keep the survey weight from drifting to high values associated with weights close to 1.0 which required fixing log(Theta_survey) at the MLE to aid MCMC convergence. I can work more on this in time to explore in the assessment.
In summary, my suggestions (which you're welcome to ignore), are:

[ ] keeping semi-parametric selectivity as a sensitivity unless you have lots of time to explore further
[ ] switching line 83 in the control file to change to simple recruitment deviations
[ ] wait for info about a proposed prior on log(Theta) and consider that as an alternative to fixing the survey value for MCMC

Thanks Ian, this is a great summary and very useful information moving forward.  I appreciate you sharing it and continuing to stay involved in hake related research!
Regarding #2, this will then relieve the potential 'drifting' of rec devs in mcmc's (i.e., the median rec devs from mcmc don't necessarily have to, and in fact currently never would do so exactly, sum to 0 as they do as parameter dev_vector when doing MLE), correct?  Or no, because simple deviations don't have any penalty....?

The simple deviations (implemented as a bounded_vector rather than a bounded_dev_vector in ADMB) contain the penalty associated with sigmaR that draws all recruitment deviations toward zero in the absence of other information, but don't have the separate constraint that the "main" period has to sum to zero.
The theory behind the use of the dev_vector with the zero sum is that the R0 and B0 reference points should be reflective of the central tendency for the time period with the best information about recruitment. This prevents the model from having "all the children are above average" as in Lake Wobegon. In theory, a mismatch between R0 and the main recdev period could lead to a better fit to the indices of abundance or some other data source which you wouldn't expect to be informative about recruitment. However, for this stock the index is pretty consistent with the recruitments and that tension doesn't seem to be present. The calculations below show that calculating the mean of the main recdevs within each of the 2000 MCMC samples gets you a distribution with median = 0.086 as compared to the status-quo approach where that would be forced to equal 0. I believe that 0.086 is a pretty insignificant difference from 0.
The issue with the ADMB sampling procedure is that the zero-sum constraint isn't actually being applied in the mcmc phase where the samples are being drawn, just the mceval phase where the results get calculated. So there's a mismatch between the samples that get reported and the sampling process that was supposed to draw them. I tried re-compiling ADMB with the proposed fix (which Cole was skeptical about) and the difference was also pretty minimal for the hake model. But even if that fix proves to be reasonable, at this stage in the 2020 hake assessment, an unproven modification to ADMB seems like a less stable path forward than the alternative approach of using simple recruit devs instead of the zero-sum dev_vector.
# read 2019 model modifieid to have simple devs instead of dev_vector
model <- SS_output(file.path(mydir, 'hake2019_<IP_ADDRESS>_simple_devs'),
                   dir.mcmc = 'mcmc')

# get matrix of main recdevs (1970-2017) for each of the 2000 MCMC draws
main_recdev_matrix_MCMC <- model$mcmc[ , grep("Main_RecrDev", names(model$mcmc))]
# get MLE values of main recdevs
main_recdev_vector_MLE <- model$parameters[grep("Main_RecrDev",
                                                rownames(model$parameters)),
                                           "Value"]
# get mean of main recdevs for each of the MCMC draws
main_recdev_means_MCMC <- apply(main_recdev_matrix_MCMC, MARGIN = 1, FUN = mean)

# summary statistics
mean(main_recdev_vector_MLE)
## [1] 0.213518
summary(main_recdev_means_MCMC)
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
## -0.50835 -0.02647  0.08643  0.09200  0.21289  0.72256

Thanks @iantaylor-NOAA. I moved your exploration relating to (2) into the main model folder for 2020 and labeled it 2019.03.92_simpledevs. I hope that is okay. I also added the model to the spreadsheet of 2020 models.
Below is some text that I drafted for the assessment regarding the change. I think it might be too long winded. If others can please edit it for me, that would be great. Just copy and paste it into a comment and edit away.
The parameterization of recruitment deviations was changed from using a zero-centered vector of deviations during the main period to a simple vector that allowed the estimates from the main period to have a non-zero sum. This change to a simple vector was necessary because a mismatch was found between the mean of the estimates from the MCMC sampling algorithm and the MCMC evaluation algorithm in ADMB. A simple vector avoids this mismatch caused by ADMB and allows the penalty towards zero to operate on the entire vector of estimated recruitment deviations rather than assuming the main period should sum to zero. Penalties have implications for calculating derivatives, and thus, we believe the use of a simple vector is currently the best way forward. Additionally, removal of the mismatch in the MCMC protocol should, in theory, lead to increased similarity between the MLE and MCMC results.

I think we can close this for now, even though it has helpful explanations in here, we can always search it back out.


[ci] Require full tag names in genrelnotes.py script

This allows us to get temporary release notes with any tag or HEAD, for example:
./ci-scripts/genrelnotes.py v2.17 HEAD

@jenkins-cscs retry dom

lgtm

@jenkins-cscs retry none

Codecov Report

Merging #807 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #807   +/-   ##
=======================================
  Coverage   91.51%   91.51%
=======================================
  Files          78       78
  Lines        9791     9791
=======================================
  Hits         8960     8960
  Misses        831      831

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e68ef06...01f7759. Read the comment docs.


ci: Fix dependabot grouping

Fixes #N/A
Description

Fix dependabot grouping dependencies with exclude-patterns to ignore k8s deps on all other dependency gathering

How was this change tested?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.05%) to 81.392%

Files with Coverage Reduction
New Missed Lines
%

pkg/controllers/provisioning/scheduling/topology.go
2
86.49%

pkg/controllers/provisioning/scheduling/topologygroup.go
2
96.75%

Totals

Change from base Build<PHONE_NUMBER>:
-0.05%

Covered Lines:
7016

Relevant Lines:
8620

💛  - Coveralls


Ordering of issues doesn't update unless you press search

Not sure if this is an upstream issue or if it is a DGU specific one.

Is that on the dataset issues page, organization issues page or moderation page?


Merging from master for conf branch

Merging changes from master to prepare for delivering new conf work to master (see https://tachyon.atlassian.net/browse/TACHYON-8)

CC @haoyuan

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/2581/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/2582/
Test PASSed.


[ERROR] Could not initialize AOA

I am using Debian from Microsoft Store.
Followed the steps on the guide completely.
As i execute "usbaudio" it shows the error

[ERROR] Could not initialize AOA

aoa_init() basically calls libusb_init(). So you can't initialize libusb.
I'm not so surprise if you use it from Windows (even if it runs some Linux above).

aoa_init() basically calls libusb_init(). So you can't initialize libusb.
I'm not so surprise if you use it from Windows (even if it runs some Linux above).

So,... i am sorry but i have no idea what to do, i am new in all this :\

So no luck on Windows?? :/
welp, thankyou for making this at least there is something.

Windows Subsystem for Linux doesn't have USB access. There's nothing you can do there.


Use posix filepaths

Writing to target fails when using windows paths separator

Hi @luisvgo! Sorry it took me so long to get to your PR :(
Could you please visit https://www.jetbrains.com/agreements/cla/ and electronically sign the contributor license agreement?

Document signed
Thanks for your work and time spent on this plugin.
All the best - Luis

@luisvgo Thank you for your pull request!


Document How To Add Your Project page

Overview
Document file and code structure for How To Add Your Project page
Action Items

[ ] Document file and code structure for How To Add Your Project page

Resources/Instructions
Civic Tech Index Site Architecture documentation can be updated here

@bhaggya your documentation covers most of the How to Add page pretty well.  Just a couple items to update.  I would mention the actual HowToAdd component since that's the main component that's used to display the page content.  Also you could add markup for any component/file/variable names so that they stand out.  See the Home page documentation as an example, which is marked up all over the place.  Thanks!


Exception when Executing gremlin query to add unique vertex

Environment Properties:

Windows 10
Visual Studio 2017 Community
Azure Function Targeting .NET Standard 2.0

Packages:

Microsoft.Azure.DocumentDB.Core
Microsoft.Azure.Graphs
Microsoft.Azure.WebJobs.ServiceBus
Microsoft.NET.Sdk.Functions

I am running this query against a Cosmos Graph DB:
g.V().has('listing','id','48607473').fold().coalesce(unfold(),addV('listing').property('id','48607473'))
When I execute the query directly in the Data Explorer on Azure Portal, I see the vertex added, but when I execute in my C# azure function, I receive the below exception in the Azure function Console on my local dev machine.
System.Private.CoreLib: Exception while executing function: LeadParse. System.Private.CoreLib: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Method not found: 'Double Microsoft.Azure.Documents.Helpers.GetHeaderValueDouble(System.Collections.Specialized.NameValueCollection, System.String, Double)'.))). System.Private.CoreLib: One or more errors occurred. (One or more errors occurred. (Method not found: 'Double Microsoft.Azure.Documents.Helpers.GetHeaderValueDouble(System.Collections.Specialized.NameValueCollection, System.String, Double)'.)). System.Private.CoreLib: One or more errors occurred. (Method not found: 'Double Microsoft.Azure.Documents.Helpers.GetHeaderValueDouble(System.Collections.Specialized.NameValueCollection, System.String, Double)'.). Microsoft.Azure.Graphs: Method not found: 'Double Microsoft.Azure.Documents.Helpers.GetHeaderValueDouble(System.Collections.Specialized.NameValueCollection, System.String, Double)'

@olivertowers this is gremlin related. can you please take a look?

I can reduced the failing query down to:
g.v('bill')
Stack trace of inner exception:
at Microsoft.Azure.Graphs.GraphCommand.UpdatePartitionCost(NameValueCollection headers) at Microsoft.Azure.Graphs.GraphCommand.MergeResponseHeaders(NameValueCollection newResponseHeaders) at Microsoft.Azure.Graphs.GraphConnection.d__77.MoveNext()

Any solution for this error?

@cwoodland I was able to resolve this by having these 2 packages:

Adding DocumentDB helped to solve an unrelated issue I believe, but was necessary in my test query. I suspect the main problem lives in 2.0.0-preview of DocumentDB.Core.


Please document 'size' parameter

// Matcher<false> m(tvecs, size, qvecs, size, threshold, max_twiddles);

And for extra points, if you could kindly clarify threshold, is that the popcount difference (in %?) or the sum of hamming distance difference between any two 512bit binary descriptors?

So the thing for me is - why is there size twice here?

// Example initialization of Matcher class
// Matcher<false> m(tvecs, size, qvecs, size, threshold, max_twiddles);
//
// Options:

Apologies...
	// Number of training vectors
	int tcount;

	// Incoming 512-bit query vectors
	const void* __restrict qset;

	// Number of query vectors
	int qcount;

Yep, documentation at the parameters rather than the arguments! I agree with you that the use of 'size' to describe both vector sizes in the main.cpp is ambiguous and should be separated out. I'll do that at some point. The main.cpp is just a test driver that I threw together very quickly, and all the actual engineering effort, including proper documentation, is in LATCH.h. But I'll work on cleaning up main.cpp as well. Thanks!

And for extra points, if you could kindly clarify threshold, is that the popcount difference (in %?) or the sum of hamming distance difference between any two 512bit binary descriptors?

Sure! From the README:

a match is returned if the best match between a query vector and a training vector is more than a certain threshold number of bits better than the second-best match. Yes, that means the DIFFERENCE in popcounts is used for thresholding, NOT the ratio. This is the CORRECT approach for binary descriptors.

I know this can be confusing to people unfamiliar with 2NN, and it takes a surprising amount of effort to explain in English words. I sent you that long email with a proper explanation. For anyone else who might end up reading this thread, I'll reproduce my full explanation from that email:

So we have descriptions of all the interesting features in each image: for each feature in the first image (which we call the training), we want a way to find which corner, if any, from the second image (which we call the query), is a different view of the same physical thing. Let's take our first training feature for example. How do we find which query feature, if any, matches? The naive but wrong way to do this is to compare all the query descriptors against the training descriptor, and consider it a match if it differs by less than some threshold number of bits, say 50. In technical terms we consider it a match if the Hamming distance is less than 50. This is bad. That's because comparison should never be absolute, only relative. Imagine if you get 100 good matches this way. Then in the next frame the camera is bumped a little so the features are a bit blurry. All the matches drop in quality, and you now get 0 matches. But the matches are still exactly the same relative to each other; they've just all dropped in absolute quality. You could still get good results if you had just done relative comparison instead of absolute. So: instead, the right approach is to find the best and second-best matches to the training descriptor. Then don't do an absolute comparison between the query and training; it doesn't matter. Instead, consider the best match to be a match, if its Hamming distance to the training descriptor is less than the second-best match's Hamming distance to the training descriptor by at least some threshold. I know that's confusing but there's no other way to explain the technical process. Make sure it makes sense. This may help: a nontechnical and intuitive explanation of that process and why it works. Instead of considering a query descriptor a match if it's significantly similar to the training descriptor, consider a query descriptor a match if it is a significantly better match than any other query descriptor. Anyway, so that's second-nearest-neighbor, or 2NN, matching, and why it works. That's what K2NN and CUDAK2NN do. Oh, also, notice I use the number of bits by which they differ, i.e. the difference in Hamming distances, and not the ratio of Hamming distances. Punch anyone who tells you you should use the ratio test in the face. They don't know what they're talking about.

So the threshold is not the maximum Hamming distance from Q to T for T to be considered a match; that would be nearest-neighbor matching, and we're doing second-nearest-neighbor (2NN) matching.
Therefore, the threshold is the minimum number of bits by which the best-matching T's Hamming distance to Q must exceed the second-best-matching T's Hamming distance to Q, for the best-matching T to be considered a match with Q.
Make sense? :D
Hope that helps!
-Kareem

yes, so lets assume I have 8 bit descriptors for argument sake as my training data
00011111
00001111
00000011
00000001

And my query is 00000111 and with a threshold of 1, I can expect to see
00001111
00000011

I think this is correct

Using an example is a great idea! I should have thought of that for helping explain. Thanks!
Let's see... okay, not quite. So taking your example, we have training:
00111111
00000001
00000000

And query of 00001111. Okay, this means the Hamming distances, in order, will be:
2 3 4
The best match (index 0), then, has a Hamming distance of 2. The second-best match (index 1) has a Hamming distance of 3. Therefore the best match beats the second-best match by 1 (a.k.a. the difference is 1.) The threshold is 1. 1 is not greater than 1, so no match would be returned.
Let's take another example. same training set, but a query of 01111111. Now the Hamming distances are:
1 6 7
The best match has a Hamming distance of 1. The second-best match has a Hamming distance of 6. The difference is 5. 5 is greater than 1 so index 0 would be returned, since the best match is at index 0.

Remember that we aren't just returning the best match if it's less than threshold away; we return the best match if it beats the second-best match by more than threshold.

It's sort of like getting a signal-to-noise ratio. What does it matter if this training is only 3 away, if there are 10 other trainings that are only 3 away? Only one of them can be correct, so that by definition means we don't have a reliable match. However, even if this training is 30 away, if the next-best match is 200 away, we can be pretty confident that's a match. That's why we use this second-order approach.


Fix applying damage to starships throwing an error

Currently applying damage to a starship throws an error, saying ChoiceDialog is undefined. Adding an import fixes that and it behaves as expected.

Thanks for this one. I'm going to see if I can release a hot fix for this one, since doing damage in starship combat is basically broken without this 😬 .


TS compilation fails on Long

Hi,
Nice project. I got it working mostly. I had some compiler errors fr protobufjs/minimum, so I had to update version to 6. Now  after generating TS files from Proto, I'm getting below error:
lib/proto/google/protobuf/empty.ts:78:5 - error TS2367: This condition will always return 'true' since the types 'Constructor<Long>' and '{ default: LongConstructor; prototype: Long; MAX_UNSIGNED_VALUE: Long; MAX_VALUE: Long; MIN_VALUE: Long; NEG_ONE: Long; ONE: Long; ... 11 more ...; fromValue(val: string | ... 2 more ... | { ...; }): Long; }' have no overlap.

78 if (util.Long !== Long) {
       ~~~~~~~~~~~~~~~~~~

Version information
Typescript version: 4.2.4
NodeJS: v10.19.0

This error is repeated in other TS files which are generated from proto files as well. Any idea how can I fix this?
Let me know if you need anymore information from my side.

This looks like an issue with ts-proto.
My guess is that you have multiple versions of long package installed. I recommend adding long to the list of dependencies of your app, then checking yarn.lock or package-lock.json for duplicates of long.

Hello!
Have a look at the compiled .ts file (at the bottom):
// If you get a compile-error about 'Constructor<Long> and ... have no overlap',
// add '--ts_proto_opt=esModuleInterop=true' as a flag when calling 'protoc'.
if (util.Long !== Long) {
  util.Long = Long as any;
  configure();
}

You have two approaches to fix this issue:

invoke protoc with the esModuleInterop=true flag
disable esModuleInterop in your tsconfig

@reolat,
Thank you a lot. This nudges me back to my good old habit of reading things more carefully. Closing this


Bump elixir_sense

Closes https://github.com/elixir-lsp/elixir-ls/issues/126
Closes https://github.com/elixir-lsp/elixir-ls/issues/132
And other improvements

Thanks! :heart:


Option to use it as a cli for local repo

I think it would be useful to have this tool as as cli

I would love to be able to point local repo in fact!

after i stabilize the deployment costs, this looks like it would be the #1 feature people are asking, to run it on local / private repos

Hey! I am down to work on this


checker(dm): init checker concurrently

What problem does this PR solve?

Issue Number: close #5098
What is changed and how it works?
Check List
Tests

Unit test
Manual test (add detailed scripts or steps below)

Code changes
Side effects
Related changes
Release note

`support init checker concurrently`.

/run-all-tests

/run-dm-integration-tests

Codecov Report

Merging #5103 (08794f8) into master (687e248) will increase coverage by 0.0001%.
The diff coverage is 57.7591%.

Flag
Coverage Δ

cdc
61.5283% <52.2222%> (+0.9296%)
:arrow_up:

dm
51.8917% <81.5699%> (-0.5779%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
@@               Coverage Diff                @@
##             master      #5103        +/-   ##
================================================
+ Coverage   56.1240%   56.1241%   +0.0001%
================================================
  Files           522        524         +2
  Lines         65325      67397      +2072
================================================
+ Hits          36663      37826      +1163
- Misses        25094      25851       +757
- Partials       3568       3720       +152

/merge


Touch friendly viewer

Awesome project, allows building tools that make sharing and collaborating on 3D objects easier.  Much needed!
Any plans or efforts on making the viewer usable on touchscreen devices?
This would be my feature request.

There is literally no code commit since 6 years, so I think the answer is no :p

See www.openjscad.org

http://vectorslave.com/csg/CSGShinyDemo.html

@manthrax with newer 3js you can now do this, which is way faster, but does not scale very well.

https://github.com/jscad/csg-viewer
This one is touch friendly, hurray!


Add pixie.js wherever there are animations, moving etc.

It is efficient game like engine for js.

Make sure that you perform extensive performance tests once you add this and compile this library into our app!
As a "webapp" (pixijs.com website opened directly) it seems awfully slow. Each and every example pulses "P" (loader) for at least 3-4 seconds before actually showing some render. And PC's fan gets crazy even though its 13th Gen Intel i7-1355U + 32 GB here!
Task Manager in Windows 11 shows CPU consumption boosting up to 15-20% from 0.2-0.5% when only switching from GitHub to their website opened in another tab. And to even up to 40-50% when rendering some new scene / example.
I don't know how about you, but I'd call that a terrible performance! :)

The three.js and pannellum.org gets roughly 10-15% with a very rare situations when getting picks up to 20%. Rendering is immediate, without any loader or waiting. But these are both 3D libraries, so I don't know, if you can adapt them into 2D game.
There's Box2D library (blog post, library) for 2D physics, but doesn't have any on-line examples, so I can't check performance.
And there's this 10+ years old article, if you'd like to consider crafting own solution based on canvas.

Okay, that scraps it up basically :D We do not want that for sure.
I'll write own animations engine :)

Yeah, that was terrible idea -> react alone will probably will quicker. If performance for React will be poor I then use special library for that.


Flow errors when using union types

I have been trying to use union types when working with redux, but I am getting an error.
Accessing action.payload.id inside the state.map() call causes an error, however destructing outside of the map doesn't cause an error.
Example flow link.
/* @flow */

type NotificationsType = Array<{id: string, show: boolean}>;

type ActionType =
  { type: 'SUCCESS_NOTIFICATION', +payload: { +message: string } } |
  { type: 'HIDE_NOTIFICATION', +payload: { +id: string } } |
  { type: 'ANOTHER_THING' }
;

export default function reducer(state: NotificationsType = [], action: ActionType): NotificationsType {
    switch (action.type) {
      	case 'SUCCESS_NOTIFICATION':
        	return [...state, { id: 'FOOBAR', show: true }]

        case 'HIDE_NOTIFICATION': {
            const { payload } = action;
            const { id } = payload;

            return state.map((notification) => {
              	if (action.payload.id !== notification.id) {
                	return notification;
                }

                if (id !== notification.id) {
                    return notification;
                }

                return { ...notification, show: false };
            });
        }

        default:
            return state;
    }
}

Flow 0.41.0 errors with.
21:               	if (action.payload.id !== notification.id) {
                              ^ property `payload`. Property not found in
21:               	if (action.payload.id !== notification.id) {
                       ^ object type

The problem here is that whenever you do a function call, Flow removes all refinements to the types. So as soon as you call .map Flow reverts back to the unrefined type for action.
This is because Objects are mutable in Javascript and any function call may have changed the value of the Object.
However, since you marked you actions as covariant, maybe the type refinements should work differently.

As of right  now, you've got the right idea, which is to store your refined types in variables and use those instead.

I've stumbled into the same problem, example code:
Flow version 0.48.0
// @flow

type Diagram = {
  name: string,
  dateCreated: number,
  dateModified: ?number,
  xml: string,
  id: string,
};

type Diagrams = Array<Diagram>;

type DiagramsState = {
  diagrams: Diagrams,
};

type DiagramsAction =
  | {|
      +type: 'SET_DIAGRAMS',
      +payload: Diagrams,
    |}
  | {|
      +type: 'ADD_DIAGRAM',
      +payload: Diagram,
    |}
  | {|
      +type: 'EDIT_DIAGRAM',
      +payload: Diagram,
    |};

const diagrams = (state: Diagrams, action: DiagramsAction): Diagrams => {
  switch (action.type) {
    case 'SET_DIAGRAMS':
      return action.payload;

    case 'ADD_DIAGRAM':
      return [...state, action.payload];

    case 'EDIT_DIAGRAM':
      return state.map(diagram => {
        return diagram.id === action.payload.id ? action.payload : diagram;
      });

    default:
      return state;
  }
};

41:         return diagram.id === action.payload.id ? action.payload : diagram;
                                                 ^ property `id`. Property not found in
41:         return diagram.id === action.payload.id ? action.payload : diagram;
                                  ^ Array
41:         return diagram.id === action.payload.id ? action.payload : diagram;
                                                      ^ array type. This type is incompatible with
11: type Diagrams = Array<Diagram>;
                          ^ object type

https://flow.org/try/#0PTAEAEDMBsHsHcBQiAuBPADgU1AEQJYCGA5gE6EC2oAvKAN6KigB2lWAXKAM4qn7PEANI1AATQiiwBhUlglZRnZgFcKAIyylhTcZICysUfkj4FnAPwr1m7aAAeFaJx58Bt-Iu69+QxAF8AbmR0bDwiMkouGlAAQVJyNAAeAhJyCgA+INRMHBSIii4AZRR5aIYdcLSuTjyq4UDgnLDUyJiAYxR8WGYaEQAfej6RJiYAahCOUAByQoBRABUAfVwASRiAcQAlGL1CqdsR0FGMQjQ4Qk9ayIPQPr9+weGR8ZzOKZjcXGW1rZ39p7GJzOsAuNUqlBudwedCGhzGEzes1WS1WG22en+cKOQPOl3BFEhDUQbW6PDE+KitAAFDx5GCWgVBKBCB0usx6fkuO1Ot0AJQcqo0dL0ERceD4FBtAAWoCpLJ5zAAdBNeSLDm1CFwcDMFt80Ts9uwAaBZChlKQevK2YqcSDRFl1ZrtR8vqjfhijVjTeaegBtRUB2mSJlW7o2064gC6DpGGq10yRKxRP3RU09cO9Fq88kVFEIGCpRgZQrVWJNWDNWaL+UVHho1FooaVtoutdEoHMzNZYZb7c41bSQSxfl5MZ0WEghGU0BQ6cOmZ6QawQ9A90CQA

I'm having the exact same problem. Sample code:
type PERMISSIONS_ADDED_TYPE = {
  type: 'PERMISSIONS_ADDED',
  payload: Array<string>
}
type PERMISSION_REMOVED_TYPE = {
  type: 'PERMISSION_REMOVED',
  payload: string
}
type PERMISSIONS_CLEARED_TYPE = {
  type: 'PERMISSION_REMOVED'
}
type PERMISSION_ACTION_TYPE =
  | PERMISSIONS_ADDED_TYPE
  | PERMISSION_REMOVED_TYPE
  | PERMISSIONS_CLEARED_TYPE

type State = Array<string>;

const permissions = (state: State = [], action: PERMISSION_ACTION_TYPE): State => {
  const { type, payload } = action;
  switch(type) {
    case types.PERMISSIONS_ADDED: {
      if (typeof payload !== 'undefined') {
        return [...state, ...payload];
      }

      return state;
    }
    case types.PERMISSIONS_CLEARED: {
      return [];
    }
    case types.PERMISSION_REMOVED: {
      return state.filter(
        cpermission => cpermission !== payload
      );
    }
    default:
      return state;
  }
}

And got the following error:
Error: src/scripts/reducers/permissions.js:8   8:   const { type, payload } = action;                      ^^^^^^^ propertypayload. Property not found in   8:   const { type, payload } = action;                                  ^^^^^^ object type
To circumvent this issue I replaced type PERMISSIONS_CLEARED for the following:
type PERMISSIONS_CLEARED_TYPE = {
  type: 'PERMISSION_REMOVED',
  payload?: Object
}

But I feel this is not ideal. I thought union type would take care of this.

I think that in your case the error is legit. You're attempting to access the payload attribute of a type that potentially doesn't have that attribute, before the refinement.
Also, I found that most type errors I get come from any function/method call, which apparently makes flow very suspicious and requires more comforting assurances.
Try this refactor:
type PERMISSIONS_ADDED_TYPE = {
  type: 'PERMISSIONS_ADDED',
  payload: ?Array<string>
}

type PERMISSION_REMOVED_TYPE = {
  type: 'PERMISSION_REMOVED',
  payload: string
}

type PERMISSIONS_CLEARED_TYPE = {
  type: 'PERMISSION_REMOVED'
}

type PERMISSION_ACTION_TYPE =
  | PERMISSIONS_ADDED_TYPE
  | PERMISSION_REMOVED_TYPE
  | PERMISSIONS_CLEARED_TYPE

type State = Array<string>;

const permissions = (state: State = [], action: PERMISSION_ACTION_TYPE): State => {
  switch(action.type) {
    case types.PERMISSIONS_ADDED: {
      if (typeof payload !== 'undefined') {
        return [...state, ...action.payload];
      }
      return state;
    }

    case types.PERMISSIONS_CLEARED: {
      return [];
    }

    case types.PERMISSION_REMOVED: {
      const { payload } = action;
      return state.filter(
        cpermission => cpermission !== payload
      );
    }

    default:
      return state;
  }
}

@DrummerHead thanks! that was a great explanation! I didn't even know about the refinement concept. Thank you again.


window close event can't be canceld

when the "x" of the window is clicked,
because it can't be canceled at sdl level.
Tried using SDL_SetEventFilter but it just prevent the event from going to the event queue,
it doesn't prevent the internal state modification.
Maybe it could be done if we used a "native" window + SDL_CreateWindowFrom,
if the native gui toolkit has a cancelable closing.
But not sure if that would allow the sdl window events to still work.

I thought this was possible if you ignore SDL_QUIT events?

No it's received after the last window is closed, that allows the application to run even with no window open, but doesn't prevent any window from closing.


Automated Grammar Specs

test.rb was just a quick hack to check syntax highlighting. We should create real grammar specs like Atom's built-in language-* packages (e.g. in language-ruby)

Working on this..


Unable to specific older versions of extension.

Apologies if this is raised in the wrong place.
Unable to download versions of this extension other than the latest and versions older than (or equal) to 0.3.23.
Examples:
Working:
https://ms-python.gallery.vsassets.io/_apis/public/gallery/publisher/ms-python/extension/python/2020.3.71113/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
Working:
https://ms-python.gallery.vsassets.io/_apis/public/gallery/publisher/ms-python/extension/python/0.3.23/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
Not working:
https://ms-python.gallery.vsassets.io/_apis/public/gallery/publisher/ms-python/extension/python/2020.3.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
Not working:
https://ms-python.gallery.vsassets.io/_apis/public/gallery/publisher/ms-python/extension/python/2020.2.1/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
Not working:
https://ms-python.gallery.vsassets.io/_apis/public/gallery/publisher/ms-python/extension/python/2019.11.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage
It seems that any non-latest versions which have the YEAR.RELEASE pattern are not working.

We don't control these links and you would have to ask the Marketplace about downloading it.
If you want to use an older version of the extension you can select it through VS Code itself. Otherwise you could also check out this repository to the appropriate tag and build the VSIX manually.


Error creating a process: 2

I'm getting this error after installing this plugin a several LSP servers.  It seems to happen with all LSP servers that I installed.  Is a reboot required or something?
02/01/23 11:20:12 [ERROR] C:\Users\username.config\lite-xl/plugins\lsp\server.lua:258: Error creating a process: 2. at ...name\AppData\Local\Programs\Lite XL\data/core/init.lua:1226
stack traceback:
[C]: in function 'process.start'
C:\Users\username.config\lite-xl/plugins\lsp\server.lua:258: in function 'plugins.lsp.server.new'
...name\AppData\Local\Programs\Lite XL\data/core\object.lua:54: in function 'plugins.lsp.symbolresults.__call'
C:\Users\username.config\lite-xl/plugins\lsp/init.lua:683: in function 'plugins.lsp.start_server'
C:\Users\username.config\lite-xl/plugins\lsp/init.lua:2047: in function <C:\Users\username.config\lite-xl/plugins\lsp/init.lua:2046>
[C]: in function 'xpcall'
...name\AppData\Local\Programs\Lite XL\data/core/init.lua:1225: in function 'core.try'
(...tail calls...)

Related to: https://github.com/liquidev/lintplus/issues/28

I think this was fixed by #69. Feel free to reopen if that's not the case.


Open search results in editor not showing results

Steps to Reproduce:

Search a term in Notebooks View
When results load on the side bar, click on Open in Editor
Results are not loaded in the editor

Not a regression. May want to disable the results editor in this scenario.


[release-v1.79] Automated cherry pick of #8639: Bump golang from 1.21.2 to 1.21.3

/kind enhancement
Cherry pick of #8639 on release-v1.79.
#8639: Bump golang from 1.21.2 to 1.21.3
Release Notes:

/lgtm
/approve


Rename box to boxplot

Request
Rename .box to .boxplot
Motivation
I'm updating the core.py docstring of box. When comparing to the Pandas .plot documentation, I can see they call it boxplot.
Thus users familiar with the Pandas .plot api will not be able to frictionless use .hvplot.
See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.boxplot.html

I can see that Pandas also have a .plot.box method that we are consistent with.
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.plot.box.html


I cant install sqlite 3 on a pi

I have a pi 4 model b and every time I run npm i better-sqlite3 my pi ends up crashing

It's impossible for us to help you with this little information. And it's very unlikely that this is an issue with the better-sqlite3 package.
My guess is that you are running out of memory during compilation. Make swap bigger.

Closing due to inactivity and a lack of information given


Grid: colspan, rowspan support

What I want to achieve:

variant 1
Use separate grid per each header row with ScrollSync:
<Grid
  className={styles.HeaderGrid}
  columnWidth={columnWidth*3}
  columnsCount={columnsCount / 3}
  height={rowHeight}
  overscanColumnsCount={overscanColumnsCount}
  renderCell={this._renderSpannedHeaderCell}
  rowHeight={rowHeight}
  rowsCount={1}
  scrollLeft={scrollLeft}
  width={width}
/>
<Grid
  className={styles.HeaderGrid}
  columnWidth={columnWidth}
  columnsCount={columnsCount}
  height={rowHeight}
  overscanColumnsCount={overscanColumnsCount}
  renderCell={this._renderHeaderCell}
  rowHeight={rowHeight}
  rowsCount={1}
  scrollLeft={scrollLeft}
  width={width}
/>

variant 2
override cell width in rendered content and hide "unused" cells
<Grid
  className={styles.HeaderGrid}
  columnWidth={columnWidth}
  columnsCount={columnsCount}
  height={rowHeight*2}
  overscanColumnsCount={overscanColumnsCount}
  renderCell={this._renderHeaderCell}
  rowHeight={rowHeight}
  rowsCount={2}
  scrollLeft={scrollLeft}
  width={width}
/>
/* ... */
  _renderHeaderCell ({ columnIndex, rowIndex }) {
    if (rowIndex === 0) {
      return (columnIndex % 3 === 0) ? (

          {`C${columnIndex}-long-long-long`}

      ) : (

      )
    } else {
      return (

          {`C${columnIndex}`}

      )
    }
  }

variant 3 (not available currently)
But would be great to have colspan, rowspan support in grid to avoid extra hidden cells in DOM.
Don't sure in implementation but one possible is:
<Grid
  className={styles.HeaderGrid}
  columnWidth={columnWidth}
  columnsCount={columnsCount}
  height={rowHeight*2}
  overscanColumnsCount={overscanColumnsCount}
  renderCell={this._renderHeaderCell}
  rowHeight={rowHeight}
  rowsCount={2}
  scrollLeft={scrollLeft}
  width={width}
/>
/* ... */
  _renderHeaderCell ({ columnIndex, rowIndex, Grid__cell }) {
    if (rowIndex === 0) {
      return (
        <Grid__cell colspan={3}> { /* since colspan="3" _renderHeaderCell calls for R0,C1 and R0,C2 will be skipped */ }

            {`C${columnIndex}-long-long-long`}

        </Grid__cell>
      )
    } else {
      return (

          {`C${columnIndex}`}

      )
    }
  }

Sorry @Guria. This isn't going to be implemented. I've considered it for and it would add a lot of complication.

@bvaughn this is ok, since there is a way to acheive desired result. btw, which existing variant do you like more?

I think your first variant, with ScrollSync, is better. It's more inline with how I'm using RV components in other projects.

I'm currently working on a similar requirement. I'm using Collection instead of two Grid. Works pretty well so far.

Nice! If you run into performance issues with Collection (which isn't as performant as Grid since it has to do more work, not being able to assume things about its underlying data) you may try switching to Grid and using the cellRangeRenderer property to render the rows/columns Grid tells you to, below your own fixed header.

Oh @phahn, do you mind to share you solution with Collection? I have a problem with grid 2. I want to achieve both colspan, and rowspan

Hey @hung-phan I have to see if I can share some code after the weekend but in the meantime here's what I'm doing:
I have configuration objects for columns like ColumnGroup which can contain other ColumnGroups and Columns
I pass this list of ColumnGroups to my grid which flattens it and calculates for every entry it's level (distance from top) and the totalWidth of that ColumnGroup which is the sum of it's leaf Column widths.
With the information totalHeaderHeight, number of levels, current level and width of each ColumnGroup and Column I can then create the appropriate div  elements in cellSizeAndPositionGetter
For row spans the concept is similar.

Interesting. Initially, I think it may be possible to do something similar with Grid by overriding (forking) cellRangeRenderer. And Grid would be more efficient than Collection since it can make some shortcut assumptions about its data due to the fact that it's all checkerboard/linear.

Thanks @phahn, and @bvaughn for the idea

If you create a cellRangeRenderer implementation this that you like, consider submitting a PR. If it seems generalizable enough, I'll add it for others. :)

Out of interest, has anyone had any luck doing this within a grid? I am migrating a major angular 1.x project to React and am looking to do something like the following:

IOW some non-fixed rows should span the entire grid and contain an SVG. At the moment I am pretty new to react-virtualized, so apologies if this has already been discussed.

I don't really have enough info to say for sure @csvan, but I believe you might want to look into using List for that. Grid is really only necessary when you want to have (a lot of) scrolling horizontal content as well.

@bvaughn I do need both horizontal and vertical scrollsync, since we will be rendering several thousand rows and columns at once. Sorry for not providing sufficient detail.
I looked into alterative 2 in the OP, and it does approximately what I want (with some hacks). It's regrettable that it leaves redundant invisible columns, but there is no noticeable performance degradation so I can live with that. Will update here if I find a better solution.

Thanks for the update!

It's regrettable that it leaves redundant invisible columns

For what it's worth, you can return null for these "invisible" columns.

@bvaughn awesome, thanks!

@Guria @bvaughn unfortunately, I hit a roadblock.
If I set the width  of the "spanning" cell to that of the visible width of the Grid, and set display: none for all remaining cells (as per example 2 in the OP), the "spanning" cell will still pop out of existence when the user scrolls the initial section of it out of the visible part of the grid.
For example, if the Grid expects each cell to be 75px wide, and I force my "spanning" cell to be 1200px wide, it will still disappear when the first 75px (approx) are scrolled out the visible part.
Is there any way I can tell the Grid to only remove elements after a certain scroll threshold?

I don't really have a clear picture of what you're doing to be honest, but if you're using a custom cellRangeRenderer then you should be able to cheat a little in terms of when the extra-width cell gets hidden.

@bvaughn I am using a MultiGrid so that I can have nice distinct column and row headers to render a spreadsheet-type grid. Working very nicely so far. I also have an upcoming requirement to support grouped columns. This would affect only how the cells are rendered in the top right Grid. I like some of your suggestions here like trying cellRangeRenderer and/or returning null for "invisible" cells. I haven't tried anything yet, but I was thinking of just trying to handle it in the cellRenderer by having the grouped column cell that spans multiple child columns inject a position: relative div that has the appropriate width. This allows it to bleed out of its containing div. Heck, maybe I could just modify the containing div's width directly. Needs more thought and experimentation. I like your response about being able to return null for "invisble" cells.
But one question for you is with a MultGrid, what is the best way to get the reference to the contained Grids? Where in the React life-cycle can I do this? I would need to know this if I wanted to try the cellRangeRenderer technique since I want that for only the top-right Grid. Thanks for all this great body of work!

@csvan @bvaughn I have made some progress with MultiGrid in trying to render grouped columns. The approach I started with follows some of the suggestions here:

Render "spanning" cells and modify their width to be the number of child columns spanned.
Render null for the other cell slots in a grouped column that are "underneath" the spanning cell.

It looked good until I started scrolling horizontally. I hit the same roadblock @csvan did. That is, when the spanning cell falls out of the viewport being rendered (because MultiGrid thinks it has a much smaller width), it disappears. I understand this behavior from the Grid's POV, but at this moment I am scratching my head. I guess I will investigate @bvaughn 's suggestion of using cellRangeRenderer. Haven't had any experience with that yet...

Yes, I have done this.  cellRangeRenderer is the solution I used and it works well.  Just adjust the columnStartIndex to make sure it captures the first 'real' column.
I also started with hiding cells, then returned null to prevent them rendering at all.  Ends up quite a neat solution.

@MarkBarbieri Thanks, Mark. Yes, I am starting down that path of using cellRangeRenderer. To render my grouped column headers above the leaf column, I think my strategy has to be to examine the columnStartIndex passed to my override of defaultCellRangeRenderer for the MultiGrid._topRightGridRef. Find the leaf column at that index and then walk up its ancestor chain of columns and if the "cell slot" for those ancestor columns have a columnIndex that is less than columnStartIndex, then I have to manually rerender those (in my screen shots above that means the 2 cells for Group 0 and Group 0.0) and push them on to the children returned from defaultCellRangeRenderer. Making progress... What could go wrong? (A common saying in our little pod of developers as the code then crashes and burns...)

I used this approach to create a TV guide layout with the whole grid full of irregular blocks.  I pre-processed the data and stored a 2D array with indices matching the column/row indices of the grid.  Each array object stores whether the cell should be rendered, the width (which is really a columnSpan), the column index of the first visible cell for this element and the column index of the next visible cell.  This provides easy manipulation of the focused element.
For example, a cell the spans columns 8-12, it stores
    {
        render: false (except for column 8)
        thisProgramColumn: 8
        nextProgramColumn: 13
        width: 5
    }

For cellRangeRendered, I then looped through the rows to be rendered,
               for (var rowIndex = rowStartIndex; rowIndex < rowStopIndex; rowIndex++) {

                    thisProgramColumn = gridCellParams[rowIndex][columnIndex].thisProgramColumn;

                    if (typeof thisProgramColumn !== 'undefined') {
                        columnStartIndex = Math.min(columnStartIndex, thisProgramColumn);
                    }

                }

                ref.columnStartIndex = columnStartIndex - fixedColumnCount;

            }

I used this approach to create a TV guide layout with the whole grid full of irregular blocks. I pre-processed the data and stored a 2D array with indices matching the column/row indices of the grid. Each array object stores whether the cell should be rendered, the width (which is really a columnSpan), the column index of the first visible cell for this element and the column index of the next visible cell. This provides easy manipulation of the focused element.
For example, a cell the spans columns 8-12, it stores
    {
        render: false (except for column 8)
        thisProgramColumn: 8
        nextProgramColumn: 13
        width: 5
    }

For cellRangeRendered, I then looped through the rows to be rendered,
               for (var rowIndex = rowStartIndex; rowIndex < rowStopIndex; rowIndex++) {

                    thisProgramColumn = gridCellParams[rowIndex][columnIndex].thisProgramColumn;

                    if (typeof thisProgramColumn !== 'undefined') {
                        columnStartIndex = Math.min(columnStartIndex, thisProgramColumn);
                    }

                }

                ref.columnStartIndex = columnStartIndex - fixedColumnCount;

            }

Wow this sounds like a really useful modification!
I was looking for a component which can create timetables. I'd really appreciate any tips you could give me about modifying rect-virtualized to support this timetable approach!

If any one is still interested in this. I have created a component with colspan and rowspan support with dynamic cell height.
https://github.com/pisharodySrikanth/virtualized-table
The component takes in data prop in a format similar to how we create the table in html. The dummy cells are hidden behind the rowspan div. This component has some caveats like the one mentioned by @pete-moss, border goes when the colspan / rowspan div is removed from the DOM. I handled this by showing border of the hidden dummy cells. This works because I don't need to vertically center align by rowspan div and knew beforehand that the content of the rowspanned cell is relatively smaller.
Also, I have created a wrapper around cell measurer and I calculate the height of the cell with rowspan by using the sum of cache.rowHeight for the next rowSpan no.of rows.
https://github.com/pisharodySrikanth/virtualized-table/blob/master/src/js/components/VirtualizedTable/CellMeasureWrapper.js

Thanks everyone for the advice. It turned out to implement a table with hierarchical columns.
I forked "defaultCellRangeRenderer" with 3 changes:

I do not call render for "invisible" cells;
I calculate the size of each cell, taking into account "invisible" cells overlapped by it;
I render cells outside the visibility zone if visible indexes depend on them.

I also used a separate grid for the header, synchronized with the body via ScrollSync.

I used this approach to create a TV guide layout with the whole grid full of irregular blocks. I pre-processed the data and stored a 2D array with indices matching the column/row indices of the grid. Each array object stores whether the cell should be rendered, the width (which is really a columnSpan), the column index of the first visible cell for this element and the column index of the next visible cell. This provides easy manipulation of the focused element.
For example, a cell the spans columns 8-12, it stores
    {
        render: false (except for column 8)
        thisProgramColumn: 8
        nextProgramColumn: 13
        width: 5
    }

For cellRangeRendered, I then looped through the rows to be rendered,
               for (var rowIndex = rowStartIndex; rowIndex < rowStopIndex; rowIndex++) {

                    thisProgramColumn = gridCellParams[rowIndex][columnIndex].thisProgramColumn;

                    if (typeof thisProgramColumn !== 'undefined') {
                        columnStartIndex = Math.min(columnStartIndex, thisProgramColumn);
                    }

                }

                ref.columnStartIndex = columnStartIndex - fixedColumnCount;

            }

Did you modify the entire cellRangeRendered or just the visible part of it. trying to the same as you here!

Thanks everyone for the advice. It turned out to implement a table with hierarchical columns.
I forked "defaultCellRangeRenderer" with 3 changes:

I do not call render for "invisible" cells;
I calculate the size of each cell, taking into account "invisible" cells overlapped by it;
I render cells outside the visibility zone if visible indexes depend on them.

I also used a separate Grid for the header, synchronized with the body via ScrollSync.

Would you mind sharing a bit of your cellRangeRendered? @dkskv

Thanks everyone for the advice. It turned out to implement a table with hierarchical columns.
I forked "defaultCellRangeRenderer" with 3 changes:

I do not call render for "invisible" cells;
I calculate the size of each cell, taking into account "invisible" cells overlapped by it;
I render cells outside the visibility zone if visible indexes depend on them.

I also used a separate Grid for the header, synchronized with the body via ScrollSync.

Would you mind sharing a bit of your cellRangeRendered? @dkskv

I signed all the differences in cellRangeRenderer with numbers. columnManager - my internal object for getting information about columns.
// ... some code
for (let rowIndex = rowStartIndex; rowIndex <= rowStopIndex; rowIndex++) {
const rowDatum = rowSizeAndPositionManager.getSizeAndPositionOfCell(rowIndex);
  // 1. An out-of-sight cell must be mounted if visible cells depend on it
  const startIndex = columnManager.getSpanningColumnIndex(rowIndex, columnStartIndex);

  for (let columnIndex = startIndex; columnIndex <= columnStopIndex; columnIndex++) {
    // 2. Render for "invisible" cells is not called
    if (columnManager.isCellSpanned(rowIndex, columnIndex)) {
      continue;
    }

// ... some code
        function getHeight(index: number) {
          return rowSizeAndPositionManager.getSizeAndPositionOfCell(index).size;
        }

        function getWidth(index: number) {
          return columnSizeAndPositionManager.getSizeAndPositionOfCell(index).size;
        }

        style = {
          // 3. Overridden height
          height: columnManager.getCellHeight(getHeight, rowIndex, columnIndex),
          left: columnDatum.offset + horizontalOffsetAdjustment,
          position: "absolute",
          top: rowDatum.offset + verticalOffsetAdjustment,
          // 3. Overridden width
          width: columnManager.getCellWidth(getWidth, rowIndex, columnIndex),
        } as const;

// ... some code


update the price

Hello i want to update the price in my cart how i can do ?

ME TOO

Hi @dassimanuel000 @prodigeomid,do you want to update the price of specific item? Can you please elaborate it more so, I can understand it better thanks.

@osamaasifoak, yes i want to modify the price of specific items who is already on cart, thanks you.

@osamaasifoak any update sir ?

@prodigeomid, why not remove that item from the cart and add a new one with the updated pricing?

@osamaasifoak because i want to add discount but not when the user add the user, i want to add the discount when the cart is already on item cart is better

@prodigeomid Alright, if you want to apply discount on a specific item. You can use a function named applyDiscount in which you can simply just pass the productId, product variant and discount it will update the price of an item. I'm still here if you need any help thanks🙂


Can't find ver 1.0 in composer

dev-master and dev-develop upload only 0.9.8

I have this problem too. Can fix soon ?

I have sent a update request for https://packagist.org. It will be available now.
https://packagist.org/packages/catfan/medoo


Fix dump summary being cropped

Dump summary is missing from the output, so threads=1 fixes it.

I'm hesitant to approve this and merge it, not because I have a deep issue with the fix but more because I'm concerned that by fixing it this way we are actually sidestepping some other, more serious, problem.  I wish we knew what the problem is and why this fixes it -- I have a hard time believing it is just the thread count.  Any thoughts @atheurer ?

will submit a new pr with different approach


Frontmatter Syntax Highlighting

Just curious, are you using a specific color scheme to get the frontmatter area to properly highlight syntax? For some reason, mine is always just a single color. I've tried over a dozen different flavors of monokai, plus a few others.

Frontmatter is highlighted using TypeScript syntax with main scope source.ts.
It should be present as it is a default syntax of ST4.

Please note Astro 2.0 syntax requires ST4126+.
All earlier ST builds get Atro v1.2 which may have several issues.
v1.x is no longer maintained.

FYI, I'm on linux where tried to install sublime with snap package manager but I had the same problem, my solution was try to install the official .deb package from sublime site, and frontmatter came back highlighted


Metrics for outlier detection

@jameswnl Can you add labels for your metrics to indicate the account number?
You can refer this PR that I just created - https://github.com/ManageIQ/aiops-data-collector/pull/41 for labels
In this PR I have added a label for collection_date also.
So account and collection_date as labels, would be two good additions to get more in depth understanding on the metrics

@AparnaKarve your collection_date is not needed, because the timestamp of the data point is it.

@jameswnl Agree, we may not need the date.
I just thought it would be handy to do some quick querying with PromQL

As for the account, it may not be useful, or not suitable for charting.  1) there'll be too many accounts.  2) in metrics/charting, we want to see overall trends, a bigger picture.
If we really want account, data-collector may be a better place (since we already blocking the duplicates there) where you can find out how many attempts for each account coming from incoming-listener

Yes, data-collector is absolutely the place to record per-account metrics.
However, I still do think that adding a label for account could be very useful here -- since some accounts could take more time for analysis while some accounts could take less, and it would be good to observe/affirm/validate that trend for each account.

@AparnaKarve Metric/charting is quite different in its use case.  We use that to find out trends and find out overall picture of populations and distributions.  Investigation of particular accounts have to be carried out in logs, data set study, re-train etc.
Although I don't think it's likely, but if it does prove to be needed, we can always look into adding labels for account later.  But for now, I believe this is a good enough start.
Again, we shall iterate fast in order to get feedback.  Currently, we have nothing in this use case's space

@jameswnl Agreed that this is a non-stop loop of iterate and improve!
The reason I am requesting account number is solely because of this graph..
(I have posted it above as well)

The initial spike of 85 seconds is on account "xyz" (say)
The 3rd, 4th and the 5th run are with my account , where the analysis took approx. 6 seconds
I am curious to know what will happen if I run the pipeline for account "xyz" again, would I see the 80-ish seconds spike again or not?
Also, by merely looking at the graph there is no way to know how the spikes map to the account #
So with whatever data we have today in CI, these are my observations (and I already have a lot of questions with the current data!)
As far as the code goes, it's looking good!
I can go ahead and merge if it helps you iterate better.

@AparnaKarve  the pipeline could be processing tens or hundreds requests. in a typical Grafana graph/dashboard setup, we'll be monitoring the distributions of the request_seconds and won't be looking at each number.  We will just from time to time have a glance of the charts to see if the shapes are normal.  We are likely to set up an alert to e.g., alerting us when a request_seconds > 2 minutes.
When that happens, we can look into the log to find out which one giving us the problem.
And we'll need to put this to run in order to determine what threshold numbers make sense.

If the alert pops up for total_analysis_time > 15 minutes sporadically when the normal trend is 10 minutes, having the account number available right then and there, would be nice, right? rather than digging up logs.
Anyway, IMO there are no right answers or wrong answers at the moment.
Right now, it's just mostly observing and improving as and when we see a need

@AparnaKarve just for your reference, there's another aspect to this decision, if I understand prometheus document correctly, when a label gets a new value, a new time series will be created.  Adding account label can easily add thousands time series to just one metric.
Like I said, we can re-evaluate this needs later. But I would like to hold back this potentially costly addition until we have obtained some good data to base on first.

Pretty sure there have been much more worse and challenging data-intensive m&m scenarios where Prometheus was tried and tested successfully -- so adding a few extra labels here and there is a non-issue.
Merging this so you can freely evaluate! :)

Regarding the discussion of label of account_id, please refer to Prometheus best practise https://prometheus.io/docs/practices/naming/, quoted as follows,

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

@jameswnl Hmm, each account_id is unique in terms of memory consumed and total time taken to complete the data collection. If we combine these metrics for all the accounts, then I am not sure how valuable that combined metric will be.
One option might be to add a new metric ..something like maximum_ceiling_hit and create this metric with account number labeling only in the situation where the maximum ceiling for memory and time(in seconds) is hit -- that way we would be creating only a handful of new time series DBs, if at all.
This metric of maximum ceiling possibly falls into the Alert Manager realm, and may help the alerting process to a great deal.


draft1 of the dtos

@saaja-sfdc please take a look. Is this the correct direction?

@jbhatt-salesforce This looks pretty good.  Please see my comments for the few minor issues I found.

@saaja-sfdc adding some more changes. I added Infraction, SuspensionLevel. Modified Policy. I still have a problem with jalopy that i'm trying to figure out. and the other problem is with integration tests.

@jbhatt-salesforce I've added my comments.


Recover when messages are removed from kafka

OffsetOutOfRange error can also happen when the returned offset is a positive number but corresponding messages started from the offset were retentioned. By @ymattw: https://github.com/bpot/poseidon/pull/102

Ups! I wanted to create the pull request to https://github.com/codeclimate/poseidon/


'node._relayoutBoundary == _relayoutBoundary': is not true.

Steps to Reproduce

...
...
...

Logs

Unfortunately, this report does not have any actionable information - it’s not clear if this is a bug in user code, a bug in the framework, or something else entirely.
Please feel free to re-open if you can add the following information:
The version of Flutter you’re using
The output of flutter doctor -v
The full exception you see
A minimal reproduction that leads to this exception.


Adjustments for Docusaurus v3

Hi 👋
Thanks for the great work, we're using the Docusaurus plugin to generate GraphQL docs and it's brilliant!
Is your feature request related to a problem? Please describe.
We've updated our docs site to Docusaurus v3 and noticed a couple of things related to the code generated by the plugin:

It uses the caution admonition (at least for deprecations) which is now deprecated in favor of warning or danger.
It uses the :::<admonition-name> <title> format instead of the Markdown directive syntax :::<admonition-name>[<title>] now used by Docusaurus v3.
Note: Docusaurus v3 still supports the previous format by transforming it to the new syntax, however we ran into an issue when it didn't properly transform it when in a blockquote, which is how it's generated for deprecations; we let the Docusaurus team know about it and they fixed the transformation process, but eventually I suspect support for the old syntax may be dropped

Describe the solution you'd like
Ideally the plugin can know the version of Docusaurus it's used with, and adapt the output.
Describe alternatives you've considered
One option is to not do anything for now since the caution admonition is deprecated but still supported, and the old title syntax should now be correctly transformed when a new version is released.
Additional context
Nothing that comes to mind, but please let me know if you need more information/details about how we use the plugin.

Hi @mderriey
Thank you for point out the deprecation. I missed that one 😓 .

I suspect support for the old syntax may be dropped
I agree.

Ideally the plugin can know the version of Docusaurus it's used with, and adapt the output.
That should not be an issue since the package @docusaurus/utils exports DOCUSAURUS_VERSION

Now available with the release 1.21.2 :rocket:

Hi @edno,
Wow, thank you so very much for your proactivity with this, it's highly appreciated!
I updated @graphql-markdown/docusaurus to v1.21.2, but unfortunately the emitted admonition still has the old syntax.
I looked into it a bit, and found that the @graphql-markdown/printer-legacy package v1.5.2, which was released at the same time, doesn't contain the code changes from #1101.
I confirmed I have the right version installed locally, and double checked with the "Code" section of npm, see https://www.npmjs.com/package/@graphql-markdown/printer-legacy?activeTab=code, then look at dist/common.js, and observe the new printWarning function doesn't exist in that file.
Do you know what could have caused this?
Thank you so much again, let me know if I can help.

This is more likely an issue with the branch build at the time of publishing the new packages.
I will get the packages fixed asap.

No problem at all, there's no rush on our side, we really appreciate your desire to fix this quickly, but please do take your time.

The issue was an issue when pulling the main branch before building the new package.
This has been fixed in release 1.21.3 :bug:

Thank you so much once again for addressing this so quickly!
I can confirm that 1.21.3 fixed the issue and the generated docs contain both the :::warning admonition instead of :::caution, and that the title appears in square brackets as expected by Docusaurus v3.