Regression: "error: websocket: close 1006 (abnormal closure): unexpected EOF"

Required information

Distribution: Ubuntu Xenial
Distribution version: 16.04.2
This issue is reproducible from 2.15 (built from commit https://github.com/lxc/lxd/commit/62908649b6e64e6f6cd15c709901e3dbacc90c3e to 2.17 as released in the current xenial-backports version 2.17-0ubuntu2~ubuntu16.04.1)
driver: lxc
driver_version: 2.0.8
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.10.0-29-generic
server: lxd
storage: zfs

Issue description
The script that I am appending next works without issues in 2.12 , but fails to
run on 2.17 (xenial-backports) version.
I have bisected the regression and I tracked it down to the following offending
commit:

https://github.com/lxc/lxd/commit/e4bd65bbff29675a8e0e24c623c7038ab89e0bbe - Signed-off-by: Stéphane Graber<EMAIL_ADDRESS>
And the subsequent commit:

https://github.com/lxc/lxd/commit/62908649b6e64e6f6cd15c709901e3dbacc90c3e -  Signed-ff-by:
Stéphane Graber<EMAIL_ADDRESS>
I built a LXD/LXC from the previous commit https://github.com/lxc/lxd/commit/665f9013f70ae99a92c6c8c8bd08892b5dfd6cbf
and the script works well as well as with any other previous version to the offending commits.
start.sh
#!/bin/bash
for x in `seq 1 20`; do
./reproducer.sh test$x &
done
wait

reproducer.sh
#!/bin/bash

NAME=$1
lxc init ubuntu: $NAME
while pgrep -f 'lxc init' >/dev/null; do
echo "$NAME: wait for all lxc done"
sleep 2
done

lxc start $NAME

while true; do
lxc exec $NAME -- sleep 5
done

Please check the full reproducer description next.
Steps to reproduce

Run lxd init and create a zfs pool as follows

+---------+-------------+--------+--------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |             SOURCE             | USED BY |
+---------+-------------+--------+--------------------------------+---------+
| default |             | zfs    | /var/lib/lxd/disks/default.img | 19      |
+---------+-------------+--------+--------------------------------+---------+

Run ./start.sh (described previously)
Check for the output of the previous command and wait for the multiple occurrences
of the error to appear:

2017-09-05 20:23:57 Creating test3
2017-09-05 20:23:57 Creating test15
2017-09-05 20:23:57 Creating test13
[...snip]
2017-09-05 20:23:58 Creating test4
2017-09-05 20:23:58 Creating test14
test11: wait for all lxc done
error: websocket: close 1006 (abnormal closure): unexpected EOF
2017-09-05 20:24:15 test8: wait for all lxc done
2017-09-05 20:24:17 test11: wait for all lxc done
2017-09-05 20:24:17 test8: wait for all lxc done
error: websocket: close 1006 (abnormal closure): unexpected EOF
test13: wait for all lxc done
[...]
Information to attach

Lsof of all the LXD processes.
Logs from LXC Client with --debug --verbose
Logs from LXD running with --debug --verbose
Strace for all the LXD processes

strace-lxd-monitor.log.txt
lxd-debug.log.txt
lxc-client-debug.log.txt
lsof-debug.log.txt

Unable to reproduce this on my local system. Will try on a clean Ubuntu 16.04 next.

Same result on a fresh system:
stgraber@castiana:~/data/code/lxc/lxd-pkg-snap (latest-candidate)$ rssh<EMAIL_ADDRESS>Warning: Permanently added 'vm01.maas.mtl' (ECDSA) to the list of known hosts.
root@vm01:~# apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@vm01:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  snapd ubuntu-core-launcher
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.7 MB of archives.
After this operation, 4,466 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-core-launcher amd64 2.27.5 [1,562 B]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 snapd amd64 2.27.5 [10.7 MB]
Fetched 10.7 MB in 0s (45.8 MB/s)
(Reading database ... 59856 files and directories currently installed.)
Preparing to unpack .../ubuntu-core-launcher_2.27.5_amd64.deb ...
Unpacking ubuntu-core-launcher (2.27.5) over (2.26.10) ...
Preparing to unpack .../snapd_2.27.5_amd64.deb ...
Warning: Stopping snapd.service, but it can still be activated by:
  snapd.socket
Unpacking snapd (2.27.5) over (2.26.10) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up snapd (2.27.5) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Setting up ubuntu-core-launcher (2.27.5) ...
root@vm01:~# apt install -t xenial-backports lxd lxd-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ebtables xdelta3
Suggested packages:
  criu lxd-tools
The following NEW packages will be installed:
  ebtables xdelta3
The following packages will be upgraded:
  lxd lxd-client
2 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 5,123 kB of archives.
After this operation, 1,945 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ebtables amd64 <IP_ADDRESS>-3.4ubuntu2 [79.4 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd amd64 2.17-0ubuntu2~ubuntu16.04.1 [3,000 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd-client amd64 2.17-0ubuntu2~ubuntu16.04.1 [1,976 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 xdelta3 amd64 3.0.8-dfsg-1ubuntu2 [67.1 kB]
Fetched 5,123 kB in 0s (14.8 MB/s)
Selecting previously unselected package ebtables.
(Reading database ... 59859 files and directories currently installed.)
Preparing to unpack .../ebtables_<IP_ADDRESS>-3.4ubuntu2_amd64.deb ...
Unpacking ebtables (<IP_ADDRESS>-3.4ubuntu2) ...
Preparing to unpack .../lxd_2.17-0ubuntu2~ubuntu16.04.1_amd64.deb ...
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
Unpacking lxd (2.17-0ubuntu2~ubuntu16.04.1) over (2.0.10-0ubuntu1~16.04.2) ...
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
Preparing to unpack .../lxd-client_2.17-0ubuntu2~ubuntu16.04.1_amd64.deb ...
Unpacking lxd-client (2.17-0ubuntu2~ubuntu16.04.1) over (2.0.10-0ubuntu1~16.04.2) ...
Selecting previously unselected package xdelta3.
Preparing to unpack .../xdelta3_3.0.8-dfsg-1ubuntu2_amd64.deb ...
Unpacking xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up ebtables (<IP_ADDRESS>-3.4ubuntu2) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up lxd-client (2.17-0ubuntu2~ubuntu16.04.1) ...
Setting up xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Setting up lxd (2.17-0ubuntu2~ubuntu16.04.1) ...
Installing new version of config file /etc/init.d/lxd ...
Removing obsolete conffile /etc/init/lxd.conf ...
Setting up lxd dnsmasq configuration.

Old bridge configuration detected in /etc/default/lxd-bridge, upgrading
Unsetting deprecated profile options
Attempting to kill current lxd-bridge
Stock LXD installation detected, resetting to new defaults
Cleaning up lxd-bridge state files

Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@vm01:~# lxd init
Do you want to configure a new storage pool (yes/no) [default=yes]?
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, btrfs, lvm) [default=dir]:
Would you like LXD to be available over the network (yes/no) [default=no]?
Would you like stale cached images to be updated automatically (yes/no) [default=yes]?
Would you like to create a new network bridge (yes/no) [default=yes]?
What should the new bridge be called [default=lxdbr0]?
What IPv4 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]?
What IPv6 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]?
LXD has been successfully configured.
root@vm01:~# mkdir a
root@vm01:~# rmdir a
root@vm01:~# cd /home/ubuntu/
root@vm01:/home/ubuntu# ls -lh
total 8.0K
-rwxr-xr-x 1 ubuntu ubuntu 209 Sep  6 04:04 reproducer.sh
-rwxr-xr-x 1 ubuntu ubuntu  77 Sep  6 04:04 start.sh
root@vm01:/home/ubuntu# ./start.sh
If this is your first time using LXD, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:16.04

Creating test1
Creating test2
Creating test10
Creating test16
Creating test13
Creating test19
Creating test12
Creating test8
Creating test3
Creating test6
Creating test20
Creating test14
Creating test17
Creating test18
Creating test4
Creating test9
Creating test11
Creating test7
Creating test5
Creating test15
test9: wait for all lxc done0% (8.54MB/s)
test19: wait for all lxc done
test18: wait for all lxc done
test1: wait for all lxc done
test13: wait for all lxc done
test14: wait for all lxc done
test3: wait for all lxc done
test16: wait for all lxc done
test11: wait for all lxc done
test8: wait for all lxc done
test10: wait for all lxc done
test2: wait for all lxc done
test15: wait for all lxc done
test5: wait for all lxc done
test6: wait for all lxc done
test20: wait for all lxc done
test7: wait for all lxc done
test4: wait for all lxc done
test12: wait for all lxc done

Been sitting there for 5 minutes or so, running "lxc exec" against the various containers.

What kind of load is your system under when those errors show up?
I've managed to get LXD to get some of those when I interact with over 100 containers in parallel, at first glance those are caused by the system being so loaded that it's not scheduling enough time for the websocket client to react to events before the daemon closes the session as a result of the process exitting.

It looks like in the past such errors were being silently dropped, the new client code is much stricter about catching such websocket errors and surfacing them back to the user.
My guess is that the old code would suffer from the exact same issue but rather than showing you an error, it would simply exit 0 and never report that some of the data got lost.

Increasing parallelism even more, we eventually even hit a bunch of "error: not found", which is what you'd expect if the client gets so busy that it fails to retrieve the operation data within 5s of the operation having completed (the client attempts to fetch state of an operation that's already been garbage collected).

@stgraber
Thanks for looking into this.
OK, I can't definitively reproduce the problem in a newly deployed xenial virtual machine, but on a production system with moderate load average (56 cores, 22 points of load average, 5 minutes window) it's fairly easy to get the "error: websocket: close 1006 (abnormal closure): unexpected EOF" errors during the lxc exec loop" on 2.17. Maybe as part of the reproducer we should stress and induct
some cpu load into the system.
As you already mentioned these errors were probably dismissed in earlier versions (before https://github.com/lxc/lxd/commit/e4bd65bbff29675a8e0e24c623c7038ab89e0bbe) and this
was getting unnoticed by the tools that make use of  lxc exec in parallel.
Also, I am not versed yet on the lxd internals, but  assuming that the idea of " being so loaded that it's not scheduling enough time for the websocket client to react to events before the daemon closes the session " is true, why is this happening? Shouldn't be the writer waiting for all the message/data to be flushed/read on the socket before ending the session?

The LXD daemon is pretty aggressive on websocket handling, if the client doesn't handle events in a timely manner, the connection gets dropped. That's needed so that we don't end up with a bunch of open connections on the server when a network problem occurs. It's obviously less relevant for clients using the unix socket, but the LXD daemon doesn't know how someone connected to the API.

@stgraber Could you please point me to the current timeouts/deadlines at the server and client side? I don't think my system was so overloaded as I'd expect ms latency as no swap was in use at the moment of the problem and the load average was high, but not extreme at the seconds levels ..
Also, if you look at the specific websocket error 1006 (abnormal closure) this seems
to indicate that the problem is that the connection was dropped without a proper close frame being sent [0] and according to the documentation this is typically caused by proxies that doesn't
handles the termination properly.
"Used to indicate that a connection was closed abnormally (that is, with no close frame being sent) when a status code is expected."
My understanding is that at that point the client was either reading or terminating the connection
when the server decided to abruptly close the connection without sending the close frame. Is that behavior expected?
[0]  https://developer.mozilla.org/en-US/docs/Web/API/CloseEvent

Sent a pull request which fixes a race in the server side code. This fixes at least one cause for the message you've been getting at least on my test systems.

@niedbalski LXD never sends a close frame because it never closes that connection. A /1.0/events API call is never ending. The only way it can get disconnected is the server going down or network failure, both of which cause a network disconnection without a websocket close frame.

@stgraber
I just tested a fresh build of 2.17 with the patch for #3771 applied on top, and it seems
that the reported message (close 1006) isn't being returned anymore.
Thank you!

Closed? I still see that message with lxd --version 3.0.0.
Error: websocket: close 1006 (abnormal closure): unexpected EOF
I installed lxd using snap, container is running, connected to it with exec bash and leave the session open. Some time later the error occurs, the open terminal sessions close.

I suspect that when snaps refresh with a updated version, the LXD kick out the user in a session.
I was just doing one job using two containers on LXD, when after some minutes I was kicked out in both ...
Then without any reason I just tried to look on the version I had installed 3.1 candidate, and I saw that it has updated my version in less than one minute.
I don't know how exactly how snaps manages this but now I just set a scheduled update for it.

Yes, I just noticed the same issue; it seems I get this error whenever the snap updates itself.

@mpontillo then only thing you can do about it is to schedule your updates to a specific day of month or
else compile your own lxd server

Yes, my workaround for now is sudo snap set lxd refresh.timer=fri,23:00-01:00; this way I'm not interrupted with updates during the work week.

I saw this today, on an unloaded system running lxd 3.16. It's not clear how to reproduce. Are there logs that can be attached?

I think one way to reproduce this is:

install a earlier version than;
start a new job inside your container
schedule the time of Snap to update your LXD software a few seconds or minutes later

Then you will see it stop your container to do the update.

The same problem on many servers with lxd 3.0.3-0ubuntu1~18.04.1 on Ubuntu 18.04.3.
Happens occasionally, difficult to reproduce.

I got this error message when running a bulk loader of data in a database inside a container. This job has to load the data and create indexes, so it takes several hours and overloaded the system. At some point I got an "Error: websocket: close 1006 (abnormal closure): unexpected EOF." With datasets of similar sizes, sometimes it succeeds, sometimes it fails. I had a console doing this work inside a lxc exec container bash terminal. After having this error the container bash terminal is closed. I am using LXD 4.11 with Ubuntu 18.04 in the host and Ubuntu 20.04 in the container.

That's usually the sign of a crash from the daemon.
Look at 'journalctl -u snap.lxd.daemon' for a crash or maybe 'dmesg' for an indication of your system running out of memory and LXD being killed because of it.

Hi @danielhz ,
if you was connected to the terminal the moment your snap updated the LXD to new build version, you will be kicked out until the process finish update or upgrade.

I have this message several times at the end of journalctl -u snap.lxd.daemon:
Feb 09 21:56:23 container lxd.daemon[869]: t=2021-02-09T21:56:23+0000 lvl=warn msg="Detected poll(POLLNVAL) event."

try to see the date and time your lxd was refreshed see it match with the time this happened
snap info lxd

Hi @cyberwillis,
In the host, I opened a tmux terminal. Then, I opened a connection to the container with lxc exec container bash. Inside the container I started the job. Then, I detached the tmux terminal. After some hours, I attached the terminal again, and I see that the connection with the container was lost.
@cyberwillis, I think that your hypothesis of that the connection was kicked out because snap was updating to a new version is very likely. Indeed, I have three machines running jobs in a tmux terminal, and LXD connection was lost in all of them. In one of them the job continues after the LXD connection was lost. The container is still running the job I initiated. I checked that I have two LXD revisions installed 19300 and 19325. I guess that revision 19325 was installed automatically while I was running the job.
There are the following messages in the host:
$ journalctl -u snap.lxd.daemon
...
Feb 10 03:15:17 hostmachine systemd[1]: Stopping Service for snap application lxd.daemon...
Feb 10 03:15:18 hostmachine lxd.daemon[57298]: => Stop reason is: snap refresh
Feb 10 03:15:18 hostmachine lxd.daemon[57298]: => Stopping LXD
Feb 10 03:20:19 hostmachine lxd.daemon[57298]: ==> Stopped LXD
Feb 10 03:20:19 hostmachine systemd[1]: Stopped Service for snap application lxd.daemon.
Feb 10 03:20:23 hostmachine systemd[1]: Started Service for snap application lxd.daemon.
Feb 10 03:20:23 hostmachine lxd.daemon[58060]: => Preparing the system (19325)
Feb 10 03:20:23 hostmachine lxd.daemon[58060]: ==> Loading snap configuration

And I have the following information for the lxd snap:
$ snap info lxd
...
refresh-date: today at 03:20 CET

Hi @danielhz ,
If you execute :
snap changes you will see the ID of the snap that refreshed the LXD then you can pass
snap tasks <ID> to verify too.
This happened to me before, so in my case was the update that kicked me out.
There are one solution for that if that is the case.
sudo snap set system refresh.hold="2030-01-01T01:00:00-01:00"
then start frequently update your package manually.

Hi @cyberwillis, your hypothesis is very likely. It also helps me to know how I can delay snap updates to start updating packages manually.

Hi,
this seems to be a long running issue and coming up every now and then but stays hard to recreate or debug.
I'm used to see it break about 1/5 of my Tests runs (each about 5-8 hours) which heavily utilize LXD on a 12 core system.
So far I also just assumed that it might be the snap update that runs in background and a few times that was even true - at least the timing matched. But last night I had another case where a snap update was not even close. I'm providing the info of that for the sake of helping to find and fix the remaining underlying issue.
First of all, what I see is the same Error: websocket: close 1006 (abnormal closure): unexpected EOF that so far everyone reported and is just the message for an incomplete close in the go websocket implementation.
But this time I had a rather clean lxd snap as I re-installed 4-0-stable yesterday to clean up some old crap.
$ snap changes
ID   Status  Spawn                   Ready                   Summary
29   Done    yesterday at 08:31 UTC  yesterday at 08:46 UTC  Remove "lxd" snap
30   Done    yesterday at 08:48 UTC  yesterday at 08:48 UTC  Install "lxd" snap from "4.0/stable" channel

For a more detailed version, I'm on 4.0/stable:       4.0.9-8e2046b 2022-03-26 (22753)  71MB -
There also isn't any further change since then.
And for more cleanup I rebooted the machine right after that LXD re-install.
The websocket issue occurred at ~22:20 14h after the last install/update.
In the journal of the daemon I saw some transaction issues that weren't reported here before. And timing wise they match perfectly to when I found the issue in my logs.
$ journalctl -u snap.lxd.daemon -n 8
Jun 21 09:56:54 node-horsea lxd.daemon[3334]: - pidfds
Jun 21 09:56:54 node-horsea lxd.daemon[3334]: Reloaded LXCFS
Jun 21 09:57:22 node-horsea lxd.daemon[128359]: => LXD is ready
Jun 21 22:19:57 node-horsea lxd.daemon[128505]: t=2022-06-21T22:19:57+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" me>
Jun 21 22:19:58 node-horsea lxd.daemon[128505]: t=2022-06-21T22:19:58+0000 lvl=warn msg="Transaction timed out. Retrying once" err="failed to begin transaction: context deadline exceeded" me>
Jun 21 22:19:58 node-horsea lxd.daemon[128505]: t=2022-06-21T22:19:58+0000 lvl=eror msg="Failed getting volumes for auto custom volume snapshot task" err="failed to begin transaction: contex>
Jun 21 22:19:59 node-horsea lxd.daemon[128505]: t=2022-06-21T22:19:59+0000 lvl=warn msg="Failed to rollback transaction after error (Local node name: sql: transaction has already been commit>
Jun 21 22:19:59 node-horsea lxd.daemon[128505]: t=2022-06-21T22:19:59+0000 lvl=eror msg="Failed to load instances for snapshot expiry" err="Local node name: sql: transaction has already been>

I checked further if/what happened around that time and I've seen this right before:
Jun 21 22:19:07 node-horsea sshd[3930842]: pam_unix(sshd:session): session closed for user ubuntu
Jun 21 22:19:07 node-horsea systemd[1]: session-2714.scope: Consumed 1.855s CPU time.
Jun 21 22:19:07 node-horsea systemd-logind[1583]: Session 2714 logged out. Waiting for processes to exit.
Jun 21 22:19:07 node-horsea systemd-logind[1583]: Removed session 2714.
Jun 21 22:19:40 node-horsea sshd[3931657]: Accepted publickey for ubuntu from <IP_ADDRESS> port 60856 ssh2: RSA SHA256:KyONnhWWzlbscZNTHPZ25GWCXDQY5u/UD72EtQcwtqU
Jun 21 22:19:40 node-horsea sshd[3931657]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)
Jun 21 22:19:40 node-horsea systemd-logind[1583]: New session 2715 of user ubuntu.
Jun 21 22:19:40 node-horsea systemd[1]: Started Session 2715 of User ubuntu.
Jun 21 22:19:41 node-horsea systemd[1]: Started snap.canonical-livepatch.canonical-livepatch.4b97ec9a-5360-4bff-aa43-cb7486ab8728.scope.
Jun 21 22:19:42 node-horsea systemd[1]: snap.canonical-livepatch.canonical-livepatch.4b97ec9a-5360-4bff-aa43-cb7486ab8728.scope: Deactivated successfully.
Jun 21 22:19:42 node-horsea sshd[3931819]: Received disconnect from <IP_ADDRESS> port 60856:11: disconnected by user
Jun 21 22:19:42 node-horsea sshd[3931819]: Disconnected from user ubuntu <IP_ADDRESS> port 60856
Jun 21 22:19:42 node-horsea sshd[3931657]: pam_unix(sshd:session): session closed for user ubuntu
Jun 21 22:19:42 node-horsea systemd[1]: session-2715.scope: Deactivated successfully.
Jun 21 22:19:42 node-horsea systemd-logind[1583]: Session 2715 logged out. Waiting for processes to exit.
Jun 21 22:19:42 node-horsea systemd-logind[1583]: Removed session 2715.

There are two potential triggers to think about if they might be related.
First the ssh disconnect, I happened to find that my connection seems to live on the edge only relying on transparent auto-reconnect. The IP is my computer and this is computer -> ssh -> script -> issuing lxd cli commands.
sshd keeps the connection stable from my POV, but the continuous loss and reconnect might still influence the lxd execution.
Secondly there is a snap refresh, not lxd but livepatch.
I checked the log, but I had no further correlations between the two,
In any case, maybe the logs might start a new idea what could be left causing this issue occasionally.
And if not, well it is not getting worse by providing this info :-)


Revisit Flow of working with dozens of Issues in GitHub

Is a Project Manager and Software Engineer, I find the flow of opening dozens of GitHub Issues for the single tasks of the agent very unintuitive and confusing. Especially, as it does not reflect what humans would do. In addition, I think it's weird that the agents post the code they want to generate as comments to a GitHub issue instead of opening a Pull Request, where code can be properly discussed. Lastly, closing issues before the actual work on them is done also looks like a bad practice to me.
Here is, how an ideal flow would look like to me:

Human Project Manager opens a GitHub issue with a description of what to achieve
Human Project Manager labels the issue with dev-agents label, to trigger the AI work
First Agents starts the work and describes the project plan in Markdown as a comment to the issue
Human Project Manager can give feedback and iterate on it in the comments.
Once satisfied, the issue get labeled differently (e.g. coding-plan), which triggers another Agent to plan the coding work
An Agent generates the step-by-step plan (To Do List) and posts it as a comment to the issue
Human Project Manager can give feedback and iterate on it in the comments.
Once satisfied, the issue gets labeled again (e.g. start-coding), which triggers another Agent to start the coding work
An Agent creates a new branch, commits code and creates a Pull Request, with the code changes and links the PR to the issue
Human Developer can give feedback on the code and iterate on it in the PR comment.
Once satisfied, the PR gets merged and the Issue automatically gets closed

Am I missing anything, why the flow is not like this but happens across so many issues?
Am I misunderstanding anything, which makes my suggested flow not possible?

One issue I see with that is parallelism. But I still think I would massively prefer multiple Pull Requests over 30 issues...

I agree, we can make the flow more fluent and branch out less. The idea with PRs is interesting, as they are Issues technically, so the branching out (devs) can happen on different PRs and still don't break the implementation logic.
One way would be to create a PR for a group of subtasks, for example PR for adding the frontend and PR for adding the APIs.


Issue: 'expand' query parameter exposes internal domain class names

Hi Benjamin,
when we send request like:
GET /api/Users
?filter=        Age>=20, Name$*Smith|Bullock
&orderby=       -Age, Name
&select=        Id, Name, Age
&expand=        UserRoles
&pageNumber=    1
&pageSize=      10

we expose our internal domain class name (navigation property) UserRoles to end users and I think it violates some of the REST API recommendation which suggests that we should hide our domain classes.
Accordingly, it would be good in CanExpand method that we could set string key like in other methods (CanSelect, CanFilter, CanOrder) which maps to our internal domain class like CanExpand("RolesOfUser", x => x.UserRoles)  to hide our internal class name.
Thanks,
Branislav

Thank you for the suggestion, I will investigate how viable it is to implement this. I remember debating whether to add this as a feature or not, though forgot the reason why I didn't end up doing this.

Had a deeper look and while it is definitely possible it will require a rewrite in certain places. I will keep this issue open until I get around to adding this functionality.


Uses fixed positioning instead of absolute

I think there's one issue with this. If you're in an iframe, where does the calendar popup appear?


Create classes: Person, Student and Teacher.

Hello, dear reviewer.
For this milestone I met all of this requirements:

Create class Person with  the following:

Instance vars: @id, @name, and @age.
Constructor with name,  age, and parent_permission as parameter. name and parent_permission are optional and have default values of "Unknown" and true.
Getters for @id, @name, and @age.
Setters for @name and @age.
Private method of_age? that returns true if @age is greater or equal to 18 and false otherwise.
Public method can_use_services? that returns true if person is of age or if they have permission from parents.

Create class Student with the following:

Inherits from Person.
Constructor extends parent's constructor by adding @classroom and a parameter for it.
Method play_hooky that returns "¯\(ツ)/¯".

Create class Teacher with the following:

Inherits from Person.
Constructor extends parent's constructor by adding @specialization and a parameter for it.
Override can_use_services? so it always returns true.

Each class should be saved in a separate file.

Have a very good day.

Thank you very much for your valuable time.


NIFI-1164 Fixed race condition and refactored

Changed ControllerServiceNode by adding enable(..), disable(..) and isActive() operations. See javadocs for more details in both ControllerServiceNode and StandardControllerServiceNode
Refactored service enable/disable logic in StandardProcessScheduler and StandardControllerServiceNode . Below are some of the notes:

No need for resetting class loader since its going to derive from the class loader of the service. In other words any classes that aren’t loaded and will be loaded within the scope of the already loaded service will be loaded by the class lower of that service
No need to control 'scheduleState.isScheduled()’ since the logic has changed to use CAS operation on state update and the service state change is now atomic.
Removed Thread.sleep(..) and while(true) loop in favor of rescheduling re-tries achieving better thread utilization since the thread that would normally block in Thread.sleep(..) is now reused.
Added tests and validated that the race condition no longer happening

Added additional logic that allows the initiation of the service disabling while it is in ENABLING state. See javadoc of StandardProcessScheduler.enable/disable for more details.

@olegz - I'm very happy that you are digging in here. This bit of code certainly could use some TLC and refactoring. I very much like the approach of moving this logic into the StandardControllerServiceNode object instead of the scheduler. I did notice one issue that appears to be a race condition if enable() and disable() are running in parallel (commented inline). Otherwise, I've not seen any other issues but will continue to review this. Thanks again for tackling this!


Post method 3

This is a PR for the issue #3 .

Some payloads for testing:
Route: /api/genres
Payload 1:
"sci-fi"

Payload 2:
["sci-fi", "horror"]

Route: /api/laptops
Payload:
{
    "id": 150,
    "brand": "Intel"
}

Route: /api
Payload:
{
    "note": {
        "text": "Some very important note",
        "date": "2020-06-30T10:04:35.507Z"
    }
}


GPU stream error when running with gpu context

Trying to add a new operator in C++.
The code compiled and run as expected with cpu context,
but when I ran it with gpu context, it throw the following error,
Default GPU stream was used when MSHADOW_FORCE_STREAM was on.
My guess is I used NewTensor in the forward/backward operation in a way not as expected.
Any clue here ?

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!


Separate the validate function and do not use from node

We cannot require() an entry point from node because it may contain browser-specific logic, for example window.navigator.

In more discussion with @blink1073, we discussed moving the function into the jupyterlab package, since the extension builder can then treat the plugin definition as a black box and just hand it to jupyterlab. However, we want a plugin to be able to have this initial vetting at compile time, and a plugin only has to import the extension builder, but doesn't have to import jupyterlab.


Feature: Add way to uniquely identify a tile within a map by exposing Gid

What problem does this solve or what need does it fill?
I'm currently using rs-tiled to load in a tilemap. When I load it into the engine, I need to do some processing on the data of each tile in each tileset to import it into my engine. This processing is considered costly so doing it each time I spawn a tile would be not ideal, so I'd like to cache it at the time of loading by having some kind of map from Tile to processed Data and store it with my Tilemap. My request is to have some cheap way to uniquely identify a tile within a map.
What solution would you like?
In the tmx file itself the tile layer uses a Gid to identify which tile it is. This Gid would be great as a key for a HashMap<Gid, UserData>, but it's currently discarded at the end of the loading process: https://github.com/mapeditor/rs-tiled/blob/1ffab0381abdea1fab49e66fcc65dab3ed4f60cd/src/map.rs#L275-L276
What alternative(s) have you considered?
https://github.com/mapeditor/rs-tiled/pull/303 adds the source path of each tileset, allowing you to create a HashMap<PathBuf, HashMap<TileId, UserData>>, but this would not be as ideal as using the Gid since this is a much more expensive key with lots of indirection.

Regarding the HashMap<Gid, UserData>, just note that the Gid for a given tile can (and often will) be different depending on the map the tile is referenced from. Hence it also can't be exposed on the TileData, Tile or even Tileset. To expose the Gid in the API, we'd need to add a function to Map that returns the Gid for a given Tile, which would be "tile.tileset's first_gid for this map + tile_id". So it adds another lookup that's not exactly fast either.
In C++ I'd probably solve this case by raw pointers, but this is likely problematic in Rust due to lifetimes? Like having a std::unordered_map<Tile*, UserData>.
Alternatively, I guess you could create an entirely parallel data structure for each loaded map, which you can use to look up the UserData for each location. You could set this up just after the map is loaded and it would avoid the need for looking up UserData by the tile entirely, at the cost of using a bit more memory.


Add insertText and run ember-mobiledoc-kit editor helpers

Usage:
// acceptance test
import { insertText, run } from '../../helpers/ember-mobiledoc-editor';

test('visit /', function(assert) {
  visit('/');
  andThen(() => {
    let editorEl = find('.mobiledoc-kit')[0];
    return insertText(editorEl, 'here is some text');
    /* Or:
      return run(editorEl, (postEditor) => ...);
    */
  });
  andThen(() => {
    // assert text inserted, etc.
  });
});

fixes #100

@chrisdpeters These are released in v0.5.6. Does that work for your needs in #100?

@bantic I switched to a bunch of API work in October, which lasted quite some time, so I was unable to try any of this out until now. I'm now messaging you 3 months into the future to say, "Thank you!"
The only problem that I've experienced so far has been with your example.
let editorEl = find('.mobiledoc-kit')[0];

With that, editorEl ends up being null.
I seem to have better luck if I use this selector: .mobiledoc-editor__editor. It looks like this class is also set on the editor element: .__mobiledoc-editor.
If you think this is indeed the correct behavior, then I could submit a small pull request with an updated README.
I'll let you know if I run across any other issues, but this seems to be working pretty well so far.

@chrisdpeters Thanks for the comment to let me know this is working for you! Glad that is working.
Yes, I think you are correct about using the other selector. If you are able to submit a PR for this I'd be happy to merge it. thanks!


Can't open rnnoise-master/src/denoise_training

Hi, I'm trying to open rnnoise-master/src/denoise_training but can seem to view the contents, any idea how to view the contents of the file?

嗨，我正在尝试打开 rnnoise-master/src/denoise_training 但似乎可以查看内容，知道如何查看文件的内容吗？

你好，能问你几个问题吗，方便留个联系方式吗


Folders collpasing

Issue Type: Bug
Since last update I get bugs that when I try to open/close a folder, it will do that to another folder or completly mixup what folder goes where
VS Code version: Code 1.24.1 (24f62626b222e9a8313213fb64b10d741a326288, 2018-06-13T17:41:50.128Z)
OS version: Darwin x64 17.6.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (8 x 4200)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledvideo_decode: enabledvideo_encode: enabledvpx_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
3, 4, 4

Memory (System)
8.00GB (0.02GB free)

Process Argv
/Applications/Visual Studio Code.app/Contents/MacOS/Electron

Screen Reader
no

VM
0%

Extensions (4)

Extension
Author (truncated)
Version

fish
Ted
0.0.4

graphql
mqu
0.1.2

debugger-for-chrome
msj
4.6.0

change-case
wma
1.0.0

Could you provide step to reproduce, one step at a time, and before/after screenshots?

Also try reproducing with latest vscode insiders https://code.visualstudio.com/insiders/

Hi sorry for the delay, I wasn't able to reproduce this in a way I could share, it happened twice since posting the bug, but idk what steps I made to get to there, it happened after heavy load of working, some hours in using the software.
Next time it happens I will attach a video so you can at least have an idea of what i was talking about.


Remove creation of breadcrumbs in the next level dir

This PR prevents breadcrumbs from being recreated everytime we issue a listStatus request.
The main idea is to check for breadcrumb's existence before creating it and only create it when a directory is listed not when it is listed as a child of another directory.

Codecov Report

Merging #13209 (0463532) into master (7824e89) will decrease coverage by 26.65%.
The diff coverage is 20.00%.

:exclamation: Current head 0463532 differs from pull request most recent head 55fbe12.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d41fb8d...55fbe12. Read the comment docs.

Before

After

ignore the s3 credential requests in the before picture... it was the first request from alluxio, so it needed to authenticate itself..

alluxio-bot, merge this please

helps with #13259

alluxio-bot, cherry-pick this to branch-2.5 please


Not showing in integrations

Hi.
I am running the latest version of HA. I have installed via HACS but when i try to add / configure via integrations i dont see it listed. Anything i can check or do?

Not to worry. seemed it was cache issue.

This seems to be a common HA issue that it caches the list so you need to do a shift F5 refresh on your browser to refresh the page not using cache and it should appear.Sent from my Galaxy
-------- Original message --------From: McFarTech @.> Date: 08/02/2022  20:35  (GMT+00:00) To: msp1974/homeassistant-jlrincontrol @.> Cc: Subscribed @.***> Subject: [msp1974/homeassistant-jlrincontrol] Not showing in integrations  (Issue #65)
Hi.
I am running the latest version of HA. I have installed via HACS but when i try to add / configure via integrations i dont see it listed. Anything i can check or do?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>


Lightning

Hello,
is it possible to output the lightning_num and lightning as an object. They will be apparentlyed received.
2023-03-18 17:05:10.477  - debug:  sainlogic.0 (26466) Listener received EW update:  "stationtype":"WH2650A_V1.7.6","runtime":"762631","dateutc":"2023-03-18   16:05:09","tempinf":"66.74","humidityin":"45","baromrelin":"31.943","baromabsin":"29.379","tempf":"62.06","humidity":"54","winddir":"183","windspeedmph":"0.00","windgustmph":"0.00","maxdailygust":"8.05","solarradiation":"59.82","uv":"0","rainratein":"0.000","eventrainin":"0.161","hourlyrainin":"0.000","dailyrainin":"0.161","weeklyrainin":"0.480","monthlyrainin":"2.591","yearlyrainin":"8.071","totalrainin":"8.071","temp1f":"60.98","humidity1":"56","temp2f":"51.08","humidity2":"74","soilmoisture1":"0","lightning_num":"1","lightning":"31","lightning_time":"1679135954","wh65batt":"0","wh25batt":"0","batt1":"0","batt2":"0","soilbatt1":"1.3","wh57batt":"5","freq":"868M","model":"WH2650A"}
Greetings

Added since june 6th.
#129


Added mysql docker-compose

Pull Request type

[x] Feature: add mysql docker

Changes in this PR
Update mysql from 5.6 to 5.7
match mysql password with docker-compose and config
Start elasticsearch in cluster instead of single node, due to timeout for Green. https://github.com/Netflix/conductor/issues/2262
Describe the new behavior from this PR, and why it's needed
Issue #2262
docker-compose in grpc does not work because mysql 5.6 is not longer supported by flywaydb

Please update the ReadMe with steps on how to this.

Please update the ReadMe with steps on how to this.

Done, added a README file specifically for mysql dir.


add: cachyos-ananicy-rules-git

[!] ERROR: cachyos-ananicy-rules-git is not installed
?

[!] ERROR: cachyos-ananicy-rules-git is not installed
@Elsie19 what's wrong here?

It happens with pacdeps. It's a CI only issue.


Revert "FIX: missing underscore"

Reverts hhu-propra2/abschlussprojekt-mopse#342

This should work. I'm not entirely sure but I'll shame @KaFe23 if it doesn't work. So it's okay I guess.

This is bullying of the highest level. I will seek further actions.

This should work. I'm not entirely sure but I'll shame @KaFe23 if it doesn't work. So it's okay I guess.

This is unacceptable.


Add preliminary support for First-Order Logic

Proposed changes
This PR adds preliminary support for First-Order Logic and an implementation of natural deduction.
In particular, the added files in commit 3c49b23488b98dce3ecac73fde8cc1a7a4796729 are taken from the external repository https://github.com/paulotguerra/pylogics-extras/commit/6848d5f9d9ecc470be811532ea7b8c080967e6e4, and the copyright holder is the author of that commit (Paulo T. Guerra<EMAIL_ADDRESS>(@paulotguerra)). The commit 3c49b23488b98dce3ecac73fde8cc1a7a4796729 is authored by @paulotguerra to recognize the authorship of the code (but it has been committed by @marcofavorito for practical purposes).
Paulo will be added as a contributor to the project and, therefore, will hold the copyright. The license will implicitly reflect this using the notice "Copyright (c) 2021-2024 The Pylogics contributors" (see https://github.com/whitemech/pylogics/pull/139). The fine-grained information on the contributions is stored in the Git history of the project.
The code needs relicensing of his code with the MIT license rather than the GPL (see pylogics-extras LICENSE). This is needed since after #139 the Pylogics' LICENSE is the MIT License, which is incompatible with GPL. This has been agreed upon in private communication with @paulotguerra, and he can confirm this licensing change on his code by commenting on this PR.

I do confirm the repective licensing change.

Merging this now, we can work on this in separate PRs.


Fix Firefox printing

This PR resolves #3115.
This PR removes the page-break-before: always and page-break-after: always properties from the page elements in both the Blank and 5ePHB theme styling. This corrects the alternate blank page issue seen when printing from Firefox (and presumably other browsers) while having no determinable effect on printing from Chrome.

Ok. I agree I don't see any negative effect to doing this. Merging!
Thanks @G-Ambatte 💯 🎉 ⭐


Documentation and Implementation of Flood fill Algorithm

Implement Flood Fill Algorithm in Java/Cpp and explain the algorithm (proper documentation).
Note :

The code should be well commented so that anyone can understand that.
Proper documentation should be provided (either edit the readme or create a new md file and add a link in the readme file under relevant header.

Can you assign this to me?

@sahanihit sure. You can work on this.

can u please assign it to me


Logger for XPRA reports & change of application log sub-object to customer name space

I have implemented a compatible logger for XPRA report execution which will add messages to the transport log instead of creating an application log in client zero.
As I am still rather new to git I have probably made a few mistakes, and there's a little more included here than what was strictly needed. For instance, I should probably not have staged the package - I see the application component has been committed as a change. Sorry about that.
One thing that should be included, but should probably have been in a separate pull request is the change from 'LOGGER' to 'ZCL_LOGGER' as application log sub-object for unit tests to avoid problems with name space.
I don't know why the zcl_logger.clas.xml is updated with removal of the class documentation. I have even tried changing the documentation and activating it, but it just won't come back.
Unit tests.... yes, I should get started on that topic.
If there's too many beginner mistakes to fix, please reject the pull request. In a while I should be able to create one (or several) that can be accepted more smoothly.

Thank you for your pull request. I am always happy with every contribution. I need time to dig into it (I don't know XPRA). And I'd like to do a pull request with the refactoring I have done first so maybe you will have to adapt some of the code then.

XPRA is simply a mechanism to execute a report as part of a transport import, so it's main use (AFAIK) is to initialize values for existing records in tables, e.g. if you add a field to a table and the existing records all should have the same value or can get their value calculated. This is particularly important when there are other changes in the code in the same transport, and these depend on the value in the new field. By using an XPRA you ensure that the new code works as intended immediately and there is no need to execute an initialization report manually.
XPRAs can of course be used for anything, they are just reports executed without taking a variant into consideration (uses the default values for all parameters and select-options) and executed in client zero. Client zero is the main reason output should be added to the transport log. That's what the new class does. I guess an option could be added to optionally also save the messages in an application log, although I have never seen a need for that.
No problem with waiting, I discovered your refactoring initiative after submitting my pull request. I can adjust the code and create a new pull request later.

I just merged all the refactoring into the master branch. Could you adopt your coding?

I will. I am working Monday and Tuesday before taking some days off, so I am not sure if I can find the time this coming week, but will do it as soon as I find time.
However, I am not very git or github experienced, so how should we handle this? Will you cancel this pull request, and I create a new one after updating the code?


add dm.parallelized_with_batches

Checklist:

[x] Add tests to cover the fixed bug(s) or the new introduced feature(s) (if appropriate).
[ ] Update the API documentation is a new function is added or an existing one is deleted.
[x] Added a news entry.

copy news/TEMPLATE.rst to news/my-feature-or-branch.rst) and edit it.

This is not the first time I needed such a function so likely best to have it here.

$$\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)$$


Open source startup code

At the moment the startup code is provided in binary libmain. The reset vector is in ROM which performs low level initialisation then maps the SPI flash and calls into the entry point at call_user_start.
Startup code includes:

"standard" libc-style setup of memory & interrupts
calling into libpp for radio calibration
setup of the FreeRTOS task which handles wifi functions.
Other things?

As a first step, this process needs to be documented in detail.
It should be possible to write open source routines that give more fine-grained control to the program author.
Some more details (ROM & IoT SDK based, but could be extended to document RTOS startup code):
https://github.com/esp8266/esp8266-wiki/wiki/Boot-Process

First stage is enabling SPI flash cashing so functions from flash can be called (obviously).
@pvvx has a decompiled reconstruction of that sequence here (in Russian):
http://esp8266.ru/forum/threads/revers-sdk-espressif-i-rom-bios-dlja-sozdanija-otkrytogo-sdk.292/#post-5190

setup of the FreeRTOS task which handles wifi functions.

Can we make FreeRTOS be optional at all? Both because it's extra bloat and because of its questionable licensing.

Another effort: http://richard.burtons.org/2015/05/17/decompiling-the-esp8266-boot-loader-v1-3b3/

Can we make FreeRTOS be optional at all? Both because it's extra bloat and because of its questionable licensing.

I think this is something that might become possible. there are quite a few FreeRTOS-specific calls in the binary libraries, but most of them are in libmain. Some shims some other symbols (mostly mutexing) and it might be possible.
I've thought that Contiki would be interesting to try to port at some point in the future, so having an OS-agnostic base some day would be great from that point of view too.
At the very least, the current set of goals would form a solid base for that kind of porting.
Re: OTA bootloader support & rBoot, yes this is an excellent reference thanks. I'll add another issue to track OTA support.

because of its questionable licensing.

To clarify, by "questionable licensing" of FreeRTOS I of course mean not GPL, but exception 2 to it, which, arguably, make the RTOS license not an open-source license.  Whereas FreeRTOS license essentially denies users of basic right of optimizing their software (because if you benchmark software built on FreeRTOS, it's never possible to say precisely how much of your app and how much of FreeRTOS you benchmark, so conservative treatment is that you can't do that at all; and oftentimes, different people do benchmarking vs optimization based on known problem spots, so publishing of benchmarking results is essential, and that's again prohibited by FreeRTOS license).
All in all Open Software/Hardware community should be more selective about what tool they use and avoid double standards in treating hardware vs software vendors.

Yes, I fundamentally agree. And I appreciate that the FreeRTOS restriction is a sticking point to a fully open framework.
The reason I'd like to avoid the Espressif MIT License is primarily that it's currently possible to avoid that restriction, without a large drop in functionality.
Avoiding the FreeRTOS licensing restrictions is currently not possible without disabling nearly all functionality. At least the scope of that license is restricted to a small set of esp8266-specific FreeRTOS functions and some binary linked symbols. In time, the current goals of this project will hopefully bring those binary linked symbols down to a set which can be "shimmed out" and replaced with something else. If that turns out to be impossible, then the very least I can say is that those license restrictions will only ever cover the part of the code which relates to FreeRTOS itself.
I 100% agree with you that becoming RTOS-agnostic would be a great thing. It's the kind of benefit that a FOSS licensed framework can uniquely bring. I definitely don't plan to do any work that goes against the possibilities you're hoping to see.

Ok, thanks for fundamentally agreeing.

currently not possible without disabling nearly all functionality.

Well, nothing in the system calls for the need of RTOS or specifically FreeRTOS: MAC layer delivers packets by an interrupt, the lwIP needs only few timers to run (can be done on interrupts too). The need for "RTOS" comes only at the level of BSD sockets API. And then I'm not 100% sure if it's unavoidable either (let's just assume that lwIP sample code is implemented that way), but then it's still not FreeRTOS-specific. Summing up: starting with plain SDK, it would be possible to arrive at the same functionality, but without dependency on particular RTOS (or on RTOS at all).
All the above can be just considered an independent feedback, and from my side, is an analysis and justification why I may want to go with integrating open builds on lwIP and axTLS into esp-open-sdk (there will still be enough area of collaboration, but no matter what, it's duplication of effort which might have been avoided).

There are a large number of FreeRTOS symbols both implemented in and linked from the binary libraries, particularly libmain.a and libpp.a.

Yes, and libmain.a and libpp.a of normal SDK has 0 number of FreeRTOS symbols, which means you won't need to work them around with shims and stuff. Taking into account that all this work is already a big workaround for lack of open SDK, limiting number of sub-workarounds may seem helpful.
I point this not for the sake of argument or to make you suddenly drop FreeRTOS approach, but to properly document implications of both choices and see whether there would be good grounds for somebody to follow another approach than selected by you. (Well, of course if you'd assess that you haven't gone too far with FreeRTOS and still could switch, that would be even cooler ;-) ).

You make a good point, I missed that by "plain SDK" you meant "IoT SDK".
To be honest I haven't looked too closely at the IoT SDK. When I started this project it was under the "Espressif GPL", and the RTOS SDK was plain MIT, and that more or less made the decision for me. Maybe if I was starting today, now everything has been relicensed, I would think differently.
There are still some unusual symbols in the IoT SDK. IoT SDK libmain.a actually contains an object file "mem_manager.o", linked from the other libraries, whose functions all match FreeRTOS names ie vPortFree, vPortMalloc, xPortGetFreeHeapSize, etc. Looks like it was probably derived from one of the FreeRTOS heap_xx.c memory management implementations.
Also AFAIK noone has figured out the signatures of IoT SDK ROM functions like ets_task and ets_post, which seem to be used internally for message passing. Maybe they are wrappers around XTOS primitives? You might be able to ignore these though, if you just want to run on "bare metal" with a black box wifi stack, and ignore the RTOS in the background.
I'm going to stick with FreeRTOS for now as I have a much better handle on how its parts fit together. There are also a couple of other minor reasons -

Not wanting to rewrite axTLS for lwIP raw sockets (I started down this path back in April but it's quite a substantial set of changes).
Wanting to avoid the "helpful" nature of some of the IoT SDK functions that remember SSIDs, etc.

I will keep an eye out for developments on people deciphering more of the IoT SDK internals, though. There are definitely upsides, like the amount of other code people are writing against it, and it seems like it's getting a lot more attention from Espressif.

IoT SDK libmain.a actually contains an object file "mem_manager.o", linked from the other libraries, whose functions all match FreeRTOS names ie vPortFree, vPortMalloc, xPortGetFreeHeapSize, etc. Looks like it was probably derived from one of the FreeRTOS heap_xx.c memory management implementations.

Yes, I noticed that too, and had the same suspicion either - we know that SDK implementation is a mix of bunch of open-source projects, so maybe even "plain SDK" has some bit taken from FreeRTOS.

Also AFAIK noone has figured out the signatures of IoT SDK ROM functions like ets_task and ets_post, which seem to be used internally for message passing.

Meet system_os_post() and friends in SDK documentation which are wrappers around them.

You might be able to ignore these though, if you just want to run on "bare metal" with a black box wifi stack, and ignore the RTOS in the background.

So, let's just pronounce what the fuzz is all about - it's about getting rid of proprietary callback-style espconn API, and being able to use standard BSD socket API, and standard sequential-style programs. To achieve that, one would need to deal with ets_task and ets_post (and more).

Not wanting to rewrite axTLS for lwIP raw sockets (I started down this path back in April but it's quite a substantial set of changes).

Was it? To say it explicitly, I'm interested in having a compliant MicroPython implementation on esp8266, so there's bit of headroom between providing Python-level socket API and the way it may be implemented in C, but still: Let's look how lwIP's tcp_write() works - it buffers data, and then a callback will be called when it's actually sent. Pardon me, but that's just how send() works - on Linux, it will happily slurp megabytes and return immediately. If there was an error, you will learn that much later. If buffer is full, it will block before space is available. That's all doable with tcp_write + tcp_sent callback (callback sets flags for next call to send() impl to check).
Now recv() part: Well, that's done with tcp_recv callback. No problem, let it put packets in queue. Then recv() will either take next from queue, acknowledge it with tcp_recved(), or block if queue is empty (block here and above means busy-wait, sure).
So, /me fails to see how lwIP API is "substantially" different from BSD API. The only caveat with all the above is to have the data pumping. As long as main loop belongs to SDK code, that's not doable. So, need to apply "inversion of inversion of control" pattern ;-).

and it seems like it's getting a lot more attention from Espressif.

Yes, exactly. For a lot of time it looked like RTOS SDK was  a once-off code drop.

I'm going to stick with FreeRTOS for now

Well, sounds good. There's certainly a difference between knowing how something can be done and actually doing and have something working, if you're well on the road with FreeRTOS SDK on that, makes sense.

Hiho..  I wanted to let folks here know about an effort I've been working on over the past week or two to set up a public site/wiki/maillist specifically for reverse engineering efforts around the ESP8266:
http://esp8266-re.foogod.com/
The site is intended to be fully open to anybody who wants to post whatever info they have gleaned from pretty much any area of reverse engineering these devices.  In particular, something that you may be interested in is I have now posted most of the notes I have from my own efforts at reverse-engineering the startup code (and related data structures, etc) of the Espressif RTOS SDK (About 70-80% thus far, and I hope to have a bit more stuff up soon), in a form which (I hope) should be useful to anybody wanting to reimplement things themselves.
I've also set up a maillist/group as<EMAIL_ADDRESS>which I hope to be useful for people to get together and coordinate this sort of work, so we can all know what everybody else is doing etc.  Anyone interested is welcome and encouraged to join in!

Hi @foogod,
This is super useful, thank you for publishing it! I was somewhat dreading having to do this RE myself (and I don't have an IDA license either - although I have ScratchABit woking with xtensa now which is very helpful).
Although I was a little dismayed to see yet another esp8266 wiki, you've packed it with good info (and it's open access so thank you heaps for doing that.)
I've subscribed to the esp8266-re mailing list. I will definitely be referring to the docs you've published when it comes time to write new startup code for esp-open-rtos, and I'll make sure to contribute edits for any subtle differences I'll work out (and link back to the wiki where appropriate).
Thanks again!

@foogod: Thanks for coming forward with this! My first comment would be that I'd personally prefer that was done on a popular hosting site (like, well, github), to make sure it can be easily and seamlessly continued even if original creator lost interest/resources to keep it up.

Well, I had originally debated whether to set up a new wiki for this, but I really think there are some advantages to having a site specifically dedicated to RE (and all the mess that often entails) separate from the more general-purpose resources out there.  The idea is that the ESP8266-RE wiki can be used for all the gory details of ripping things apart (that only those involved in the work really care about), and then over time the insights which are useful to "normal" folks (those who just want to write apps for the ESP8266, etc) can be cleaned up and posted back to the more mainstream sites in forms that are actually useful to people who aren't reverse-engineers.
As for using GitHub for this, to be honest, I really just don't think GitHub is a good resource for this sort of general community wiki, and there are real advantages to using a much more full featured and extensible wiki system for this sort of thing..  This is something we can certainly discuss more on the esp8266-re maillist if folks want to, though (I don't want to hijack this issue thread too much with those sorts of side-discussions, and talking about the wiki is exactly the sort of thing that the list was created for)..  Regarding the maintainership question, I do definitely understand the concerns, but rest assured that the wiki isn't likely to be going anywhere any time soon, at least (the server it's on is already paid up for the next 10 months or so, MediaWikis are not hard to move from one place/person to another if necessary, and I've got no problems giving additional folks wiki admin access (and access to the offsite backups, etc) to help keep things running if I'm suddenly not around for some reason)
FWIW, most of the RE work on the SDK I've been doing thus far hasn't been with any kind of IDA, but just using xtobjdis (https://bitbucket.org/foogod/xtobjdis), a (non-interactive but full-featured) Xtensa disassembler I put together a couple of months back (I actually wrote it before I knew about ScratchABit, but I tend to prefer working with more unixy non-interactive tools anyway)..  I've also put together a few miscellaneous scripts/tools which I've been using to make working on the ESP8266 SDK easier as well, and am hoping to post some more info on those in the next day or two in case anyone's interested..

So, let's just pronounce what the fuzz is all about - it's about getting rid of proprietary callback-style espconn API, and being able to use standard BSD socket API, and standard sequential-style programs. To achieve that, one would need to deal with ets_task and ets_post (and more).

So, need to apply "inversion of inversion of control" pattern ;-).

Just wanted to drop a note regarding this "inversion of control" — it is quite achievable by calling a custom version of ets_run from a function which needs to block (e.g. read). I used this approach early on during arduino core development, but ultimately discarded it because this caused stack to grow considerably.
Another note regarding "being able to write standard sequential-style programs" in context of IoT SDK. Since we have blocking network APIs in arduino, I know all too well that this causes issues when you have more than one network service running on ESP. While one service (e.g. HTTP) is waiting for more data to arrive, another (like mDNS) gets flooded with data, overflowing all the scarce buffers you have. Without a generous amount of RAM and multithreading, blocking network APIs just don't work well.

Hi @igrr,
Thanks for letting us know about your experiences. I'm guessing it's something of a twofold problem in the Arduino case because of the single threaded default programming style, so there's no obvious way to change the API and add either threads or lots of callbacks without making the API style a lot more complex than the typical Arduino code. Interested to see what you end up doing.
Very impressed by your esp8266 Arduino project in general btw, seems to be coming along in leaps and bounds! :)
Angus

Just wanted to add a note here that I now have a reimplementation of almost all of libmain/app_main.o (including all the startup code) in C source form.  There are a couple of bits I didn't want to finish until we'd sorta sorted out how we were going to be doing the register headers, so I've been sidetracked by issue #8 instead for the past couple of days, but once I've finished up my work on that (hopefully in the next day or two), next on my list is working on integrating and testing the new startup code and submitting a PR here..

Fantastic news! Let me know if I can help with either task.

So I have one question:
For my replacement app_main.c (which is currently a drop-in replacement for libmain/app_main.o), should I keep all the functions/etc prefixed with "sdk_" or not?
Is the "sdk_" prefix intended to convey "these are binary-only routines from the Espressif libs which we don't have source for" (in which case I should be removing the prefix when I create source versions) or does it mean "these are APIs invented by Espressif, not by us" (in which case they should presumably keep the prefix regardless of source/binary form)?

Hmm, yeah. I originally meant is as the first one ("this is a binary routine"), and also just to get around the fact that there were a few namespace collisions otherwise (Espressif had renamed things in lwip/axtls to avoid collisions, and I didn't want to go around renaming parts of upstream code).
However I think if something is a 1:1 recreation of the SDK code, you can make a good case for keeping the prefix. If that feels right to you then do it.
Long-term, I'd like to look at evolving the startup code a bit so it probably won't resemble the SDK startup code as closely. I'm sure you have ideas about this as well though, and just recreating the binary SDK functionality is a sensible first step before we talk about them. :)
(Long-long-term, once the first milesteones goals are all completed I'm thinking the logical second milestone is "remove SDK libmain entirely")

Ok, yeah, the more I think about it the more I think keeping the prefix is probably a good idea..  For one thing, that way if people write apps that use the SDK routines, they don't have to keep changing their code every time we reimplement another piece of things.
It also means that there's a clear distinction between "old API" and "new API", and over time we can pick and choose which routines from the SDK we also want to become "official" parts of our esp-open-rtos API (and give them non-sdk names), and then eventually deprecate the others, etc..
It'll probably also be a bit easier to work with if we don't have to keep tweaking the rename-exclude lists whenever we write new replacements for SDK routines..

Ok, getting closer, but I'm running into another issue..
Because I'm now adding app_main.o to the list of object files to remove, there are some symbols defined in app_main which now aren't getting auto-renamed, because the renaming stuff doesn't consider them part of the SDK libs anymore (but they do need to be renamed in the other libs in order for them to call my versions properly).
I can see a few different ways to resolve this:

Add a way to specify a (manual) list of "extra" symbols which should be renamed even though they don't show up automatically as part of the SDK-defined list.  The problem with this is it would become more and more of a pain to maintain by hand as we do more of this sort of thing.
Split the "remove" step into two:  One round runs before symbol substitution (like now) and another one that runs after (so all the symbols in those objects still go through the renaming phase, but then the objects are ultimately removed after so we can cleanly replace them)
Don't try to dynamically figure everything out each time, but just generate (once) a static list of "SDK symbols" to be renamed, and use that list going forward.  This has the benefit that it would also simplify the makefile considerably (and speed up the build slightly), and we wouldn't have anything change out from under us without realizing it.  The only drawback is it wouldn't be updated automatically if we changed the version of the SDK libs we're using (we'd need to regenerate it), but (given the current licensing situation) it seems very unlikely we'd end up doing that anyway.

I'm personally leaning toward option 3, but wanted to get opinions from other folks..

Oh snap, I should have realised that yesterday when I replied. Sorry.
There's an option (4), which is to drop the sdk_ prefixes from those functions after all. Unless you really feel strongly about it, that's probably the easiest option for now. Especially if you agree that keeping a 1:1 app_main.c is probably an interim measure anyhow, and it's going to evolve pretty quickly from there.
Angus

I dunno.. as I said earlier, the more I think about it the more I think it'd be good to keep the 'sdk_' prefixes for this stuff.. if nothing else it means that the API won't be changing out from everybody as we go along, which I think is kinda important..
The other thing is that some of these exported symbols that Espressif has come up with are named things like "info", which (on top of being wonderfully non-descriptive as a structure name) is just begging for naming conflicts with user code, so I was actually kinda happy to see it turned into "sdk_info" in esp-open-rtos..
What are your feelings on the other options?  I have no problem doing the work to implement one of them, if folks are OK with the approach being taken..

Hmm, solid point re:"info" and similar. And as we dig out more parts of the binary libraries, there will only be more of those names cropping up.
Let's do the static rename list option 3, then. Less makefile complexity is a big plus, and as you say the libraries are unlikely to change in the future. (The more binary functionality gets replaced, the bigger a job that would potentially become anyhow, above and beyond any symbol name changes.)

Good news, everyone!
I finally have a fully working implementation of app_main.c!  It boots and runs and everything!  I'm not going to submit a PR just yet, as I want to do a bit more testing (to be honest, thus far I've only actually confirmed that the 'blink' example works)..  However, it is currently up in my fork in the 'open-startup' branch (https://github.com/foogod/esp-open-rtos/tree/open-startup) if anybody else wants to take a look at it / test it out / etc.  (I'd welcome any help in testing it, as it's kinda a fundamental part of things I'd like to make sure it's really right before potentially merging it into the main line.)

Wow, really impressive work! Just reading the code is quite educational about the ESP startup process.
I can run the 'simple' and 'http_get' examples, and 'http_get' connects to my wifi and pulls down a page so that's very encouraging. :D
I got some very odd behaviour loading the ota_basic example with rboot. Lots of very early boot loops (possibly rboot crashing?), I got it to run only once and I'm not sure how (I think it was a combination of chip erase and power cycling to clear RTC memory), then the next time I pressed  reset it landed in the boot loop. The board I have here (nodemcu v1.0 knockoff) doesn't support 74800bps (nor or 74906bps) so I'll try again tomorrow with a logic analyzer when I'm back in my workshop, see what it's actually outputting before it crashes.

Hmm, actually I get this even without the OTA stuff. If I upload http_get then it works immediately when the bootloader jumps out, but then if I press 'Reset' I get a boot loop. Will investigate further.

Hmm, ok, I see part of the problem, I think.. When it reboots, it's reporting a system param error (which basically means the checksum check on the g_ic saved data it read from flash failed).  This might just be a flaw in the checksum validation, or it might somehow be reading the wrong bit of the flash or something.  Unfortunately, the real problem is that I apparently left out a bug fix I had made to the config-sector-dump routines, so when it reports the system param error, it then calls those routines, which go into an endless loop (oops)..
I've pushed up the fix for the config dump stuff, so at least it shouldn't go into an endless loop when it calls that (sorry about that).  Now I just need to figure out why the checksum validation is failing..

Ok, I think I found it.. I had an off-by-one error when calculating the flash sector to read the g_ic data from..  Should be fixed in my branch now.

Yeah, that was it. Working great now!

Not wanting to rewrite axTLS for lwIP raw sockets (I started down this path back in April but it's quite a substantial set of changes).

@projectgus Just as an information: Some time ago I wrote a "glue" for LWIP raw mode and axTLS and it works quite nicely. If you are interested take a look at Lwirax


Implement unit and e2e tests

This PR adds basic unit and e2e tests to ensure we won't accidentally break functionality while implementing #30
What the pr does:

first restructuring of the source code to improve readability and prepare for further changes
adds unit tests for core functionality and the node api (mostly snapshots, more precise tests follow later on in other PRs)
e2e tests for cli usage

Would be great if @technopagan could enable travis for this repo now :)

Codecov Report

:exclamation: No coverage uploaded for pull request base (next@84ef7d4). Click here to learn what that means.
The diff coverage is 95.52%.

@@           Coverage Diff           @@
##             next      #32   +/-   ##
=======================================
  Coverage        ?   86.53%
=======================================
  Files           ?        5
  Lines           ?      104
  Branches        ?       15
=======================================
  Hits            ?       90
  Misses          ?       11
  Partials        ?        3

Impacted Files
Coverage Δ

src/utils/primitive.js
100% <100%> (ø)

src/utils/helpers.js
100% <100%> (ø)

src/utils/svg.js
100% <100%> (ø)

src/index.js
71.73% <70%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84ef7d4...9aece2e. Read the comment docs.

LGTM, feel free to merge :+1:


Feature request: Receiving messages

Q
A

Bug report?
no

Feature request?
yes

RFC?
no

How used?
Symfony/

Swiftmailer version
Symfony CLI version v4.4.4

PHP version
7.3.1

Observed behaviour

Expected behaviour

Example

Now, Swiftmailer only sends mails...but it hasn't got any method to receive ones.
Regards.

That's something very different and won't be added to Swiftmailer.


Add Metadata Associated to a Series

Proposal: It would be nice if we could store metadata about the series itself, so it could be used for presentation, graphing, etc.
Here is a scenario:
Suppose we have series representing pageviews:
company.uri1.pageview
company.uri2.pageview
company.uri3.pageview
....
The series contain two columns value and timestamp
Ex.
bloghouse.b7cb47c1a9a4642b9882b0cb4393a71fc4dbff44.pageview
value timestamp
1<PHONE_NUMBER>3
32<PHONE_NUMBER>5
42<PHONE_NUMBER>7
....
bloghouse.b7cb47c5454656459882b0cb4393a71fc4dbff44.pageview
value timestamp
1<PHONE_NUMBER>3
52<PHONE_NUMBER>5
76<PHONE_NUMBER>7
....
Now I need to visualize these series in a graph. However, I want to have a nice name to  my series,i.e, the title of the page ("Post About Furnish")  instead of "bloghouse.b7cb47c1a9a4642b9882b0cb4393a71fc4dbff44.pageview" .
It would be nice if we could store metadata about the series itself, so it could be used for presentation, graphing, etc.
Once stored, the metadata could be obtained by querying as well.
:)

:+1:
My main use case would be storing the unit of the data. IMO it's insane that we need to manually add this to graphs in the frontend when we know the unit and kind of data on the collector side.
Metadata for series would also make it possible to ask for 'all networking related stats for cluster A' or 'all postgres stats for server X'. Series metadata is much better suited for this than tags.

You can actually do this using tags. Particularly since there's no limit to the number of tags you can have on a series. The current tag functionality lets you do things like that kind of discovery.
There's a few examples in this post: http://influxdb.com/blog/2015/02/24/influxdb-v0_9_0-rc4_released_now_with_tags_and_more_unicorns.html

@pauldix This is a valid concept and cannot be achieved with tags alone. I have put some additional info on the challenges with the tags approach in this issue: https://github.com/influxdb/influxdb/issues/3445


Deleted obsolete razor views and models

[Fixes #295 ] Disambiguate duplicate Microsoft.AspNetCore.DiagnosticsViewPage.Views.* type names #295

ping


Implement Transcription and Translation types

Removes Audio type in favor of Transcription and Translation types to better reflect what the API requests are returning.

Solves issue #50


Join cluster with kube-ovn enabled fails

Summary
Installing microk8s and enable kube-ovn afterwards works, but joining the nodes to build a cluster fails
Some informations about my setup :
OS : Ubuntu 20.04 LTS
Snap version : 2.59.5
MicroK8s channel : 1.27/stable
What Should Happen Instead?
joining a cluster sucessfully.
Reproduction Steps
Prep nodes:

snap install microk8s --classic --channel=1.27/stable
microk8s enable kube-ovn --force
wait for kube-ovn to settle and then joining the nodes brings then the following error:

# microk8s join <IP_ADDRESS>:25000/11d8000ec66ea79eee24f04c4fee9a88/bbeb88e587ae
WARNING: Hostpath storage is enabled and is not suitable for multi node clusters.

Contacting cluster at <IP_ADDRESS>
Waiting for this node to finish joining the cluster. .. .. .. ..
Traceback (most recent call last):
  File "/snap/microk8s/5372/scripts/wrappers/join.py", line 1033, in <module>
    join(prog_name="microk8s join")
  File "/snap/microk8s/5372/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/snap/microk8s/5372/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/snap/microk8s/5372/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/microk8s/5372/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/snap/microk8s/5372/scripts/wrappers/join.py", line 1024, in join
    join_dqlite(connection_parts, verify, worker)
  File "/snap/microk8s/5372/scripts/wrappers/join.py", line 785, in join_dqlite
    join_dqlite_master_node(info, master_ip, token)
  File "/snap/microk8s/5372/scripts/wrappers/join.py", line 901, in join_dqlite_master_node
    try_initialise_cni_autodetect_for_clustering(master_ip, apply_cni=False)
  File "/snap/microk8s/5372/scripts/wrappers/common/cluster/utils.py", line 309, in try_initialise_cni_autodetect_for_clustering
    if cni_is_patched():
  File "/snap/microk8s/5372/scripts/wrappers/common/cluster/utils.py", line 276, in cni_is_patched
    with open(yaml) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/var/snap/microk8s/5372/args/cni-network/cni.yaml'

aftewards, running microk8s status on this node brings
# microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

calling kubectl binrgs:
# microk8s kubectl get nodes
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "<IP_ADDRESS>")

Introspection Report
inspection-report-20230703_215301.tar.gz

Hi @emalzer
Indeed, this looks like a bug that has slipped through. As a temporary workaround, can you try touch /var/snap/microk8s/current/args/cni-network/cni.yaml before running the join command? Thanks!

Hi @neoaggelos
yes, with this the join works. It seems that the cluster works, though I see two errors:

<30>2023-07-04T20:01:27+02:00 gs-dev01.ong.lab microk8s.daemon-cluster-agent[1253184]: 2023/07/04 20:01:27 WARNING: failed to update cni configuration: "failed to apply cni configuration: failed after 20 retries: command [/snap/microk8s/5372/microk8s-kubectl.wrapper apply -f /var/snap/microk8s/5372/args/cni-network/cni.yaml] failed with exit code 1: exit status 1"
I see this only on the master node and two times, looks like for every join of the two other nodes.

<30>2023-07-04T20:12:30+02:00 gs-dev01.ong.lab microk8s.daemon-apiserver-kicker[728043]: error: the server doesn't have a resource type "felixconfigurations"
I see this on every node every 5 seconds.
I can't tell if this has an impact or not.

Hi @emalzer

Yes, this looks like a bug. it should not break things, other than cause delays when adding cluster nodes.

this does look like a bug. i imagine it will persist trying to patch calico offloading, which should probably be avoided.

Thanks for both points, we will prioritize and fix them soon.

This issue is fixed by https://github.com/canonical/microk8s/pull/4132. Closing this now.


vector: 0.33.1 → 0.34.0

Description of changes
https://github.com/vectordotdev/vector/compare/v0.33.1...v0.34.0
https://vector.dev/releases/0.34.0/
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Thank you !


[docs:#376]: Add README instructions to access LinDB web interface

What changes are proposed ?
Adding instructions to access LinDB web interface.
How was the change tested ?
Markdown on IDE
Issue ?
#376

Thanks for merging @stone1100


Status Codes - Who wins?

Nice to have - if there were some state variable for who is in checkmate.

You should be able to use the game status, which is from the perspective of the active player. So if you reach status == Game.CHECKMATE, you can find the loser with loser = game_instance.state.player
I'm not sure what you're use case is, but be aware that Chessnut doesn't validate boards when a Game instance is instantiation; the behavior is undefined if you're trying to feed in an invalid board -- e.g., if both players are in checkmate. You could work around it by testing the same state for both players if it matters to you, but it's not something I plan to support in the library.

Okay - I see why getting the info I need doesn't require any change in the code. Thanks


Move CI/dev tool usage to use go run pkg@version to reduce the number of vendored dependencies

Which issue this PR addresses:
Fixes my local dev environment because go mod tidy breaks
What this PR does / why we need it:
This:

Removes some dev/tool only dependencies and replaces them with go run pkg@version invocations. This removes a large amount of dependencies that clutter up our vendor folder.
Fixes one or two little imports that include more than they need
Updates one or two dependencies that don't have valid downloads anymore for unknown reasons

Test plan for issue:
CI
Is there any documentation that needs to be updated for this PR?
N/A

for things like go-mock and similar things I'd rather have it in a dockerfile instead of compiling it everytime (since we could run those on github actions and it would be less restrictive)

@facchettos It uses the go module cache, so it should ideally be not too flaky :)


AttributeError: 'builtins.WriteOptions' object has no attribute 'set_sync'

The issue I'm  facing is related to using the annlite executor from Jina Hub, which requires the rocksdict library. The error message "AttributeError: 'builtins.WriteOptions' object has no attribute 'set_sync'"  indicates that the set_sync method is not available in the version of rocksdict being used.
It appears that the newer versions of rocksdict have changed the attribute name from set_sync to sync. This means that you need to update the code that references set_sync to use sync instead.
i tried to install an older version of rocksdict (specifically version 0.2.16) to resolve the issue, but encountered an error that no matching distribution was found.

Thanks for your report. Since rocksdict v0.3.9 it has some broken changes. I just fixed this issue in https://github.com/jina-ai/annlite/pull/225, and please upgrade annlite==0.5.9 to try it.


I ran into an issue

Problem encountered on https://dotnet.microsoft.com/en-us/learn/dotnet/hello-world-tutorial/install
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.

Can you share what problem you ran into?


swap fee now reinvested back into pool

closes #314.
Need to update test cases.

I see something weird.
you mentioned that fee is decomposed to fee-rebate and remainder goes to vault ( fee = fee-rebate + remainder)
but I don't see the remainder part sent to vault.
for example on fixed weight pool, I think there should be a term like (remainder (- fee fee-rebate)), where remainder is sent to vault.
Please let me know if I got something wrong

You see that, for swap-x-for-y, dx moves from sender's wallet to vault, but the sender receives dy based on dx-net-fees. So vault receives dx, but returns dy net of fees. At the same time, balance-x (ie amount of token-x due to LPs by vault) increases by dx-net-fees + fee-rebate, so net net vault balance increases by (fee - fee-rebate), while LPs earn fee-rebate and sender receives dy against dx net of fees.


logger.info(output.decode().strip()) of common.py  will raise a mistake when use convert_office_doc to convert .doc file to .docx file because it is not encode by 'utf-8'

Issue with current documentation:
I want to load doc type file use UnstructuredFileLoader, andI installed all require library, but I got this mistake:
Traceback (most recent call last):
File "D:\soft\Anaconda3\lib\site-packages\unstructured\partition\doc.py", line 67, in partition_doc
convert_office_doc(
File "D:\soft\Anaconda3\lib\site-packages\unstructured\partition\common.py", line 294, in convert_office_doc
logger.info(output.decode().strip())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 42: invalid continuation byte.
Then I see the code of function convert_office_doc that can not change encoding type, this mistake is 'utf-8' is not valid encoding type, so I changed this to 'gb2312', it worked.
So, whether a parameter should be added for the encoding type when reading the file?
Idea or request for content:
So, whether a parameter should be added for the encoding type when reading the file?
For example, UnstructuredFileLoader(file_path, file_name, coding='utf-8').

I solved the problem like this：
import locale
output.decode(locale.getpreferredencoding(False)).strip()

Because encode type is unknow.


Fix and tooltip

work around bug in GuiDropdown by using OptionDataAdvanced
add a tooltip
change to prefix for possible speed up

This PR could also go into Dev.

shouldn't we make this generic?

Finally got the drop down to resize to fit


Use vue-svg-inline-loader with Laravel-Mix

This is not a direct issue but I struggle configuring this with laravel mix. I implemented this in the frontend, on Vue2, and I used the basic webpack config file. In the meantime, the dev team implemented the backend on Laravel along with Laravel-mix. Since then, vue-svg-inline-loader has stopped working. I think the culprit is the misconfiguration, as with laravel mix I do not have the "vue.config.js" file anymore, but I have "webpack.mix.js".
I tried this:
mix.webpackConfig({
    module: {
        rules: [
            {
                test: /\.vue$/,
                use: [
                    {
                        loader: "vue-loader",
                        options: { /* ... */ }
                    },
                    {
                        loader: "vue-svg-inline-loader",
                        options: { /* ... */ }
                    }
                ]
            },
        ]
    },
});

and
    module : {
        rules : [
            {
                use : [
                    'vue-svg-inline-loader',
                    {
                        loader: 'vue-svg-inline-loader',
                        options : {}
                    },
                ]
            },
        ]
    },
});

What am I missing?
My initial, vue.config.js file
module.exports = {
	css: {
		loaderOptions: {
			sass: {
				data: '@import "@/css/all.scss";'
			}
		}
	},
	chainWebpack: config => {
	    config.module
	        .rule("vue")
	        .use("vue-svg-inline-loader")
	            .loader("vue-svg-inline-loader")
	            .options({ /* ... */ });
	}
}

Thank you

Laravel Mix Version: 4.1.2
Node Version: v10.16.0
NPM Version: v6.9.0
OS: macOS Mojave 10.14.6

This appears to fix the problem. Thank you @devonzara !


Removing hpx::util::polymorphic_factory

This PR primarily removes the need of hpx::util::polymorphic_factory and replaces it with the new facilities provided by hpx.serialization. It also simplifies the plain action registration which does not require to register a fake component type anymore.
This is a breaking change since the plain_action registration macros changed to mirror those of the component_action ones.

Could you rebase this on master, please?

done.

everything looks very good, thanks.

The review comments have been addressed.

LGTM


Update task list

Please describe the changes this PR makes and why it should be merged, as well as any and all proof of successful tests:
Added emojis to the task list to more easily show that it's completed or not
Semantic versioning classification:

[ ] This PR changes the framework's interface (methods or parameters added)

[ ] This PR includes breaking changes (methods removed or renamed, parameters moved or removed)

[ ] This PR only includes non-code changes, like changes to README, etc.

Not my commits adding the entire history every single time 💀


Make it more obvious that this board is how we keep track of the tasks on the repo itself

I'll close this one since this is really part of #13


Add KVFile as a root type

I made it extend KVObject for backwards compatibility.
KV3 requires a special root object because it has a header that contains format and encoding data. KV2 might have some extra stuff too.
Perhaps there's a better way of doing this?
    public class KVHeader
    {
        public Guid Encoding { get; set; }
        public Guid Format { get; set; }
    }

I think Serialize should have a new overload added to accept this, rather than changing the existing one because this would be a compat break. But how would this work with Serialize<TData>?

I would avoid using "File" to refer to something that can exist without ever touching disk. Other libraries have gone the route of "Document".
I don't see an obvious "better way".
I would expect that for Serialize() this encoding data should be provided through the KVSerializerOptions object? Ideally with well-known constants/values rather than BYO-GUID (though that should also be allowed).

KVDocument sounds fine.

Ideally with well-known constants/values rather than BYO-GUID (though that should also be allowed).

This is covered in the kv3 pr: https://github.com/ValveResourceFormat/ValveKeyValue/blob/3a0c305f14cccc19860a64c93ee76da245586099/ValveKeyValue/ValveKeyValue/KeyValues3/Encoding.cs
For serializing text files, you wouldn't really need to provide these (default to text encoding in generic format).
I'm not sure about adding them to KVSerializerOptions because deserialize gives you a KVDocument that contains these, would make sense for serialize to also take this, which works for generic kv objects, but not the typed variant.


Find and replace

There is Find Functionality working, working on getting it improved to replace and more powerful capabilities.


serde and serde_derive crates contain a Cargo.toml.orig

I downloaded 1.0.183 manually from crates.io, and both crates contain a Cargo.toml.orig which differs significantly from the Cargo.toml.
I couldnt see any build step here which is causing this file to be created.

This is automatically created by cargo-publish.
If you check other crates, you will notice the same thing.

I learn something new .. thanks.

This file in the published crate:

https://docs.rs/crate/serde_derive/1.0.183/source/Cargo.toml.orig

and this file in the git repository:

https://github.com/serde-rs/serde/blob/v1.0.183/precompiled/serde_derive/Cargo.toml

have the same contents.