systemd timers can result in changed services not being restarted reliably

The original title of this issue was

systemd services are not restarted reliably when package is upgraded

It was renamed after the root cause was identified to be systemd timers in https://github.com/NixOS/nixpkgs/issues/49415#issuecomment-601936292 and after.
Issue description
I just deployed the upgrade from Consul 0.9.3 -> 1.3.0 (#49165) to my 18.03 server cluster (in my case, I upgraded from 1.0.6).
The machines run identical configurations deployed by nixops, and deploying should have stopped all old consul processes and launched the new version.
Yet I found that on 2 out of 7 machines, the old consul version was still running.
On a node where the deployment worked, the journalctl looks like this:
Oct 29 14:44:42 node-1 nixos[22687]: switching to system configuration /nix/store/fxfk4mfi5b90h1vl0hj99gy57zzpl9hy-nixos-system-node-1-18.03pre-git
Oct 29 14:44:42 node-1 systemd[1]: Stopping rsync CDN files to CDN nodes...
Oct 29 14:44:42 node-1 systemd[1]: Stopping ceph-mount.service...
Oct 29 14:44:42 node-1 systemd[1]: Stopping cephfsRunningCheckWatchdog.service...
Oct 29 14:44:42 node-1 systemd[1]: Stopping Nginx Web Server...
Oct 29 14:44:42 node-1 systemd[1]: Stopping consul-alerts.service...
Oct 29 14:44:42 node-1 systemd[1]: Stopped target Local File Systems.
Oct 29 14:44:42 node-1 systemd[1]: Stopped target All Network Interfaces (deprecated).
Oct 29 14:44:42 node-1 systemd[1]: Stopped target Remote File Systems.
Oct 29 14:44:42 node-1 systemd[1]: Stopped server.service.
Oct 29 14:44:42 node-1 systemd[1]: Stopped consul-alerts.service.
Oct 29 14:44:42 node-1 systemd[1]: Stopping consul.service...
Oct 29 14:44:42 node-1 consul[22232]:     2018/10/29 14:44:42 [INFO] Caught signal:  terminated
Oct 29 14:44:42 node-1 systemd[1]: Stopped cephfsRunningCheckWatchdog.service.
Oct 29 14:44:42 node-1 systemd[1]: Stopped Nginx Web Server.
Oct 29 14:44:43 node-1 systemd[1]: Stopped rsync CDN files to CDN nodes.
Oct 29 14:44:43 node-1 systemd[1]: Reloading.
Oct 29 14:44:43 node-1 systemd[1]: consul.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Oct 29 14:44:43 node-1 systemd[1]: Started Create or wait for ceph..
Oct 29 14:44:43 node-1 systemd[1]: Starting ceph-mount.service...
Oct 29 14:44:43 node-1 systemd[1]: Reloading D-Bus System Message Bus.
Oct 29 14:44:43 node-1 systemd[1]: Reloaded D-Bus System Message Bus.
Oct 29 14:44:43 node-1 systemd[1]: Started ceph-mount.service.
Oct 29 14:44:43 node-1 systemd[1]: Starting ceph-mounted.service...
Oct 29 14:44:43 node-1 consul[22232]:     2018/10/29 14:44:43 [INFO] Exit code: 1
Oct 29 14:44:43 node-1 systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 14:44:43 node-1 systemd[1]: consul.service: Failed with result 'exit-code'.
Oct 29 14:44:43 node-1 systemd[1]: Starting consul.service...
Oct 29 14:44:43 node-1 systemd[1]: Started consul.service.
Oct 29 14:44:43 node-1 systemd[1]: Starting consulReady.service...
Oct 29 14:44:43 node-1 systemd[1]: Started ceph-mounted.service.
Oct 29 14:44:43 node-1 systemd[1]: Starting Nginx Web Server...
Oct 29 14:44:43 node-1 systemd[1]: Started Nginx Web Server.
Oct 29 14:44:43 node-1 systemd[1]: Starting ceph-mounted.service...
Oct 29 14:44:43 node-1 systemd[1]: Starting camPostgresRole.service...
Oct 29 14:44:43 node-1 systemd[1]: Reached target Local File Systems.
Oct 29 14:44:43 node-1 systemd[1]: Reached target All Network Interfaces (deprecated).
Oct 29 14:44:43 node-1 systemd[1]: Reached target Remote File Systems.
Oct 29 14:44:43 node-1 systemd[1]: Started ceph-mounted.service.
Oct 29 14:44:43 node-1 systemd[1]: Started camPostgresRole.service.
Oct 29 14:44:43 node-1 systemd[1]: Starting server.service...
Oct 29 14:44:44 node-1 systemd[1]: Started consulReady.service.
Oct 29 14:44:44 node-1 systemd[1]: Started rsync CDN files to CDN nodes.
Oct 29 14:44:44 node-1 systemd[1]: Starting consul-alerts.service...
Oct 29 14:44:44 node-1 systemd[1]: Started server.service.
Oct 29 14:44:44 node-1 systemd[1]: Started consul-alerts.service.
Oct 29 14:44:44 node-1 nixos[22687]: finished switching to system configuration /nix/store/fxfk4mfi5b90h1vl0hj99gy57zzpl9hy-nixos-system-node-1-18.03pre-git

Note the presence of Stopping consul.service....
On a node where it didn't work, it looks like this:
Oct 29 13:52:38 node-2 nixos[27726]: switching to system configuration /nix/store/1x475px1qyk9f9gcdk0pvpbfhip0v1rs-nixos-system-node-2-18.03pre-git
Oct 29 13:52:38 node-2 systemd[1]: Stopping rsync CDN files to CDN nodes...
Oct 29 13:52:38 node-2 systemd[1]: Stopped target Multi-User System.
Oct 29 13:52:38 node-2 systemd[1]: Stopping server.service...
Oct 29 13:52:38 node-2 systemd[1]: Stopping consul-alerts.service...
Oct 29 13:52:38 node-2 systemd[1]: Stopped target Local File Systems.
Oct 29 13:52:38 node-2 systemd[1]: Stopped target All Network Interfaces (deprecated).
Oct 29 13:52:38 node-2 systemd[1]: Stopped target Remote File Systems.
Oct 29 13:52:38 node-2 systemd[1]: Stopped server.service.
Oct 29 13:52:38 node-2 systemd[1]: Stopped rsync CDN files to CDN nodes.
Oct 29 13:52:38 node-2 systemd[1]: Starting ceph-mounted.service...
Oct 29 13:52:38 node-2 systemd[1]: Started ceph-mounted.service.
Oct 29 13:52:51 node-2 systemd[1]: Stopped consul-alerts.service.
Oct 29 13:52:51 node-2 systemd[1]: Starting consulReady.service...
Oct 29 13:52:51 node-2 systemd[1]: Started consulReady.service.
Oct 29 13:52:51 node-2 systemd[1]: Started rsync CDN files to CDN nodes.
Oct 29 13:52:51 node-2 systemd[1]: Reloading.
Oct 29 13:52:51 node-2 systemd[1]: consul.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Oct 29 13:52:51 node-2 systemd[1]: cdn-rsync.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Oct 29 13:52:51 node-2 systemd[1]: nginx.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Oct 29 13:52:51 node-2 systemd[1]: Reloading D-Bus System Message Bus.
Oct 29 13:52:51 node-2 systemd[1]: Reloaded D-Bus System Message Bus.
Oct 29 13:52:51 node-2 systemd[1]: Stopping Nginx Web Server...
Oct 29 13:52:51 node-2 systemd[1]: Stopped Nginx Web Server.
Oct 29 13:52:51 node-2 systemd[1]: Starting ceph-mounted.service...
Oct 29 13:52:51 node-2 systemd[1]: Started ceph-mounted.service.
Oct 29 13:52:51 node-2 systemd[1]: Starting Nginx Web Server...
Oct 29 13:52:51 node-2 systemd[1]: Started Nginx Web Server.
Oct 29 13:52:51 node-2 systemd[1]: Starting ceph-mounted.service...
Oct 29 13:52:51 node-2 systemd[1]: Starting consulReady.service...
Oct 29 13:52:51 node-2 systemd[1]: Starting camPostgresRole.service...
Oct 29 13:52:51 node-2 systemd[1]: Reached target Local File Systems.
Oct 29 13:52:51 node-2 systemd[1]: Reached target Remote File Systems.
Oct 29 13:52:51 node-2 systemd[1]: Starting stolonCreatePostgresDataDir.service...
Oct 29 13:52:51 node-2 systemd[1]: Reached target All Network Interfaces (deprecated).
Oct 29 13:52:51 node-2 systemd[1]: Started ceph-mounted.service.
Oct 29 13:52:51 node-2 systemd[1]: Started stolonCreatePostgresDataDir.service.
Oct 29 13:52:51 node-2 systemd[1]: Started camPostgresRole.service.
Oct 29 13:52:51 node-2 systemd[1]: Starting server.service...
Oct 29 13:52:51 node-2 systemd[1]: Started consulReady.service.
Oct 29 13:52:51 node-2 systemd[1]: Starting stolonClusterDataReady.service...
Oct 29 13:52:51 node-2 systemd[1]: Starting consul-alerts.service...
Oct 29 13:52:51 node-2 systemd[1]: Started stolonClusterDataReady.service.
Oct 29 13:52:51 node-2 systemd[1]: Starting stolonConfigChange.service...
Oct 29 13:52:51 node-2 systemd[1]: Started stolonConfigChange.service.
Oct 29 13:52:52 node-2 systemd[1]: Started consul-alerts.service.
Oct 29 13:52:52 node-2 systemd[1]: Started server.service.
Oct 29 13:52:52 node-2 systemd[1]: Reached target Multi-User System.
Oct 29 13:52:52 node-2 nixos[27726]: finished switching to system configuration /nix/store/1x475px1qyk9f9gcdk0pvpbfhip0v1rs-nixos-system-node-2-18.03pre-git

Note the absence of Stopping consul.service...
This is despite nixops having the following output:
ceph-deployment> closures copied successfully
node-2..............> uploading key ‘secrets’...
node-2..............> uploading key ‘nginxSSLprivkey’...
node-2..............> uploading key ‘cdn-rsync-ssh-privkey’...
node-2..............> updating GRUB 2 menu...
node-2..............> stopping the following units: cdn-rsync.service, server.service, consul-alerts.service, consul.service
node-2..............> Job for consul.service canceled.
node-2..............> activating the configuration...
node-2..............> warning: user ‘postgres’ has unknown group ‘postgres’
node-2..............> setting up /etc...
node-2..............> setting up tmpfiles
node-2..............> reloading the following units: dbus.service
node-2..............> restarting the following units: nginx.service
node-2..............> starting the following units: cdn-rsync.service, server.service, consul-alerts.service, consul.service
node-2..............> activation finished successfully
ceph-deployment> deployment finished successfully

Observe how:

stopping the following units: ..., consul.service was printed
but on the machine, Stopping consul.service didn't occur.

Possibly related is the systemd warning consul.service: Current command vanished from the unit file.

Did you have
 } // (optionalAttrs (cfg.leaveOnStop) {
          ExecStop = "${cfg.package.bin}/bin/consul leave";
        });

Enabled?  Perhaps leaving took to long and systemd bailed out?
If you can tell me whether you had this option enabled or not I can try to reproduce the issue

I actually think that that config is buggy!
It should be ExecStopPost= consul leave
ExecStop is responsible for killing the PID which consul leave doesn't do iirc

@arianvp No, I have no ExecStop in systemctl cat consul.
I have:
ExecReload=/nix/store/kl0vlvk2bd6f1hls53xkvkixlz6vp72r-consul-1.3.0-bin/bin/consul reload
<EMAIL_ADDRESS>consul agent -config-dir /etc/consul.d -config-file /etc/consul.json -conf>
ExecStartPre=/nix/store/453nm1q1r7rzqzh5arx6z04qh1klg7f0-unit-script/bin/consul-pre-start

Also, I think that if systemd tried to stop consul at all, even via ExecStop, it would have printed Stopping consul.service... (which it didn't), wouldn't it?

The difference to #33172 is that there it at least says Stopping Nginx Web Server....

I just noticed
node-2..............> Job for consul.service canceled.

This is a string that systemctl stop returns when  it fails to stop the job (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768208 for example)

Current theory:
This is a string that systemctl stop returns when  it fails to stop the job (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768208 for example)
This can occur when two conflicting actions are queued at the same time. Like a restart and a stop
Now, you consul process seems to have failed with an non-zero exit code:
Oct 29 14:44:43 node-1 consul[22232]:     2018/10/29 14:44:43 [INFO] Exit code: 1

and Restart="on-failure" is set. Which means it would have been triggered to restart but at the same time
we were trying to systemctl stop it, causing a Job for consul.service canceled. error.
Could you look up in the logs, why  consul crashed with a non-zero exit code?

Further research:
SIGTERM causes a consul server to exit with exit code 1, which could trigger the Restart , and thus a race condition occurs between systemctl stop and Restart, causing the stop job to be canceled.
$ nix run nixpkgs.consul
$ consul agent -data-dir /tmp/consul -server=true &
$ pkill -TERM consul
    2018/10/31 16:54:40 [INFO] Exit code:  1

Whilst non-servers gracefully exit on SIGTERM:
$ consul agent -data-dir /tmp/consul -server=false &
$ pkill -TERM consul
    2018/10/31 16:54:40 [INFO] Exit code:  0

Job for consul.service canceled.

This is a string that systemctl stop returns when it fails to stop the job

I can actually reproduce this part.
If I just type systemctl stop consul on my deployed system, I immediately get Job for consul.service canceled.
Afterwards systemctl status consul shows the service as running.
However, journalctl also shows that consul did get a stop command (agent: Caught signal:  terminated) and exited 1. But then it restarted (with Starting consul.service... before it).

[root@node-3:~]# systemctl cat consul
# /nix/store/amy5ccykgajhc2q75mkbjv362z599xvr-unit-consul.service/consul.service
[Unit]
After=network.target
X-Restart-Triggers=/nix/store/0aa2x43nwmid5vwcbqxq6a1m5hpck6jh-etc-consul.json /nix/store/fiawx2232c8z11jca84b1ibd7zfjjz9d-etc-server-service.json /nix/store/5pk>

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/v2hvbpzhndgjm04djqql93bmb4w69gql-glibc-locales-2.26-131/lib/locale/locale-archive"
Environment="PATH=/nix/store/2afkicxfldn57gp22l45xavbpi4sa57g-iproute2-4.15.0/bin:/nix/store/iywd02hbirf506q740z0v6zyrvsa9gcf-gnugrep-3.1/bin:/nix/store/crda0ambd6cjfgp>
Environment="TZDIR=/nix/store/k8n4sf0yfas70yfv156knrpv9xfipjwm-tzdata-2017c/share/zoneinfo"

ExecReload=/nix/store/kl0vlvk2bd6f1hls53xkvkixlz6vp72r-consul-1.3.0-bin/bin/consul reload
<EMAIL_ADDRESS>consul agent -config-dir /etc/consul.d -config-file /etc/consul.json -config-file /et>
ExecStartPre=/nix/store/453nm1q1r7rzqzh5arx6z04qh1klg7f0-unit-script/bin/consul-pre-start
PermissionsStartOnly=true
Restart=on-failure
TimeoutStartSec=0
User=consul

From the doc regarding Restart= :

Setting this to on-failure is the recommended choice for long-running services, in order to increase reliability by attempting automatic recovery from errors. For services that shall be able to terminate on their own choice (and avoid immediate restarting), on-abnormal is an alternative choice.

I'm not sure but maybe setting Restart=on-abnormal in the service unit would allow you to stop the service manually? i.e. the stop job would not be canceled immediately

Related:
The NixOS consul systemd service sets TimeoutStartSec = "0";, which is likely wrong (at least neither me nor boucman from IRC #systemd can guess what that's for).
I asked about that on https://github.com/NixOS/nixpkgs/commit/9ce0c1cb7113#r23138805 but never got an answer from @wkennington.

Another service that have a suspicious TimeoutStartSec=0:
https://github.com/NixOS/nixpkgs/blob/89ede978a0614fd7215e0aa2c483df142cc79af9/nixos/modules/services/monitoring/osquery.nix#L81
(I commented).

I have filed one systemd issue that seems to be at least related: https://github.com/systemd/systemd/issues/10626
I don't think the timer issue explains the full situation, because of the

2 out of 7 machines

from the issue description where consul didn't get restarted, only 1 had a systemd timer installed.

I am getting more and more convinced that the Job for ... canceled. issue and the problem of an old version running are totally separate.
In particular, I suspect that the old-version problem has more to do with Current command vanished from the unit file, execution of the command list won't be resumed..
I have done the following experiment, which supports this:
I've started modifying https://github.com/NixOS/nixpkgs/blob/c9e13806267f7fd3351d52a19cc6db5fa2985ca9/nixos/modules/services/networking/consul.nix#L181
adding more and more -ui switches and deploying with nixops in between. After the deploy is done, I run ps aux | grep consul | grep agent to see what command line is running.
After a deploy with -ui -ui -ui -ui -ui (5 -ui), ps shows only 4 -ui:
Nov 02 17:52:52 node-1 systemd[26291]: consul.service: Executing: consul agent -config-dir /etc/consul.d -ui -ui -ui -ui -config-file /etc/consul.json -config-file /etc/consul-addrs.json

That is even though systemctl cat shows 5 -ui:
# systemctl cat consul.service | grep ExecStart
<EMAIL_ADDRESS>consul agent -config-dir /etc/consul.d -ui -ui -ui -ui -ui   -config-file /etc/consul.json   -config-file /etc/consul-addrs.json

I had systemd-analyze log-level debug on, and in journalctl -f could see that indeed systemd did start consul with the command line showing only 4 -ui:
Nov 02 17:52:51 node-1 nixos[26199]: switching to system configuration /nix/store/z0c30mny9rmsywvisf318wafqxqcb8ck-nixos-system-node-1-18.03pre-git
Nov 02 17:52:51 node-1 systemd[1]: Stopping consul.service...
Nov 02 17:52:51 node-1 consul[14968]:     2018/11/02 17:52:51 [INFO] agent: Caught signal:  terminated
Nov 02 17:52:52 node-1 consul[14968]:     2018/11/02 17:52:52 [INFO] agent: Exit code: 1
Nov 02 17:52:52 node-1 systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 17:52:52 node-1 systemd[1]: consul.service: Failed with result 'exit-code'.
Nov 02 17:52:52 node-1 systemd[1]: Starting consul.service...
Nov 02 17:52:52 node-1 systemd[26286]: cephfs-mounted.service: Executing: /nix/store/xwb1wk1vxjvq4vwxfbphggygfqcb15si-unit-script/bin/cephfs-mounted-pre-start
Nov 02 17:52:52 node-1 systemd[26287]: consul.service: Executing: /nix/store/fqk7rpkdf9741blndfkrlxywdyl477j8-unit-script/bin/consul-pre-start
Nov 02 17:52:52 node-1 systemd[1]: Started consul.service.
Nov 02 17:52:52 node-1 systemd[26291]: consul.service: Executing: consul agent -config-dir /etc/consul.d -ui -ui -ui -ui -config-file /etc/consul.json -config-file /etc/consul-addrs.json
Nov 02 17:52:52 node-1 consul[26291]: ==> Consul agent running!
Nov 02 17:52:52 node-1 systemd[1]: Reloading.
Nov 02 17:52:52 node-1 systemd[1]: consul.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Nov 02 17:52:52 node-1 nixos[26199]: finished switching to system configuration /nix/store/z0c30mny9rmsywvisf318wafqxqcb8ck-nixos-system-node-1-18.03pre-git

And in this case, no Job for consul.service canceled was printed in nixops.

Trying this experiment more times, I also observe deploys where Job for consul.service canceled is printed, but consul starts with the correct number of -uis nevertheless.

I wrote a script to ease debugging this automatically, but encountered another systemd issue that stopped my script from being effective, because the systemd timer randomly stopped firing at all:
https://github.com/systemd/systemd/issues/6680#issuecomment-435597174

After adding the Requires= from timer to service (as described in the systemd link), with my script I can now reproduce the issue of the old ExecStart line running still after deploy within < 10 deploys on average (so in around < 2 minutes).
My script does nothing special, just adding more -uis as described above and checking whether they are reflected in ps; and that in a loop, printing after how many nixops deploys the change didn't get reflected.

OK, so some more info:
I have an rsync job with a systemd timer; the rsync job service Requires = consul.service.
Involved are: consul.service, cdn-rsync.service, cdn-rsync.timer.
See https://gist.github.com/nh2/be476d7ce3466ea5b92dfbc39b235770 for their contents.
It seems like at least for one of the "2 out of 7 machines" (the other one doens't run a timer, so there it's totally unclear to me so far), the deploy not getting the new consul started correctly is dependent on what those timers do.
@arianvp and me know from observing systemd (https://github.com/systemd/systemd/issues/10626) that a timer with OnActiveSec= or OnUnitActiveSec=, will, contrary to expectations of people on the #systemd IRC channel, immediately restart the service when it goes down.
I suspect that what happens is this:

switch-configuration does systemctl stop consul.service
That stops cdn-rsync.service; the timer immediately restarts it, and this immediately brings up consul

racing against the systemctl daemon-reload and subsequent systemctl start consul that switch-configuration does

I haven't been able to reproduce the probelm if my timer uses OnUnitInactiveSec= instead, e.g. to 3s. This is probably because in contrast to the other two options above, this one will actually wait the given amount of time after cdn-rsync.service gets pulled down by the stop of consul, and waiting 3 seconds is enough for nixops to finish the deploy in that case.
Still racy, but an apparent workaround in my situation.
It is still unclear whether this also explains the same issue on the other machine (likely not), and if #33172 has some similar situation.

Yea, it really seems to be a race condition in there.
If I set OnUnitInactiveSec = "500ms", I can reproduce the issue super quickly, with OnUnitInactiveSec = "3s" not at all.
This doesn't seem like a great solution though, magic sleeps will start breaking when other participants in the race are slower some day.

Here's my quick script btw:
#!/usr/bin/env python3

import argparse
import sys
import os
import subprocess
import time

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('-d', '--deployment', type=str, help='deployment name')
parser.add_argument('--machine-name', type=str, help='nixops machine name to --include')
parser.add_argument('--ssh-host', type=str, help='SSH connection string to connect to for running ps')
args = parser.parse_args()

deployment = args.deployment
machine_name = args.machine_name
ssh_host = args.ssh_host

consul_nix = "../../nix-channel/nixpkgs/nixos/modules/services/networking/consul.nix"
line_no = 182 - 1

def update_consul_file():
  with open(consul_nix) as f:
    consul_nix_lines = f.read().split('\n')

  line = consul_nix_lines[line_no]
  consul_nix_lines[line_no] = line[:len(line)-1] + ' -ui"'

  # print(consul_nix_lines[line_no])

  with open(consul_nix, 'w') as f:
    f.writelines('\n'.join(consul_nix_lines))

def deploy():
  subprocess.check_call(['./ops', 'deploy', '-d', deployment, '--include', machine_name])

def count_ui():
  ps_out = subprocess.check_output(['ssh', ssh_host, 'ps aux | grep "[c]onsul agent"'], universal_newlines=True)
  count = len([word for word in ps_out.split(' ') if word == '-ui'])
  return count

update_consul_file()
deploy()
initial_count = count = count_ui()
while True:
  update_consul_file()
  deploy()
  new_count = count_ui()
  print(new_count)
  if new_count == count:
    print("-ui count failed to change!")
    print("initial_count:", initial_count)
    print("new_count:", new_count)
    break
  count = new_count

I use it like ./consul-debug.py --deployment mydeployment --machine-name node-1 --ssh-host<EMAIL_ADDRESS>

@nh2 so this was an issue with timers or a problem on consul module?

This is a problem with timers giving a conflicting job queue on the systemd-side of things every once in a while.  We opened an upstream issue for this. https://github.com/systemd/systemd/issues/10626
I'm not sure if there's anything for us to fix here on the NixOS side of things. I'd be OK with closing.

Given that NixOS forces Consul to be restarted (even if it is server), we can adapt consul module to prevent such. Set skip_leave_on_interrupt = false, remove ExecStop (so stopping/leaving will be done using SIGINT) and set systemd.services.consul.stopIfChanged = false; (so stop&start are combined into single restart). I assume that single restart won't have such side effects.
But yeah, this is very service related and looks quite like a hack.

@arianvp, @danbst, I agree with you that there's nothing really generically wrong on the NixOS side of things. Feel free to open a PR to change some behaviour on the consul module if appropriate.

Given that NixOS forces Consul to be restarted (even if it is server), we can adapt consul module to prevent such.

@danbst I don't quite understand. My original issue was that NixOS does NOT restart Consul when I expect it, making rolling upgrades unreliable.
The systemd issue https://github.com/systemd/systemd/issues/10626 where I commented by now indicates that even Facebook is hitting this problem in systemd btw.

@nh2 you are right :man_facepalming:. What would be best solution, what you think?

@danbst I am not sure, but until anybody from systemd has commented there, I don't think it's clear that this is not a NixOS issue, and we should probably reopen.

We ran into the same problem at Awake Security.  I found the root cause here and I have a minimal reproduction.
First off, here is the reproducing NixOS configuration (for EC2, which you can tweak to your desired physical configuration):
let
  # To reproduce the bug:
  #
  # * Deploy this configuration "as is"
  # * Then increment "FOO" to "1"
  # * Then redeploy the configuration
  #
  # If the reproduction succeeds the `restart-child` service should still be
  # erroneously displaying `0` instead of `1`
  FOO = "0";

in
  { imports = [ <nixpkgs/nixos/modules/virtualisation/amazon-image.nix> ];

    ec2.hvm = true;

    # This is the service that the `switch-to-configuration` script will fail to
    # update correctly.  This service prints the value of the `FOO` environment
    # variable and will always lag behind by one deploy due to being prematurely
    # restarted in the middle of a `switch-to-configuration` step.
    systemd.services.restart-child = {
      enable = true;

      wantedBy = [ "restart-parent.service" ];

      environment = { inherit FOO; };

      script = ''
        while true; do echo "$FOO"; sleep 1; done
      '';
    };

    # This service is responsible for restarting `restart-child` every second,
    # since the bug is triggered by prematurely restarting a service that is
    # being upgraded by NixOS.
    systemd.services.restart-parent = {
      enable = true;

      wantedBy = [ "multi-user.target" ];

      script = "\n";
    };

    systemd.timers.restart-parent = {
      wantedBy = [ "timers.target" ];

      timerConfig = {
        Type = "oneshot";

        OnUnitInactiveSec = "1s";

        AccuracySec = "1s";
      };
    };

    # This service adds a delay to the `systemctl stop …` step of the
    # `switch-to-configuration` script so that the `restart-parent` service has
    # enough time to restart the `restart-child` service.
    systemd.services.delay-stop = {
      enable = true;

      wantedBy = [ "multi-user.target" ];

      environment = { inherit FOO; };

      script = ''
        while true; do sleep 1; done
      '';

      # The purpose of this `preStop` delay is to give the `restart-parent`
      # service a chance to restart the `restart-child` service before
      # the `switch-to-configuration` script can run `systemctl daemon-reload`
      preStop = ''
        sleep 10
      '';
    };
  }

Here's the explanation of what triggers the bug:

The user runs the switch-to-configuration switch script (either directly or indirectly via nixos-rebuild switch), which normally performs the following actions:

systemctl stop all changed services
systemctl daemon-reload to load the service changes
systemctl start all changed services

In the middle of the systemctl stop something else on the machine prematurely restarts one of the services being stopped

This prematurely restarted service will still reflect the old NixOS configuration

systemctl daemon-reload does nothing to interrupt this stale running service

I am fairly certain this is systemd behaving as expected

systemctl start does nothing to replace the old running service (because it's already running, albeit stale)

I'm also fairly certain this is systemd behaving as expected

This leads me to conclude that this is a NixOS bug and not a systemd bug.  Specifically, I believe the most appropriate fix is to change the switch-to-configuration script to replace systemctl start with systemctl restart.

@Gabriel439 Great investigation!
Restarting makes sense to me.
But would we not also have to do stop after daemon-reload?
My understanding is that stop is done to get rid of services that were deleted from the configuration. In the current racy setting, it can happen that you remove a.service, stop happens, something restarts a.service, and then daemon-reload and start happen, so the old service is still running. Even when changing the last start to restart, it would still be running.
So it seems that the "real" stop can only be after daemon-reload.

@nh2: Yeah, I like the idea of just moving the stop after the daemon-reload instead

systemctl restart has the problem that it is less correct and can lead to subtle bugs. Hence we don't default to it. See
       systemd.services.<name>.stopIfChanged
           If set, a changed unit is restarted by calling systemctl stop in the old
           configuration, then systemctl start in the new one. Otherwise, it is restarted in a
           single step using systemctl restart in the new configuration. The latter is less
           correct because it runs the ExecStop commands from the new configuration.

@arianvp: Why would moving stop after daemon-reload have the same problem?

After the daemon-reload the unit of the new generation is loaded, hence you execute ExecStop of the new version of the unit instead of the old one; which is wrong.

@nh2 @Gabriel439 would setting stopIfChanged = false; fix this issue for you?
Maybe we close this by documenting this better?   "If you want transactional stops & starts, please enale stopIfChanged = false" .
"If you want to sacrifice transactionality for  usecase where ExecStop is in the old generation and ExecStart is in the new, please leave it to the default true"
Maybe also defaulting to false is just the better option here... But I'm not sure.

@arianvp: Individually setting stopIfChanged = false would not fix it because there are too many services we'd have to mark in this way
Something that would fix things for us would be to add a custom X-RestartOnReconfiguration option or something similar.  That way we could ensure that any reverse dependency which triggers restarts of other services could always be stopped and started alongside the upgraded services to avoid triggering a premature restart.

systemctl start with systemctl restart.

And another problem: I found that restart does not start a service if it is already stopped (found as part of https://github.com/NixOS/nixpkgs/issues/72394#issuecomment-611499518, where mdadm was in state failed (Result: exit-code), and redeploying with nixops did not restart it). Only start seems to do that.
Does systemd have some restart-or-start-if-not-already-started command?

@nh2 the restart behaviour comes from nixos/modules/system/activation/switch-to-configuration.pl. Unfortunately, it's not really well documented, contains some bugs (https://github.com/NixOS/nixpkgs/issues/84633#issuecomment-610927139, https://github.com/NixOS/nixpkgs/issues/73853, https://github.com/NixOS/nixpkgs/pull/73871) and more.

Something that would fix things for us would be to add a custom X-RestartOnReconfiguration option or something similar. That way we could ensure that any reverse dependency which triggers restarts of other services could always be stopped and started alongside the upgraded services to avoid triggering a premature restart.

I don't fully understand what you are saying here.  systemd services restart in isolation unless PartOf= is set. That is the restart of one service should never trigger the restart of another
What you're suggesting is also exactly how restartIfChanged = true && stopIfChanged = false is implemented.  theyr'e a property on the unit, that gets read during activation
                    if (!boolIsTrue($unitInfo->{'X-StopIfChanged'} // "yes")) {
                        # This unit should be restarted instead of
                        # stopped and started.
                        $unitsToRestart{$unit} = 1;
                        recordUnit($restartListFile, $unit);

I commented more in detail on how to possibly adress this in https://github.com/NixOS/nixpkgs/issues/49528#issuecomment-622196884.
The issue mentioned here is an instance of the "bug" mentioned there (only reloading, even though a restart would be necessary).
Let's move the discussion over there, closing here.

I keep encountering this; posting a concrete situation again, this time with the nebula VPN service (and a timer that starts a service that requires nebula@servers.service):
updating GRUB 2 menu...
stopping the following units: my-server.service<EMAIL_ADDRESS>Job for<EMAIL_ADDRESS>canceled.
activating the configuration...
warning: user ‘postgres’ has unknown group ‘postgres’
setting up /etc...
reloading user units for root...
setting up tmpfiles
reloading the following units: dbus.service
starting the following units: my-server.service<EMAIL_ADDRESS>activation finished successfully

Here we see again: Job for<EMAIL_ADDRESS>canceled.

@arianvp

systemctl restart has the problem that it is less correct and can lead to subtle bugs. Hence we don't default to it.

The current order is:
1. stop
2. activate configuration
3. (restart if opted in with `stopIfChanged = false` ; intead of stop)
4. daemon-reload
5. start

Cannot we just replace 5. start by restart?
The service would still be stop in the earlier step.
But restart would make sure that if a timer triggered a restart duing 1. stop,
I believe that's what @Gabriella439 meant in https://github.com/NixOS/nixpkgs/issues/49415#issuecomment-601936292: To replace 1. stop + 5. start by 1. stop + 5. restart, not to replace the whole thing by a single restart.

I am reopening this issue to

The issue mentioned here is an instance of the "bug" mentioned [in #49528] (only reloading, even though a restart would be necessary).

@flokli That isn't quite correct, at least not as written.
In here, reloading isn't involved at all. This issue tracks the fact that even start + stop does not work, because the start can be replaced by a too-early restart in the presence of timers.
I am re-opening this issue because I believe we can fix this issue -- stop not happening at all -- more easily than #49528 ("Interactions of restartIfChanged, reloadIfChanged, stopIfChanged").

I have created a PR that implements stop + restart (instead of stop + start) of the stopped units:
#307562
Please help test!


low: proposal pool syntax

const a = pool(async () => {});

a.single()
a.debounce(300)
a.throttle()

// or

a.pipe.single()
a.pipe.debounce(300)

a.pipe( ... ) // what is It?

and for values and signals

const a = value(0);

a.pool.debounce(100) // what is it?

Added to "the stream of conciseness 0.7+ roadmap"


Pagination is not constrained to the max number of pages (resulting in no results showing)

Bit of an awkward title, but in a nutshell the osv list seems to assume the page param value is correct even if there's less pages for a particular database leading to no vulnerabilities being shown even though the database counters say there should be, and it's not obvious why:

https://osv.dev/list?page=2&ecosystem=DWF&q=
I can get into this state by viewing the vulnerabilities for a database that has pages, then switching to one that has less pages.

Thanks! This is indeed something we need to fix :)

I don't know if it's related, but I noticed two days ago that the number of records in the website is higher than the number you get from an API call for every ecosystem. The size of the gap can be 2 records or 2000 records...
Example:
The request I sent : https://osv-vulnerabilities.storage.googleapis.com/Linux/all.zip
As a result I got only 5321 records in this zip file, while in the site the number of linux records is 7657.
Maybe I'm missing something here or understanding something wrong?

I don't know if it's related, but I noticed two days ago that the number of records in the website is higher than the number you get from an API call for every ecosystem. The size of the gap can be 2 records or 2000 records... Example: The request I sent : https://osv-vulnerabilities.storage.googleapis.com/Linux/all.zip As a result I got only 5321 records in this zip file, while in the site the number of linux records is 7657. Maybe I'm missing something here or understanding something wrong?

Our counts are currently a rough overestimate -- it does not exclude any withdrawn/invalid entries. The dump would have the more accurate numbers. I've opened https://github.com/google/osv/issues/471 for this.

Having the ?page parameter seem to cause another related issue, loading a page with the page parameter, or refreshing a page after pressing "load more" a few times, will only display that page, with no option of going back to previous pages, and no indication that you are missing some results.
I propose either switching to a more standard pagination UI (preferred). or removing the page parameter from the URL.


WIP/ test  Support Retain reclaim policy in external storage and update all k8s.io dependencies to 1.8.1

this is a continuation of https://github.com/kubernetes-incubator/external-storage/pull/392
snapshots needs some  code changes because of changes to kuberentes' internal API.
other than that, we are fine.
@xingzhou we can contniue work in https://github.com/kubernetes-incubator/external-storage/pull/392 if I submit a PR against your branch, or we can fix it here. up to you!

$ make snapshot
cd snapshot; \
make
make[1]: Entering directory '/home/mawong/src/github.com/kubernetes-incubator/external-storage/snapshot'
go build -o _output/bin/snapshot-controller cmd/snapshot-controller/snapshot-controller.go
# github.com/kubernetes-incubator/external-storage/snapshot/pkg/apis/crd/v1
pkg/apis/crd/v1/register.go:52: cannot use VolumeSnapshot literal (type *VolumeSnapshot) as type runtime.Object in argument to scheme.AddKnownTypes:
	*VolumeSnapshot does not implement runtime.Object (missing DeepCopyObject method)
pkg/apis/crd/v1/register.go:53: cannot use VolumeSnapshotList literal (type *VolumeSnapshotList) as type runtime.Object in argument to scheme.AddKnownTypes:
	*VolumeSnapshotList does not implement runtime.Object (missing DeepCopyObject method)
pkg/apis/crd/v1/register.go:54: cannot use VolumeSnapshotData literal (type *VolumeSnapshotData) as type runtime.Object in argument to scheme.AddKnownTypes:
	*VolumeSnapshotData does not implement runtime.Object (missing DeepCopyObject method)
pkg/apis/crd/v1/register.go:55: cannot use VolumeSnapshotDataList literal (type *VolumeSnapshotDataList) as type runtime.Object in argument to scheme.AddKnownTypes:
	*VolumeSnapshotDataList does not implement runtime.Object (missing DeepCopyObject method)
pkg/apis/crd/v1/types.go:292: cannot use &vd.Metadata (type *"github.com/kubernetes-incubator/external-storage/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListMeta) as type "github.com/kubernetes-incubator/external-storage/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".List in return argument
pkg/apis/crd/v1/types.go:312: cannot use &vd.Metadata (type *"github.com/kubernetes-incubator/external-storage/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListMeta) as type "github.com/kubernetes-incubator/external-storage/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".List in return argument

I will work on fixing this issue. everything else seems okay.

@wongma7, does this PR contain all the changes I made in #392?

Yes, all I’ve done so far is change from commit hash to 1.8.1 tag and test
it pass ci. But next have to fix snapshots, which depends on the changes in
your pr
On Wed, Oct 25, 2017 at 12:15 AM Tom Xing<EMAIL_ADDRESS>wrote:

@wongma7 https://github.com/wongma7, does this PR contain all the
changes I made in #392
https://github.com/kubernetes-incubator/external-storage/pull/392?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/external-storage/pull/419#issuecomment-339209351,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMgP-DRmm9L88OrR7F0XqEnLR6JVXnPcks5svrXegaJpZM4QFFx9
.

yeap, is there any reason that we need to share dependency with snapshot? to me, snapshot is some kind of independent code, which will be compiled on its own

@wongma7 it seems going with 1.8.1 tags is not enough for some newer api objects as in the latest CI failure for CephFS.

@sbezverk ty for heads up about cephfs, added TODO for https://github.com/kubernetes-incubator/external-storage/issues/309 , cephFS namespace issue @rootfs
Added deepcopy funcs for snapshot API, CRD requires we implement them. @tsmetana PTAL

I will squash the first 3 commits @xingzhou
Then we can get this merged finally. Sorry for the month-long delay everybody.

Forgot to say, for the snapshot changes I ran
$ ./generate-groups.sh deepcopy github.com/kubernetes-incubator/external-storage/snapshot/pkg/apis github.com/kubernetes-incubator/external-storage/snapshot/pkg/apis "crd:v1"

from https://github.com/kubernetes/code-generator to get zz_generated.deepcopy.go. And manually fixed the broken ones and put into deepcopy.go. (The structs that have a member of external type i.e. from core_v1)

@wongma7 if it is possible, could you please add recent merged api change to support RBDPersistentVolumeSource which was added last week? In this case RBD provisioner can be refactored to use it.

nobody wants to review, merging

@wongma7 Tested for RBD, the change looks good, now SecretReference is available. I still see the issue for new RBDPersistentVolumeSource though, I guess it will be addressed in the next PR.
pkg/provision/provision.go:114: undefined: "github.com/kubernetes-incubator/external-storage/vendor/k8s.io/api/core/v1".RBDPersistentVolumeSource
Makefile:27: recipe for target 'build' failed


Switch to master/release deploy scheme?

From @mzikherman on July 28, 2016 17:47
Something like:

The default branch for this repo is master. All development and PR's are done against/into that branch. Upon a successful merge, the master branch will get automatically deployed to Heroku's staging environment. When you are ready to deploy to production, open a PR from master to release. CI will automatically deploy to Heroku's production environment on a successful merge to the release branch.

This way, rather than use a janky Semaphore UI to deploy to prod, we can use the familiar GitHub PR UI to kick off a deployment. We can also easily use the GitHub UI to see past deployment PR's and see what's been deployed when, and each deploy remains reviewable as a PR.
Copied from original issue: artsy/force-private#5424

I have no real opinion for or against. If this is trivial to setup and no other devs on this project object, then go for it 👍

Cool, I'm still very much in favor of this over the current flow, so will give it a shot.

I'm not sure how to QA this, Kana deployed this way last week and I don't think there was any problem, or if there was I didn't hear about it because I was offline. I think after a few successful deploys we can consider this closed.

Yea, this looks done 👍
Should we do it for microgravity too?


FE-homework-16

Drift :-
https://video.drift.com/v/abqMl8IjpAHYC57seBQUbAmTmDvBptOOrRLLTdFKc/
Screenshots for component testing using cypress :-

Screenshots for unit testing using Vitest :-

Metric
Value

alert_status
b:white_check_mark: OK/b

bugs
1

code_smells
3

reliability_rating
C (at least 1 Major Bug)br/Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)br/Note: A being best, E being worst

vulnerabilities
0

Metric
Value

alert_status
b:white_check_mark: OK/b

bugs
1

code_smells
3

reliability_rating
C (at least 1 Major Bug)br/Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)br/Note: A being best, E being worst

vulnerabilities
0


fix common_bootstrap_payload in BaseSupersetView

CATEGORY
Choose one

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY

Seems one of the pylint (etc) refactors messed up some of the formatting in BaseSupersetView. The common bootstrap payload method (and the views that use) are all broken in master
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

nvm, looks like we're using it as function now


Let toolbar buttons and menues be restrictable to route

const applyConfig = (config) => {
  config.toolbar.activities.view.top.push(ButtonA)
  config.toolbar.activities.view.bottom.push({
    match: '/documentation',
    component: ButtonB,
  })
  return config;
};

@ksuess I think it might be better to always require the { match, component } data structure, so that we don't have to special-case it. It would mean that the registration of buttons would always have to include that information. Another option would be to have another, router-aware helper component that can be used to wrap the button. Something like:
const applyConfig = (config) => {
  config.toolbar.activities.view.top.push(ButtonA)
  config.toolbar.activities.view.bottom.push(withPathRestriction(ButtonB, (pathname) => pathname.includes('/documentation))
  return config;
};

And the withPathRestriction would be:
import { matchPath } from 'react-router';
const withPathRestriction = (WrappedComponent, matcher) {
    const wrapped = (props) => {
        const location = useLocation();
        const { pathname } = location;
        return matcher(pathname) ? (<WrappedComponent {...props} >) : (<></>);
    }
    return wrapped;
}

In any case, I've noticed that the matching function that you've used is a simple equality check, we should use matchPath, similar to https://github.com/plone/volto/blob/master/src/components/theme/AppExtras/AppExtras.jsx#L11
@plone/volto-team what do you think?


EthStorage Devs Meeting #2 Agenda

Meeting Info

Oct 11, 2022, 14:00 UTC+8
Duration: 60 minutes
Meeting link
Meeting Notes
Anyone can comment on the agenda where they would like this to be discussed.

Agenda

Protocol/App/Research progress
TBA

I may be absent from the meeting since I am traveling to Bogota.  Frank Liu and Alan may share their latest PR updates.

Research

Lens Protocol by Mingpeng (adding matters and rss3 related content)
AR by Lun (2/3)

Application

Web3URL website (design progress 2/3)

Research

Blockchain[sui/aptos] performance improvements (on-going 80%)

aptos [Lack of some state sync details]
sui[Lack of some conflict resolution mechanisms]
Backend

external call: split into small pr [1/3]

Sharded storage with P2P support
-- Design doc (Keep updating)
-- Implementation: split to multi-small pr
-- Integration test (starting)

Recording: https://drive.google.com/file/d/1gS8jo-wVxluXBiIWf88ZzqzaxnXOEVhu/view?usp=sharing

The meeting note has been updated, and the issue can be closed.


Batch inference

What kind of changes do we need to make to enable batch inference? Currently the boxes have the shape
(num_detections, 4) instead of (None, num_detections, 4)

Did you figure this out?

Di you figure this out?

@qingzhouzhen - I had a look at your implementation of batch inferencing, and I had done a similar thing myself. However, the problem is, all the detections are returned as a single list.
Some images may have multiple detections, while some may have none. Is there a way to arrange the detections image wise, so that it is easy to figure out which box/detection is for which image?


The documentation is far from idiot-proof

Documentation Is:

[x ] Missing or needed
[x ] Confusing
[x ] Not Sure?

Please Explain in Detail...
Let's say you are completely new to ChartJS and want to build a line chart. After reading the installation and getting started pages, you might sensibly visit the Charts > Line page:
https://www.chartjs.org/docs/latest/charts/line.html
The example chart on the page looks great. How would you implement that in your own html page? Well look there is example code right underneath it.
var myLineChart = new Chart(ctx, {
    type: 'line',
    data: data,
    options: options
});

Right... ok that's fine but what am I supposed to put in the data or options fields? The next section is labeled "Dataset Properties". This seems relevant but I'm not really sure how it relates to the data field above. Right at the bottom of the page there is a heading called "Data Structure" which starts:

The data property of a dataset for a line chart can be passed in two formats

Ok so you might think this is referring to the missing data field above and then you might try to do this:
var myLineChart = new Chart(ctx, {
    type: 'line',
    data: [20, 10],
    options: options
});

But of course, even if you were to give a valid options value this would not work...
Your Proposal for Changes
Please define the data types for the data and options fields. What properties do they accept? Where can I find this information? Please reference these datatypes in all the locations where they are needed such as the first example code of the Line documentation. Don't assume a newcomer has any knowledge besides what has been explained in the Getting Started pages.
If this is not possible to do then please supply more full examples. Why not give the full source for the line chart that is shown at the beginning of the line chart documentation?
This is just one example of a page that could be improved, however I find myself constantly lost while looking at the chartjs documentation. For example, looking at the "Performance" page I noticed a property called ticks.sampleSize. So thinking lowering this value might improve performance on my chart but I simply have no idea on which object this property exists. It seems likely to be on the ticks object of the common configuration of some axes object - however this is not explicitly stated neither is it stated how to specify the common axes properties.

Have you seen the samples https://www.chartjs.org/samples/latest/ with full examples?

So thinking lowering this value might improve performance on my chart but I simply have no idea on which object this property exists.

This alone is always the most confusing part for me and I believe it simply arises from the way the information is presented.
Most of my colleagues frequently ask whether something goes into [dataset]. or options. and which additional keys are needed until the actual parameter can be set.
Looking at https://www.chartjs.org/docs/latest/configuration/layout.html for example, the information is clearly there:

The layout configuration is passed into the options.layout namespace.

But most of the time, the eyes immediately skip to the table (one might even scroll down, completely hiding the first sentences) and only seeing: padding.
What happens next is usually confusion, followed by searching stack overflow for a code example 😄
The issue I see here is that the current documentation implicitly assumes that people read/study it top to button and that most things need to be told only once.
What I however usually see is jumping straight into any page from a quick google search, quickly scanning for promising looking snippets/paragraphs that might answer the question somebody has and only focusing/reading these particular parts.
Having all that said, my suggesting would be to write options.layout.padding instead of padding (i.e. always starting with the root object) within parameter lists, repeating every time where exactly a parameter has to be set.

 I don't see a link to the migration guide in https://raw.githubusercontent.com/chartjs/Chart.js/master/README.md

Oups sorry it's in Breaking Changes of https://github.com/chartjs/Chart.js/releases/tag/v3.0.0-alpha

@JLuc sorry about that. I've updated the link to point to the correct spot

Having all that said, my suggesting would be to write options.layout.padding instead of padding (i.e. always starting with the root object) within parameter lists, repeating every time where exactly a parameter has to be set smiley

Thanks for this suggestion - it would be a big improvement over the current documentation. Even better if each of the breadcrumbs would link to the table that defines the other properties that object might need.
I should add at this stage, taking another look at the documentation, I cannot find the page that defines the available properties of the options field. You would think it would be on this page but this doesn't mention the scales property which I have seen in some examples: https://www.chartjs.org/docs/latest/general/options.html
Where is the documentation that describes options?

I played around with the fully qualified names. The first image shows with options at the beginning vs the 2nd where it is omitted. I'm not sure what is better. One potential downside of the 1st is confusion about the global options. i.e. Chart.defaults.xxx or Chart.defaults[chartType].xxx.
Fully Qualified

Partially Qualified

Re the options page, I think that might be improved by removing that page from underneath "General" and renaming "Configuration" to "Options". That way everything underneath "Options" would be in the Chart options. We could move the documentation about global options, scriptable options, etc to the first page of what is now "Configuration".

We would also need to think about how to display options that can go in multiple places. E.g. options.lineTension vs options.elements.line.tension vs dataset[0].tension (I'm just mentioning from memory and might have those paths slightly wrong). We might have a hard time doing this without tackling some of the other options cleanup work from the 3.0 tracking ticket.
If you want a list of all options under options then your best bet is probably to look at the TypeScript type definitions:
https://github.com/DefinitelyTyped/DefinitelyTyped/blob/96ffa5c08b8f6c880098d139a900164a47a0f077/types/chart.js/index.d.ts#L276
This might be one good reason for us to adopt TypeScript more broadly. It could help a lot with these types of questions

tension is an odd one as I think options.lineTension was only kept for backwards compatibility (if it still exists).
We actually document tension twice:

option config: https://www.chartjs.org/docs/master/configuration/elements#line-configuration
dataset config: https://www.chartjs.org/docs/master/charts/line/#dataset-properties

I think that's the best way to handle options like that. It might be worth linking the dataset option to the corresponding config option page. That would reduce any duplicate info

Going to close this; it's been a while since it was filed and we have continued to update the documentation since that point. We also now ship TS type definitions that common editors can pick up for autocomplete which will help the understanding of where options go.


Add an option to specify the AMC slot instead of the IPMB address

This pull requests improves the usability a little bit by allowing the user to specify the AMC slot (in the format of "AMCn") instead of IPMB address, and the tool automatically performs the translation in accordance with the Table 3-5 from MicroTCA.0 standard.

Thanks! Should be available now via pip install --upgrade mmcterm.


Clone submodules

Add --recurse-submodules argument to clone command

Hi @clabnet, thanks for the contribution!
There seems to be a lot of irrelevant code here, so I am going to merge this to a new branch, only pick the relevant changes, and merge back to develop. I just don't want your contribution credit to disappear.


Restart update method

Introduce the “primaryUpdateMethod” option and add “restart” to the existing switchover method for rolling updates.
Updating a cluster now requires a rollout update, where standby servers are first upgraded. Currently, when it comes to the primary, we are now issuing a switchover, and the only available knob is the “primaryUpdateStrategy” accepting “unsupervised” (immediate switchover - default) and “supervised” (for manual switchover through the cnp plugin for kubectl) for possible values.
While this implementation is OK for most cases, there are some cases where this approach can be problematic, in particular in high workload instances where standby servers are lagging with WAL replaying.In these cases, the switchover might find even the most updated standby to be lagging by an arbitrary number of seconds from the primary. Paradoxically, this method increases the risk of data loss in the cluster due to the premature promotion of a standby, as well as the downtime due to the time required to promote the standby. In such cases, a restart of the primary might be more convenient than a switchover. The proposal is to extend the current rollout strategy (update standby servers first, if available) and introduce another option called primaryUpdateMethod, accepting two values: “switchover” (current behavior) and “restart” (instead of issuing a switchover, it restarts the pod of the primary). In the extremely remote case the restart should fail, the failover procedure will be triggered.

This feature has been implemented with the following commits:

3df16291fb2672f00f964b90d5f414b900b10e51
6b5907a3bff23482197bb079e08d78586cbc23c4
3ede59aa2875b7ae34db97f134502f77291d577b

This feature has been released with version 1.15.0.


Notification ralph dev

Color please match our theme color
There is no notification for completing the task
After creating a new task, click on the project link in the prompt and the corresponding page cannot be found

Prompt once for every modification of a date?

Why not close notification after a period of time?
There is no notification for completing the task
About width of notification, if its better to let it not beyond left bar when its content is not too long?

How about limit the width of notification.
How about not show the detail of task when complete the task
There only can show one notification anytime. Should we consider displaying multiple notifications when there are consecutive tasks completed. For example, when a new notification appears below, the old one moves upwards

Should we consider making the notification text smaller

How about limit the width of notification.
How about not show the detail of task when complete the task
There only can show one notification anytime. Should we consider displaying multiple notifications when there are consecutive tasks completed. For example, when a new notification appears below, the old one moves upwards

I believe displaying a certain number of characters helps differentiate between notifications, otherwise a bunch of identical notifications seem a bit redundant. Therefore, based on your suggestion, I've limited the number of characters that can be displayed, and the effect is as follows. Similarly, I adjusted the font size, you can check if it's suitable.
As for displaying multiple notifications at once, actually, that's the simplest approach, but currently, it's specifically designed to only show one notification at a time to achieve the same display effect as the todoist page. If everyone prefers that effect, we can consider changing it back in the future.


add this figure https://www.researchgate.net/figure/Typical-NVIDIA-GPU-architecture-The-GPU-is-comprised-of-a-set-of-Streaming_fig1_236666656

https://github.com/dcelisgarza/DPhil_Thesis/blob/a3103d899af5aaff8ceec62333286bd0c791486e/src/Tractions.tex#L818-L818

done ages ago


Issue with group registration

Hi,
I have an issue while I am trying to get an average volume of 6 brains I have imaged.
I start with import and resize to the allen ccf:
python U:\magellanmapper\run.py --proc import_only -v --img "C:\Make_atlas\ANM39.tif"
python U:\magellanmapper\run.py --img "C:\Make_atlas\ANM36.tif" --proc transform --atlas_profile abaccfv3

Then I tried to run the group reg:
python U:\magellanmapper\run.py -v  --register group  --atlas_profile groupwise --img "C:\Make_atlas\ANM36"  "C:\Make_atlas\ANM37"   "C:\Make_atlas\ANM39"

My first issue is the assumption I have labels for one of the brains - line 1061 in register.py.
In my local copy I was trying to skip this and not crop in y.
I can get thru to line 1116:
    transform_filter = elastix_img_filter.Execute()

And I get an error:
root - CRITICAL - Unhandled exception. Additional log saved to: C:\Users\moharb\AppData\Local\Temp\magellanmapper_error_varcqfqz.log
Traceback (most recent call last):
  File "U:\magellanmapper\run.py", line 23, in <module>
    load_env.main()
  File "U:\magellanmapper\magmap\io\load_env.py", line 185, in main
    launch_magmap()
  File "U:\magellanmapper\magmap\io\load_env.py", line 138, in launch_magmap
    cli.main()
  File "U:\magellanmapper\magmap\io\cli.py", line 994, in main
    process_tasks()
  File "U:\magellanmapper\magmap\io\cli.py", line 877, in process_tasks
    register.main()
  File "U:\magellanmapper\magmap\atlas\register.py", line 2047, in main
    register_group(
  File "U:\magellanmapper\magmap\atlas\register.py", line 1129, in register_group
    transform_filter = elastix_img_filter.Execute()
  File "C:\Users\moharb\Miniconda3\envs\mag\lib\site-packages\SimpleITK\SimpleITK.py", line 7081, in Execute
    return _SimpleITK.ElastixImageFilter_Execute(self)
RuntimeError: Exception thrown in SimpleITK ElastixImageFilter_Execute: Y:\SimpleElastix\Code\Elastix\src\sitkElastixImageFilterImpl.cxx:191:
sitk::ERROR: ElastixImageFilter does not support the combination of 4-dimensional 32-bit float fixed image and a 4-dimensional 32-bit float moving image.

I tried this minimal example:
vectorOfImages=sitk.VectorOfImage()
origin = None
spacing = None
for filename in l2:
    reader = sitk.ImageFileReader()
    reader.SetImageIO("TIFFImageIO")
    reader.SetFileName(filename)
    img = reader.Execute()
    if origin is None:
        origin = img.GetOrigin()
        spacing = img.GetSpacing()
    else:
        img.SetOrigin(origin)
        img.SetSpacing(spacing)
    vectorOfImages.push_back(img[:300,:300,:300])

label4d=sitk.JoinSeries(vectorOfImages)
elastixImageFilter=sitk.ElastixImageFilter()
elastixImageFilter.SetFixedImage(label4d)
elastixImageFilter.SetMovingImage(label4d)
elastixImageFilter.SetParameterMap(sitk.GetDefaultParameterMap('groupwise'))
resultImage=elastixImageFilter.Execute()

Still get the same error.
Any idea what I have missing?
I found this issue, maybe could be related.
Thanks!
Boaz

Thanks for asking about this, @boazmohar! I get this error as well when I test the groupwise reg.
Looks like it's related to this issue (https://github.com/SuperElastix/SimpleElastix/issues/468), where more recent versions of SimpleITK/SimpleElastix lost support for groupwise reg.
A workaround I just tested is to install an older version of SimpleElastix. I have an older build that works, though it's a bit complicated to install to avoid version conflicts and because it's only for Python 3.6. I got it to work by changing this section in setup.py:
https://github.com/sanderslab/magellanmapper/blob/430974b4f1e8695c490be143c59f3443fdd7a9c7/setup.py#L76-L89
to install these simpleitk and bg-atlasapi (this package just bumped its Python req to >= 3.8) versions:
"simpleitk==1.1.0.dev362+gb3783 ; python_version < '3.8'",

"bg-atlasapi==1.0.2,"

and then rerunning setup:
pip install -e . --extra-index-url https://pypi.fury.io/dd8/

My first issue is the assumption I have labels for one of the brains - line 1061 in register.py.

Yeah, the groupwise reg function was originally for a more specialized case, and I'll need to remove this assumption--thanks for pointing it out! I also found a few more issues such as loading settings that I'll need to fix.

Thanks for addressing this.  I saw that ANTS has a more well documented group registration function: antsMultivariateTemplateConstruction2.sh
Any chance to incorporate that into this package?

Thanks for sharing, will definitely consider this. I have not used ANTs extensively, and my understanding is that it can be very accurate but more computationally expensive (see https://ieeexplore.ieee.org/document/8512403). Have you found it to work well for your use cases?
Also, happy to work together with you to implement it here, if you are interested.

Yes, I do find it very accurate and slow. It can take many hours to make a template, but I on;y need to do it once (well correctly once, but incorrectly many time :).
 I think there is a wrapperbut I haven't tried it. Could you take a look if that could be a dependency for your project? If so I can have a look at how to integrate it.
Thanks!!
Boaz

Thanks for pointing out that wrapper! It looks like a great way to implement ANTs here, and happy to add it as a dependency. I've been using SimpleITK here mainly just for I/O and registration, otherwise converting to NumPy arrays, and it looks like ANTsPy could at least function similarly. The extra registration accuracy would also be a definite plus! Please do feel free to integrate it if you'd like, and let me know anytime if questions come up, happy to help.
By the way, the folks at SimpleITK are fixing the groupwise reg, so hopefully that will be working again soon: https://github.com/SimpleITK/SimpleITK/issues/1891 .

@boazmohar, I've added support for ITK-Elastix as an alternate to SimpleITK, where the groupwise registration works at least in my basic tests. It should also be easier to integrate other registration toolkits like ANTs if interested in the future.


DemoMultipartForms Doesn't Work

I'm not sure if I'm doing something wrong but I don't think the multipart form demo is working at all.
if I uncomment the multipart demo in App.RequestHandler to enable the demo and visit localhost:8080 I get the form to upload files. No matter what I try to upload the request eventually times out and as far as I can see the DemoMultipartForms.RequestProcess method isn't being called.
Can you check if this demo works for you? I'm on macOS.

Also, it seems that even as I type the address for local host into Chrome or Safari, the RequestProcess method gets called before I can even see the returned HTML in the browser. It's almost like the browser is trying to be smart and grab the resources before I finish typing. Not sure if this has something to do with it. Certainly I am unable to catch anything with a breakpoint in DemoMultipartForms.RequestProcess after I click the submit button in the browser.

It seems the body remains empty which is rather odd. I maybe looking into this later if i have some time.
In the meantime if you find anything, please report it.

I have found that the body was separated by CRLF + CRLF, by changing that to a local buffer we now get full content (all chunks) but it still doesn't close (respond) properly and doesn't save the file. A new branche will be created to fix this issue.

The issue had been in Express.Request.

DataAvailable: uses Lookahead.Bytes to determine if the Content has been fully received
Fix in Method BodyGet, so that Multipart Form Content is no longer stripped
DataGet and BodyGet in now called in Method "Process"

I have fixed this along the way when creating Pull Request #10 (Express 6.1.0 | EventLog | Demo GUI).

@gkjpettet I think you can close this issue as this is fixed in Express 6.1.0.
And maybe you'd like to create a Tag and/or Release in this Repo.

Closed as fixed in 6.1.0: https://github.com/sworteu/Express/releases/tag/V6.1.0


Add db2 jdbc module

Created during the Softwerkskammer Ruhr Hackergarten together with Felix Winkler.
I have some concerns regarding the license issues, similar to Oracle. On the one hand, we to provide an environment variable to accept the license when running the official IBM DB2 image.
The other problem is the JDBC driver. IBM doesn't provide the JAR on Maven Central. A search on Maven Central resulted in finding this Maven Repository: https://artifacts.alfresco.com/nexus/content/repositories/public/
Any good ideas how to proceed from here?
The general implementation works, as well as the JDBC-URL support, however the JDBC-URL support has some vendor specific SQL, so the test fails.
closes #432

Thanks for this! Our database coverage continues to grow!
I've not looked into the licence situation yet, other than your comment above, but I think we mainly just need to divide and conquer the licence issues:

Our users' licence position; basically we shouldn't let them accidentally or deliberately circumvent the IBM licence. So:

If we automatically pull an image for users, it ought to be an official one, and it looks like this is the case.
If that image requires an env var signal to indicate licence acceptance, we should give Testcontainers users a way to confirm that they have accepted the licence on behalf of their organisation, and want this signal to be applied. We have exactly the same situation for the SQL Server module.
We shouldn't have a transitive Java dependency on unofficial DB2 drivers (i.e. they should be a 'provided' scope dependency, and let users bring-their-own drivers, which they probably will have anyway if they're using DB2).

Secondly, our own licence position with respect to running Testcontainers' own automatic 'self test' suite during our CI pipeline build:

Let's check what it means to accept the image licence for our own testing purposes. I'd be happy to accept a reasonable licence on behalf of the Testcontainers project.
Also let's check what JDBC drivers we can use. I don't want to be in a situation where we're at risk of 'distributing' drivers wrongly, so they shouldn't be binaries within our repo. Let's think about the third-party repository and see whether they're suitable for us.

If that image requires an env var signal to indicate licence acceptance, we should give Testcontainers users a way to confirm that they have accepted the licence on behalf of their organisation, and want this signal to be applied. We have exactly the same situation for the SQL Server module.

I kind of remembered, we have a similar situation for the other modules. I will implement the license agreement check in a similar fashion.
Here is the license information regarding the official IBM image:

The Dockerfile and associated scripts are licensed under the Apache License 2.0. DB2 Express-C is licensed under the IBM International License Agreement for Non-Warranted Programs. This license for DB2 Express-C can be found online. Note that this license does not permit further distribution.

http://www-03.ibm.com/software/sla/sladb.nsf/displaylis/5DF1EE126832D3F185257DAB0064BEFA?OpenDocument

We shouldn't have a transitive Java dependency on unofficial DB2 drivers (i.e. they should be a 'provided' scope dependency, and let users bring-their-own drivers, which they probably will have anyway if they're using DB2).

We are only using the (unofficial? at least unofficially hosted) JDBC driver for our own integration tests (with testCompile dependency ATM). And I 100% agree, I don't want any binaries in our repo either.

Yo. It´s me again, comming up with another idea:
In my test case scenario (for spring data jdbc) it may be imaginable if the user places the vendor´s jdbc driver in a certain directory (ressources for example) and the testcontainers images uses this jar (if this is possible).
Would that be an option to take in count?
So in short:

user gets the "official" jdbc driver from IBM (wich requires an IBM account) (other vendors may be the same procedure)
user places this jar in a certain fixed directory
if the jar is there, the testcontainer image (pulled from docker) starts
if the jar is not there, cancel the process

Maybe it would also reasonable to just start the image and the user has to deal with the jdbc driver?
I don´t know if this is possible. I either don´t know if this make sense for you.
But maybe it´s worth a try?
Let me know if i can help somewhere/somehow.
Kind regards
Thomas

Let's leave it open.
@rnorth
WDYT about ignoring the tests as default (and therefore on CI), thereby solving the JDBC driver issue. We can add something in the docs to inform users, that this module isn't automatically tested.
I still think there is some value in having this module though, I've talked with different users that very much like to use Testcontainers with DB2 (and I normally recommend them to implement this PR as their own custom container).

I use the DB2 official image for testing at my firm constantly. I’d love to see this go through.

If there is anything I can do to help please point me in the right direction.

@rnorth Do you think we can integrate after I've resolved the parts I've commented?
@sporkisfaster Thanks for the offer, I think it's basically finished from an implementation standpoint.

Should this point at Developer-C instead of Express or does it matter? As far as I know the Express image is deprecated

Any updates on this? Would be super helpful as I’m running into issues with genericcontainer for db2

Hi @kiview ..thank you for pointing me here in your comment on my issue (from another repo).    I'm interested to understand the status.    Is it the case that the issues raised in early comments have been addressed, and now it's just a matter of someone addressing the merge conflicts and completing the PR?
Or, if there are other issues still remaining to work out here, could you please give an update of where we stand when you have a chance?
Thank you.

IBM has licensed the IP of the db2 jdbc driver (jcc) to Rocket, and they have made it available in maven central https://search.maven.org/artifact/com.ibm.db2/jcc/<IP_ADDRESS>/jar.  Like the db server, it is still IPLA license though.

@frowe probably different thread, what about the source jar?

@frowe probably different thread, what about the source jar?
Since it's IPLA license, there is no source jar published.

@kiview / @aguibert: Do you need some more support in order to get this fixed and merged? Is anything else left except for fixing the Codacy/PR Quality Review, review comments and merge conflicts?

The hard parts (legal and where to get the artifact on MC) seem to be figured out. All that's left now is addressing review comments and such AFAIK.

Should this point at Developer-C instead of Express or does it matter? As far as I know the Express image is deprecated

Unfortunately you need to log-in and accept the license agreement in order to pull the new docker image for Developer-C ( https://hub.docker.com/_/db2-developer-c-edition ). Not sure if that's the default we want to provide. In addition that container image exposes additional configuration options.
Do you have any experience with that image?
On the other hand I do agree that using three year old image does not sound like a long-term solution. Upgrading the Db2Container to the new image in a later step could represent a breaking change for any clients.

I've rebased the pull request, cleaned up and incorporated the feedback from the review comments to the best of my understanding and fixed the test. I could create my own pull request with the changes if someone is interested however I do not want to step on anyone's (@kiview in particular) toes.
What do you say, @aguibert @bsideup ?

Since I'm an IBM employee I'm going to fish around a bit internally for a few days to try and get in touch with some DB2 folks and see what Docker image we should be using.
Right now my preference would be going with the more current image (Developer-C image), even if it does mean accepting a license.

FWIW I've raised this ticket around the current gap in our docs for new module contributors. I'd like to have a bit of a think about the third point specifically before merging this, but otherwise I'm fine with this new module at an overall level: https://github.com/testcontainers/testcontainers-java/issues/1503

ok, reporting back...
I anticipate a new DB2-C docker image that will not require any docker hub logins is going to be released in the near enough future that it would make sense to hold off on this PR in favor of the future DB2-C image.

@aguibert thanks a lot for pushing it forward!
It is so great to see a progress here, especially in such a sensitive topic as commercial DBs from giants like IBM :)
If this is not the magic of OSS communities then what is? 😀

@bsideup @kiview ok, the new DB2 images are here and they no longer require any sort of docker login!
https://hub.docker.com/r/ibmcom/db2
The doc is also much more clear on how to configure them. The startup time isn't great (4m on my machine), but I spoke with the DB2 team and they are have plans to reduce the startup time in the future.
I'm kicking the tires with this new image w/ Testcontainers now, and will contribute back my findings. Should I contribute my findings to this PR, or open a fresh one?

Sounds promising, @aguibert.
Could you perhaps compare the start up times between the two containers? Is the long start up time a regression? Did you start it with the default configuration which creates a database on startup? That could perhaps explain (some) of the differences.
In our custom Db2Container I've set up DB2 to run in some sort of a (unsupported) in-memory setup. In our use case we're destroying the container anyway and therefore don't care about the durability of any data.
Extra configuration:
db2set DB2_DIRECT_IO=OFF
db2set DB2_LOGGER_NON_BUFFERED_IO=OFF
Setting up the tmpfs:
withTmpFs(Collections.singletonMap("/home/db2inst1/tmpfs", "rw"));
Creating the database:
db2inst1 -c 'db2 create db " + getDatabaseName() + " on /home/db2inst1/tmpfs'

@frederikb thanks for the suggestions. I didn't do a detailed comparison, but here is a rough breakdown of the start time activities on my machine:

Configuring DB2 instances (20-30s)
Configuring DB2 instances (second pass?) not sure why we go through this twice (20-30s)
Starting text search service (15s)
Creating DB (50s)
CIE00001 Operation completed successfully.  (16s)

I'll inquire on the "configuring instances" second pass, as well as the text search service, which perhaps can be disabled with an extra switch. However, this biggest time sink was creating the DB itself (50s) and I doubt that can be sped up further, but I will check.

I put an update to the container here - https://hub.docker.com/r/ibmcom/db2 to address some of the concerns with the start up time. There is more to be done, so subsequently we will continue making updates.
With this latest update, you should set the following env variables to increase speed up:
ARCHIVE_LOGS=false - It will not set up transaction log archiving, which forces us to create a backup and restart the database
AUTOCONFIG=false - It will not invoke auto-tuning, which requires a database restart
and TEXT_SEARCH=false is by default, so you will not get the secondary instance update, which by Andrew's approximations takes up another 20-30sec.
Create database will take longest in this round, which is something that will be addressed later on. So stay tuned for updates.

Thanks, @irinadel!
On my machine and setup this reduced start up time from 159 seconds down to 103 seconds. That definitely helps! I'm looking forward to any other improvements you can make.
I'm still seeing something in the setup_db2_instance.sh script which looks like a bug:
if [[ $1 = "db2start" ]]; then
    su - ${instance_name?} -c "db2start"
fi

I'm pretty sure that ${instance_name?} is meant to be ${DB2INSTANCE?}. Otherwise you get the following error on startup:
/var/db2_setup/lib/setup_db2_instance.sh: line 193: instance_name: parameter null or not set
Workaround: Set the environment variable instance_name to db2inst1.

@frederikb that piece of code path you showed never executes, so never mind it, its legacy, ill remove it on the next iteration
Don't use db2start as a parameter to the docker run.
Example command to start up the container in faster mode is:
docker run -itd -e AUTOCONFIG=false -e ARCHIVE_LOGS=false -e DBNAME=test -e LICENSE=accept -e DB2INST1_PASSWORD=password  -v /database:/database -p 50000 --privileged=true ibmcom/db2

@irinadel you're absolutely right. That was a leftover from our previous integration of the db2express-c image.

@kiview @bsideup I opened a new PR (#1611) for the a DB2 module with the new DB2 image. Can one of you please review it?
Sorry for the separate PR, but when I tried to do a PR from aguibert:db2 --> testcontainers:db2 it dragged in a bunch of other commits since I had rebased with testcontainers:master and the testcontainers:db2 branch was pretty old.

@aguibert
Thanks, that's perfect. I'll close this PR, let's continue in #1611.


⚠ Bofrost.de has degraded performance

In 336664b, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1001 ms

Resolved: Bofrost.de performance has improved in 439b546 after 13 minutes.


Dead link in documentation

At http://music-encoding.org/support/guidelines and the other documentation pages such as http://music-encoding.org/documentation/3.0.0/chapters , the “Elements” link (http://music-encoding.org/documentation/3.0.0/elements) is dead.

What do you mean? When I look at this link: http://music-encoding.org/documentation/3.0.0/elements
I see this page:

That's peculiar. I get this:

It must be the Danish censors...

The link works for me too. Maybe you can try a "shift + reload" ?

OK. I tested the link in various environments. Here are the results:
Desktop PC at home:
Firefox 43.0.4      OK
Chrome 64.0.xxx OK
MSIE 11              OK
Desktop PC at office:
Firefox 52.0.1       404
MSIE 11               404
Chrome 64.0.xxx  404
Laptop on WiFi at home or Work (location makes no difference):
Firefox 56.0.1      404
MSIE 11              OK
Chrome 64.0.xxx OK
It doesn't seem to be a cache thing since I haven't used most of these combinations before.  Only the 'Elements' link seems to be affected. But if I am the only one experiencing this problem, it's probably not worth investigating further. But it seems to have started recently when the pages were slightly redesigned.

It looks on the screenshot that the page requested is http://music-encoding.org/documentation/3.0.0/elements/ (with a slash at the end) instead of http://music-encoding.org/documentation/3.0.0/elements. I wonder how this can be different.

Yes, I was just noticing that.   It seems to be a problem with the slash at the end of the address.
This works for me:
http://music-encoding.org/documentation/3.0.0/elements
This does not work for me:
http://music-encoding.org/documentation/3.0.0/elements/
The actual URL is:
http://music-encoding.org/documentation/3.0.0/elements.html
Github is making an alias from elements.html to elements (or it is setup this way somehow in the configuration file for the website).
To fix the problem, the contents should be moved to:
http://music-encoding.org/documentation/3.0.0/elements/index.html
This should allow these three URLs to work:
http://music-encoding.org/documentation/3.0.0/elements
http://music-encoding.org/documentation/3.0.0/elements/
http://music-encoding.org/documentation/3.0.0/elements/index.html
To fix the problem, a permalink in the Jekyll header should be added placing the page in /documentation/3.0.0/elements/index.html in this Markdown file:
https://raw.githubusercontent.com/music-encoding/music-encoding.github.io/master/documentation/3.0.0/elements.md
See: https://jekyllrb.com/docs/permalinks
Or the contents of elements.md should be moved from /documentation/3.0.0/elements.md to documentation/3.0.0/elements/index.md, which would be the easiest thing to do  since there is no need to add the permalink to the Jekyll header in this case.
Alternatively all links to the elements page should be changed to:
http://music-encoding.org/documentation/3.0.0/elements.html

(And this is probably true for other pages as well, where it might be best to force the contents to /index.html allowing for slashes to be optional rather than having .html be optional.)

I notice that Firefox (at least the newer versions, it appears) adds the slash if I omit it and therefore fails. I did use the URL without the trailing slash: I just followed the link.
Strangely enough http://music-encoding.org/documentation/3.0.0/atts has the same faulty behaviour on my laptop (Firefox adds a slash and fails), while http://music-encoding.org/documentation/3.0.0/models works on both my laptop and office PC (no slash added).

This should not be fixed because the links have .html. See http://music-encoding.org/guidelines/v3/elements.html

One possibility is to use redirections:
https://github.com/jekyll/jekyll/issues/3368
http://www.marran.com/tech/creating-redirects-with-jekyll
https://github.com/jekyll/jekyll-redirect-from

This should not be fixed because the links have .html. See http://music-encoding.org/guidelines/v3/elements.html

Note that this URL also works:
http://music-encoding.org/guidelines/v3/elements
With some sort of automatic redirect to
http://music-encoding.org/guidelines/v3/elements.html
So additionally an alias could be made to:
http://music-encoding.org/guidelines/v3/elements/
Since that typically the mapping in most webservers for a URL like
http://music-encoding.org/guidelines/v3/elements
(hence the original confusion)


any idea how can I extend the FormMessage to localise the error message?

Here is my thought:
Solution 1: pass i18n to the component
const FormMessage = React.forwardRef<
HTMLParagraphElement,
React.HTMLAttributes

(({ className, i18n, children, ...props }, ref) => {
const { error, formMessageId } = useFormField();
const body = error ? String(error?.message) : children;
const localisedBody = i18n.exists(body) ? i18n.t(body) : body;

...
Solution 2: prioritize "children" so we can pass localized children from the parent component.
const FormMessage = React.forwardRef<
HTMLParagraphElement,
React.HTMLAttributes

(({ className, children, ...props }, ref) => {
const { error, formMessageId } = useFormField();
// const body = error ? String(error?.message) : children;
const body = children ? children: String(error?.message);

This issue has been automatically closed because it received no activity for a while. If you think it was closed by accident, please leave a comment. Thank you.


APP-482: Re-factor CC32xx GPIO application

Re-base of #508 on top of master

@kaa-leeroy test this fully please


[Feat/LIVE-9342]: init swap-live-app on exchange button click

📝 Description

Init swap live app on "exchange" button click in ledger live desktop.
Add onUnknownError and onKnownError callbacks for the SwapWeb component which will allow us controls on the native side incase the web app fails for whatever reason.

❓ Context

Impacted projects: LLD
Linked resource(s): []

✅ Checklist

[ ] Test coverage
[ ] Atomic delivery
[ ] No breaking changes

📸 Demo
N/A
🚀 Expectations to reach
Please make sure you follow these Important Steps.
Pull Requests must pass the CI and be internally validated in order to be merged.

@kallen-ledger, I think @cng-ledger is on holidays. We can merge


Conda Installation Step 4 missing code

Install echolocatoR from command line so that it installs within the conda env:
what do you suggest the user does here exactly?

That just means; open R from the command line and install echolocatoR the usual way:
if(!"remotes" %in% row.names(installed.packages())){install.packages("remotes")}

remotes::install_github("RajLabMSSM/echolocatoR")


Update plugin development documentation for v8

Things to change:

[ ] Mention that trial can be async and return the data rather than calling jsPsych.finishTrial()

@bjoluc are there any other major changes here to update in the plugin documentation that I'm missing?

@jodeleeuw If we decide to drop pretty_name, that would be one thing. If we decide to accept undefined parameters without a default value and introduce required: true instead, that would be the other thing. And #3241 of course, if we go that route.


Ability to have parts of an SVG in separate files

This is a big one. It would be really nice for things like the control panel to have separate SVGs that could be combined at compile time to make an SVG for an activity. This is a big feature.

Done. This took, like, an afternoon 😛


Update to AGP 8.7 stable and 8.8 alpha04

AGP 8.7 stable seems to bring only bug fixes: https://developer.android.com/build/releases/gradle-plugin

It seems that the build failure is coming from a network timeout: https://scans.gradle.com/s/p2g2aqp7pt656/tests/task/:functionalTest/details/com.autonomousapps.android.NoOpSpec/can filter unused dependencies (Gradle 8.9 AGP 8.8.0-alpha04)?top-execution=1
I ran the test locally, and it works fine:

I've rebased my branch with main, and the build is successful now.

Thanks!


Fix ErrorProne warnings in :file-collections and :input-tracking

Reviewing cheatsheet
Before merging the PR, comments starting with

❌ ❓must be fixed
🤔 💅 should be fixed
💭 may be fixed
🎉 celebrate happy things

@bot-gradle test QFL without PTS

I've triggered the following builds with parameters: -DenablePredictiveTestSelection=false for you. Click here to see all build failures.

QuickFeedbackLinux build

@bot-gradle test this

I've triggered the following builds for you. Click here to see all build failures.

PullRequestFeedback build


pullsync, puller: add metrics

This PR adds some prometheus metrics to the puller and pullsync packages.
These are very minimal metrics, and they should reflect some basic information as to the operation of these components.
One important note to the error counters is that due to the fact that context is being used to cancel the syncing jobs on depth change/peer disconnect, it in fact means that not all errors that are being returned from certain function calls should be accounted as actual protocol errors (or errors that we as developers would like to be notified about).
This is due to the fact that once a sync job's underlying context is cancelled, the current function call will return an error context.ErrContextCancelled, which is not really an error in this case. This is why only certain branches of the code actually increment the error counters.
part of #329

pullsync.go:311

is the only one i did not add an increment for, since it does not necessarily imply an immediate Db operation right after the call. It might just wait for a chunk for a long time, hence to increment the counter will not be semantically correct.


feat: add anvil_get_interval_mine method

Motivation
Closes https://github.com/foundry-rs/foundry/issues/8813
Solution

@grandizzy Followed your suggestion and created this new PR (old #9090). Please check if there is anything to improve.

@grandizzy Followed your suggestion and created this new PR (old #9090). Please check if there is anything to improve.

Looks good, thank you! Just a small nit, maybe we should be consistent naming at par with anvil_setIntervalMininig e.g. anvil_getIntervalMining or anvil_getMiningInterval

@grandizzy  Yes, it's reasonable. The PR has been updated.

Thanks for your guidance. I'm so excited, this is my first PR for Foundry. Hope to contribute more in the future.


Training with gpu

Hello.
When I train with GPU get below exception,
resizing:  448 448
resizing:  448 448
Traceback (most recent call last):
File "/home/aic_subscription/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/aic_subscription/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/aic_subscription/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: replica_1/model_1/yolo_layer_3/cond/strided_slice/Switch/_4523 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", se
nd_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_13819_replica_1/model_1/yolo_layer_3/cond/strided_slice/Switch", tensor_
type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 280, in
main(args)
File "train.py", line 257, in main
max_queue_size   = 8
File "/home/aic_subscription/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/aic_subscription/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: replica_1/model_1/yolo_layer_3/cond/strided_slice/Switch/_4523 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", se
nd_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_13819_replica_1/model_1/yolo_layer_3/cond/strided_slice/Switch", tensor_
type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Try reducing the batch size or image size.
The model can't be trained as it cannot allocate enough data.


Ripper requests: archived.moe, archiveofsins.com, fetlife

Feature request - ripping support for threads from:
https://archived.moe/soc/thread/25537505/#q25537505 (example)
https://archiveofsins.com/soc/thread/25548343/#25548343 (example)
Seems similar to 4chan so possibly not too involved?
Also the ability to rip pictures from Fetlife? Using Chrome to inspect every picture individually and open in new tabs is obnoxious.

https://github.com/RipMeApp/ripme/issues/38


How to set images by default in value

I need to made a modify page so i fetch image from api and i want to set it by default in value of the ImageUploading componement. I get this error :
Uncaught TypeError: Failed to construct 'Image': Please use the 'new' operator, this DOM object constructor cannot be called as a function.
Is there a way to set images by default for ImageUploading ?

Hi @esdras-assikid,
You could set an initial image list like this (react hook example):
const [images, setImages] = useState([
   { dataURL: your_init_image_url_1 },
   { dataURL: your_init_image_url_1 },
   ...
]);

Thanks!


As a grid operator I want to be able to change settlement details of a metering point

Benefit hypothesis
As a grid company
We want to be able to update settlement details of a metering point
So that we can ensure that the data is always up to date
Description
It must be possible for a grid company to update metering point master data after a metering point has been created. So once a metering point has been created it should be possible immediately after to update metering point master data. The things in scope for this story is the request from the grid company, validation of the message and accept/reject the request. The information to current and future balance supplier is treated in a separate story.
This story enables the grid company to update settlement details of a metering point
Acceptance criteria

[ ] It must be possible for a grid company to send in a request for update metering point master data with business reason update metering point master data
[ ] If the request is rejected an reject message is generated and sent to the grid company.
[ ] If the request is approved an accept message is generated and sent to the grid company.

DoD

[ ] All tasks have been completed
[ ] All acceptance criteria have been completed
[ ] Unit test performed
[ ] Integration test performed
[ ] Tested
[ ] Accepted by business
[ ] Solution is deployed to test environment

Is solved in #299.


[Bug] App Crash: OutOfMemoryError

Yeah this is mostly PEBCAK but I was trying to resize an image by ratio and it refused to scale either side downwards . Now maybe I didn't understand how it's supposed to work but I was trying to make one side smaller stuff the other side would resize in proportion. So it wouldn't let me change numbers. It kept refilling and changing them. As is tradition of software users, I did nothing but pound in zeros until it broke. Instead of asking for help or doing anything that may have actually worked.
Feature request: let me delete all the numbers when changing resolution of one side window for ratio. Instead of the changing it because it has a blank space at least not so fast a little too strict on that one but it was PEBCAK. what can ya do?
Device: Pixel 5 (google - redfin), SDK: 34 (14), App: 2.3.3-alpha03 (90)java.lang.OutOfMemoryErrorat android.graphics.Bitmap.nativeCreate(Native Method)at android.graphics.Bitmap.createBitmap(Bitmap.java:1206)at android.graphics.Bitmap.createBitmap(Bitmap.java:979)atandroid.graphics.Bitmap.createScaledBitmap(Bitmap.java:831)at pc.l.l(SourceFile:325)at pc.l.V(SourceFile:13)at k6.r5.A(SourceFile:5)at la.i.I(SourceFile:74)at pc.s.l(SourceFile:15)at pc.c.l(SourceFile:72)at pc.c.V(SourceFile:13)at k6.r5.A(SourceFile:5)at la.i.I(SourceFile:74)at k6.y4.s(SourceFile:15)at pc.d.l(SourceFile:138)at pc.d.V(SourceFile:13)at k6.r5.A(SourceFile:5)at la.i.I(SourceFile:74)at pc.s.f(SourceFile:16)at ag.j.l(SourceFile:75)at ra.a.n(SourceFile:1)at nb.e0.run(SourceFile:1)at n8.m.run(SourceFile:176)at tb.i.run(SourceFile:1)at tb.a.run(SourceFile:94)Suppressed: sb.h: [r1{Cancelling}@8137ccd, Dispatchers.Main.immediate]

It seems to have been fixed, but what you are saying is that it allows you to enter other values, then not allowing this is a feature of the ratio mode, if you want so, then use flexible resize mode 🐸