[BUG] Orphan snapshot attachment tickets prevent volume from detaching

Describe the bug (🐛 if you encounter this issue)
As described in https://github.com/longhorn/longhorn/discussions/6281, I have a few volumes that are shown as "Healthy", even though they are not attached to any running pod. I believe this is one of the reason for the problems that I am describing as part of https://github.com/longhorn/longhorn/issues/6552 - an ability to properly drain the node.
I would like to understand the reason for the problem as I have about 8 such volumes (out of 42).
I've already tried a few things in the UI:

Force Detach: that will detach and immediately reattach the volume
Deleting all snapshots and backups. Not observable difference.
Deleting all snapshots and force detaching. Again, the volume gets reattached right away.

At this point in time, for the volume I did all of the above, there are 12 volumeattachments, all named backup-controller-backup-*
To Reproduce
No idea.
Expected behavior
Volumes that are not attached to a pod eventually transition into Detached state. More importantly, volumes that are not attached to a pod (and haven't been in month) should not interfere with node draining.
Support bundle for troubleshooting
Please see https://github.com/longhorn/longhorn/issues/6552 though I'd be happy to provide need bundles when required.
Environment

Longhorn version: 1.5.1
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): via ArgoCD based on Helm Chart.
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s 1.24.5

Number of management node in the cluster: 3
Number of worker node in the cluster: 8

Node config

OS type and version: OpenSUSE MicroOS
Kernel version: 6.4.12
CPU per node: 4 Cores
Memory per node: 8GB
Disk type(e.g. SSD/NVMe/HDD): NVMe
Network bandwidth between the nodes: 1GBit

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremental and KVM
Number of Longhorn volumes in the cluster: 42
Impacted Longhorn resources: 8

Additional context

Okay, I just looked into the backup details for the corresponding volumes and there were quite a few entries not showing as completed. Once I deleted all of them, I could detach. So I believe I have a way to clean things up.
However, the question remains why there were non-completed backups. And I believe keeping those entries forever might not be such a good thing either.

Here's some additional information:
$ kubectl get -n longhorn-system lhb| wc -l
    1127

$ kubectl get -n longhorn-system lhb| grep "<no value>" | wc -l
     441

If I read those numbers correctly, about 40 percent of the backups in my cluster are incomplete.

I am tempted to remove all of those incomplete backups manually but will wait to see if you need any more information from my cluster.

Do you know why the backups are incomplete? What are their states?
If possible, can you provide the support bundle? Thanks.

Do you want a new support bundle or can you just look at the one I provided yesterday for #6552?

The support bundle is deleted. Can you provide it again? Thank you.

The support bundle is deleted. Can you provide it again? Thank you.

On it's way

I can see lots of similar messages
2023-09-08T10:51:51.521119956Z time="2023-09-08T10:51:51Z" level=warning msg="Cannot take snapshot backup" backup=backup-fee13c953cf148a7 controller=longhorn-backup error="could not find snapshot 'critical-bf15f27d-5bfc-4301-8ac4-d021ac062c56' to backup, engine 'pvc-c9d2ffd8-da4f-4d0e-a240-1c06109a43f0-e-0b5296f8'" node=k3s-agent-911b31
2023-09-08T10:51:51.521324325Z time="2023-09-08T10:51:51Z" level=error msg="Failed to sync Longhorn backup longhorn-system/backup-fee13c953cf148a7" controller=longhorn-backup error="longhorn-backup: failed to sync backup longhorn-system/backup-fee13c953cf148a7: could not find snapshot 'critical-bf15f27d-5bfc-4301-8ac4-d021ac062c56' to backup, engine 'pvc-c9d2ffd8-da4f-4d0e-a240-1c06109a43f0-e-0b5296f8'" node=k3s-agent-911b31

The snapshots were deleted before backing them up.

Daily, 00:00 snapshot-cleanup
Daily, 00:00 snapshot
Daily, 00:15 backup
First of all, I am wondering if the cleanup and snapshot at the same time are such a good idea. Secondly, I am wondering if the cleanup could run long enough to interfere with the backup.

Can you try to add enough time interval between the jobs?
I'll reproduce it in my environment and check what we can improve.
Another prove we can improve is that longhorn-manager should not keep retrying failed backups.

Can you try to add enough time interval between the jobs?

Will do. Should be able to see if that makes a differences pretty soon.

Just deleted all the incomplete backups. Is it possible that I am seeing a significant decrease in CPU usage for the longhorn-manager instances?

Another prove we can improve is that longhorn-manager should not keep retrying failed backups.

This should be fixed with https://github.com/longhorn/longhorn/issues/6358. The actual fix was done by @mantissahz in https://github.com/longhorn/longhorn/issues/1249 and is backported to v1.5.2.
It looks like the following should now happen:
We should fail to enable the backup monitor and set the backup state to error (in v1.5.1, the backup state is not set) and lastSyncedAt to now (in v1.5.1, lastSyncedAt is not set). This defer statement should delete the backup's attachment ticket.

Just deleted all the incomplete backups. Is it possible that I am seeing a significant decrease in CPU usage for the longhorn-manager instances?

I think it is quite possible. see #6358 for a possible explanation.

Another prove we can improve is that longhorn-manager should not keep retrying failed backups.

This should be fixed with #6358. The actual fix was done by @mantissahz in #1249 and is backported to v1.5.2.
It looks like the following should now happen:
We should fail to enable the backup monitor and set the backup state to error (in v1.5.1, the backup state is not set) and lastSyncedAt to now (in v1.5.1, lastSyncedAt is not set). This defer statement should delete the backup's attachment ticket.

Cool. Didn't notice the PR. Thanks for the information.
Feel free to close the ticket if https://github.com/longhorn/longhorn/issues/1249 is already fixed it.
cc @ejweber @mantissahz @docbobo

Since this is in some ways an internal implementation detail and not widely documented, I want to give a brief explanation for why your volumes are not detaching. Each Longhorn volume (in v1.5.0+) has an associated volumeattachment.longhorn.io. Included in this object is a list of attachmentTickets. Each one is essentially a reason for the Longhorn volume to be attached to a node. The most obvious (and generally highest priority) reason is a CSI attachment for a workload. If the CSI plugin wants to attach a volume to a particular node, it almost always gets its way. However, a volume may also be attached to a node so that it can take a snapshot or a backup, etc. If the CSI driver allows the volume to detach, it may remain attached for these other operations.
In your cluster, we see volumeattachment.longhorn.io objects with many attachmentTickets. Included in these are backup controller tickets that are never deleted because the backups are infinitely failing. So, when a volume is detached, longhorn-manager immediately reattaches it to try to complete these backups. #6358 should prevent this behavior, though understanding how the current recurring job configuration leads to the failed backups in the first place (so we don't have a large number on the cluster) would be good.

One more thing @docbobo. @PhanLe1010 and I dove pretty deep into your support bundle, and we are also seeing:

AttachmentTickets for snapshots that no longer exist in the cluster, and
Snapshot objects without a finalizer or label representing snapshots that cannot be found in their corresponding engine.

We've been through the code with a fine-toothed comb, and can't see how this can be the case except if you hit https://github.com/longhorn/longhorn/issues/5762 while you were still running Longhorn v1.4.1 or earlier. The most recent snapshot object we see like this was created on January 11, 2023, so it seems a likely scenario. We think you'll continue to run into issues getting volumes to fully detach unless the attachmentTickets are cleaned up, and it's probably a good idea to clean up the snapshot objects as well.
To clean up the attachmentTickets:

Wait for a time of relative stability on the cluster (NOT while recurring jobs are running).
kubectl edit -n longhorn-system lhva
Remove any tickets from spec.attachmentTickets with name like snapshot-controller-*.

To clean up snapshot objects:

Identify the snapshots with no finalizers. For example (in my cluster):

-> kubectl get -n longhorn-system -o custom-columns=Name:metadata.name,Finalizers:.metadata.finalizers --no-headers snapshot
1a5ec6a0-6c72-4528-9ae2-d952bc0a36f8   <none>
6d20e643-1fb8-49a2-9adb-b2cb442454b7   <none>
b283737b-f95d-4349-83b5-57192a74cacb   <none>
f37f4692-86bd-4a4b-b18e-b2845e6db1e6   [longhorn.io]
fd267349-70fc-48a7-bbb6-9a8b1978d712   [longhorn.io]
fe1e92d7-2f3c-4c56-8cc1-eea9a854e218   <none>

-> kubectl get -n longhorn-system -o custom-columns=Name:metadata.name,Finalizers:.metadata.finalizers --no-headers snapshot | grep \<none\>
1a5ec6a0-6c72-4528-9ae2-d952bc0a36f8   <none>
6d20e643-1fb8-49a2-9adb-b2cb442454b7   <none>
b283737b-f95d-4349-83b5-57192a74cacb   <none>
fe1e92d7-2f3c-4c56-8cc1-eea9a854e218   <none>

Delete the snapshots with no finalizers. For example (in my cluster):

-> kubectl get -n longhorn-system -o custom-columns=Name:metadata.name,Finalizers:.metadata.finalizers --no-headers snapshot | grep \<none\> | awk '{print $1}' | xargs kubectl delete snapshot -n longhorn-system
snapshot.longhorn.io "1a5ec6a0-6c72-4528-9ae2-d952bc0a36f8" deleted
snapshot.longhorn.io "6d20e643-1fb8-49a2-9adb-b2cb442454b7" deleted
snapshot.longhorn.io "b283737b-f95d-4349-83b5-57192a74cacb" deleted
snapshot.longhorn.io "fe1e92d7-2f3c-4c56-8cc1-eea9a854e218" deleted

@ejweber @PhanLe1010 thanks for looking into this is so much detail and explaining the backgrounds.
I've cleaned up the snapshots according to your instructions this morning. I will also be sending you a new support bundle. Even though there are now a few hours between the backup, snapshot and cleanup jobs, I had four new incomplete backups this morning.
Maybe there is something new in there that explains this.

Maybe you can look for pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272. I tried doing a manual backup because I noticed that none ran for a while and I am seeing:
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:39Z\" level=warning msg=\"Replica sync agent at <IP_ADDRESS>:10022 is down\" error=\"exit status 2\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:39Z\" level=fatal msg=\"Error running start replica command\" error=\"exit status 2\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:45Z\" level=info msg=\"Listening on data server <IP_ADDRESS>:10021\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:45Z\" level=info msg=\"Listening on sync agent server <IP_ADDRESS>:10022\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:45Z\" level=info msg=\"Listening on gRPC Replica server <IP_ADDRESS>:10020\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:45Z\" level=info msg=\"Listening on sync <IP_ADDRESS>:10022\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:47Z\" level=info msg=\"New connection from: <IP_ADDRESS>:50624\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:50:47Z\" level=info msg=\"Opening replica: dir /host/data/longhorn/replicas/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-38a17983, size<PHONE_NUMBER>, sector size 512\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Initializing backup backup-5217bf2b0e6a45b5 for volume pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272 snapshot d4abe585-954c-4640-9f5f-8b56ac4afcd2\" pkg=backup"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Start creating backup backup-5217bf2b0e6a45b5\" pkg=backup"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Loaded driver for s3://longhorn-backup@datacenter/\" pkg=s3"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Saving config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=start"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Saved config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=complete"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:03Z\" level=info msg=\"Stored lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck type 1 on backupstore\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Prepare removing disk d104e355-5237-464d-a099-4e4c90994d56\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Coalescing volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img to volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Replacing volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img with volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Replacing disk volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img with volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Removing old file volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Hard linking volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img to volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Removing disk volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Done replacing disk volume-snap-d104e355-5237-464d-a099-4e4c90994d56.img with volume-snap-d4abe585-954c-4640-9f5f-8b56ac4afcd2.img\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Loading config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=start"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Loaded config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=complete"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Loaded lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck type 1 on backupstore\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Trying to acquire lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck\" lock=\"{ volume: pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272, name: lock-548df35d62544137, type: 1, acquired: false, serverTime: 2023-09-09 05:51:03 +0000 UTC }\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"backupstore volume pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272 contains locks [{ volume: , name: lock-548df35d62544137, type: 1, acquired: false, serverTime: 2023-09-09 05:51:03 +0000 UTC }]\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Acquired lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck type 1 on backupstore\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Saving config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=start"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Saved config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck kind=s3 object=config pkg=backupstore reason=complete"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Stored lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck type 1 on backupstore\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Loading config in backupstore\" filepath=backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/volume.cfg kind=s3 object=config pkg=backupstore reason=start"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=info msg=\"Removed lock backupstore/volumes/d4/21/pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272/locks/lock-548df35d62544137.lck type 1 on backupstore\" pkg=backupstore"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=error msg=\"Failed to create delta block backup\" destURL=\"s3://longhorn-backup@datacenter/\" error=EOF snapshot=\"&{d4abe585-954c-4640-9f5f-8b56ac4afcd2 2023-09-09T05:51:03Z}\" volume=\"&{pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272<PHONE_NUMBER> map[KubernetesStatus:{\\\"pvName\\\":\\\"pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272\\\",\\\"pvStatus\\\":\\\"Bound\\\",\\\"namespace\\\":\\\"database\\\",\\\"pvcName\\\":\\\"mariadb-0\\\",\\\"lastPVCRefAt\\\":\\\"\\\",\\\"workloadsStatus\\\":[{\\\"podName\\\":\\\"mariadb-0\\\",\\\"podStatus\\\":\\\"Failed\\\",\\\"workloadName\\\":\\\"mariadb\\\",\\\"workloadType\\\":\\\"StatefulSet\\\"}],\\\"lastPodRefAt\\\":\\\"2023-09-09T05:45:26Z\\\"} VolumeRecurringJobInfo:{\\\"critical\\\":{\\\"jobSpec\\\":{\\\"name\\\":\\\"critical\\\",\\\"groups\\\":[\\\"critical\\\"],\\\"task\\\":\\\"snapshot\\\",\\\"cron\\\":\\\"20 */6 * * ?\\\",\\\"retain\\\":8,\\\"concurrency\\\":1},\\\"fromGroup\\\":[\\\"critical\\\"],\\\"fromJob\\\":false},\\\"critical-backup\\\":{\\\"jobSpec\\\":{\\\"name\\\":\\\"critical-backup\\\",\\\"groups\\\":[\\\"critical\\\"],\\\"task\\\":\\\"backup\\\",\\\"cron\\\":\\\"0 1 * * *\\\",\\\"retain\\\":7,\\\"concurrency\\\":1},\\\"fromGroup\\\":[\\\"critical\\\"],\\\"fromJob\\\":false},\\\"daily-snapshot\\\":{\\\"jobSpec\\\":{\\\"name\\\":\\\"daily-snapshot\\\",\\\"groups\\\":[\\\"default\\\"],\\\"task\\\":\\\"snapshot\\\",\\\"cron\\\":\\\"0 0 * * *\\\",\\\"retain\\\":7,\\\"concurrency\\\":1},\\\"fromGroup\\\":[\\\"default\\\"],\\\"fromJob\\\":false},\\\"weekly-backup\\\":{\\\"jobSpec\\\":{\\\"name\\\":\\\"weekly-backup\\\",\\\"groups\\\":[\\\"default\\\",\\\"non-critical\\\"],\\\"task\\\":\\\"backup\\\",\\\"cron\\\":\\\"0 2 * * 6/1\\\",\\\"retain\\\":4,\\\"concurrency\\\":1},\\\"fromGroup\\\":[\\\"default\\\"],\\\"fromJob\\\":false}} longhorn.io/volume-access-mode:rwo] 2023-09-09T05:51:03Z   0   gzip longhorn}\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: panic: runtime error: invalid memory address or nil pointer dereference"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x848558]"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: "
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: goroutine 15 [running]:"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: github.com/longhorn/backupstore.CreateDeltaBlockBackup.func1()"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/github.com/longhorn/backupstore/deltablock.go:130 +0x168"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: github.com/longhorn/backupstore.CreateDeltaBlockBackup({0x4000130390, 0x17}, 0x40001cb450)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/github.com/longhorn/backupstore/deltablock.go:156 +0x189c"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: github.com/longhorn/longhorn-engine/pkg/backup.DoBackupCreate(0x40001270e0, 0x4000130390?)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/pkg/backup/main.go:137 +0x8c"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: github.com/longhorn/longhorn-engine/pkg/sync/rpc.(*SyncAgentServer).BackupCreate(0x40000c4180, {0x40000b8d00?, 0x40000dc150?}, 0x40000b8d00)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/pkg/sync/rpc/server.go:701 +0x214"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: github.com/longhorn/longhorn-engine/proto/ptypes._SyncAgentService_BackupCreate_Handler({0xed23e0?, 0x40000c4180}, {0x10bc120, 0x400049c7e0}, 0x40000dc150, 0x0)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/proto/ptypes/syncagent.pb.go:2069 +0x164"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: google.golang.org/grpc.(*Server).processUnaryRPC(0x40004b4000, {0x10c0080, 0x40000036c0}, 0x4000538360, 0x4000194930, 0x171fc08, 0x0)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:1336 +0xb38"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: google.golang.org/grpc.(*Server).handleStream(0x40004b4000, {0x10c0080, 0x40000036c0}, 0x4000538360, 0x0)"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:1704 +0x80c"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: google.golang.org/grpc.(*Server).serveStreams.func1.2()"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:965 +0x84"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: created by google.golang.org/grpc.(*Server).serveStreams.func1"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: \t/go/src/github.com/longhorn/longhorn-engine/vendor/google.golang.org/grpc/server.go:963 +0x278"
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=warning msg=\"Replica sync agent at <IP_ADDRESS>:10022 is down\" error=\"exit status 2\""
time="2023-09-09T05:51:08Z" level=warning msg="pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272-r-332ddc2d: time=\"2023-09-09T05:51:05Z\" level=fatal msg=\"Error running start replica command\" error=\"exit status 2\""

Another volume of interest would be pvc-61ab63ed-08e7-4399-9157-3ac52dfb1330 - apparently that failed to properly backup for as long as pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272

Okay, so apparently the volume.cfg for the mentioned volumes were 0-byte sized of S3. At least for pvc-531e70e5-1ba8-4bff-b116-eb7f27a2c272 the timestamp said August, 26th. I've deleted the volume.cfg and was able to manually create backups. Would be interesting to understand why that happened thuogh.
BTW: I am doing my backups against a local MinIO instance FWIW.

I have a similar issue, but for me all backups completed. one strictly-local pvc is not detaching after kured starts to drain the nodes. as soon as I manually detach the volume via the UI, the node is drained and kured is able to restart it.

Quick update: no more incomplete backups this morning after removing the zero-byte volume.cfg yesterday. Any idea what might've been causing this? I suspect that

I've been losing some backup data that way
The remnants of that backup will be lying there forever

Or is the latter a wrong assumption?

The snapshots in question are system-generated ones created when a new replica comes online (probably during replica auto-balance). It appears that the snapshot controller creates an attachmentTicket in the VolumeAttachment CR, then deletes the Snapshot CR without cleaning up the attachmentTicket. The finalizing logic in the attachment controller looks sound, but there must be a race somewhere.
I AM able to reproduce this rarely with this script:
for i in {1..1000}; do kubectl get replica -n longhorn-system -o custom-columns=Name:.metadata.name --no-headers | head -n 1 | xargs kubectl delete -n longhorn-system replica; sleep 30; done

Essentially the script deletes a replica every thirty seconds. Given enough time, it manages to orphan an attachment ticket.

Watching both the Snapshot and VolumeAttachment APIs, we see the following string of events:

A Snapshot gets its deletionTimestamp.
The corresponding VolumeAttachment gets a corresponding attachmentTicket.
The corresponding VolumeAttachment gets a corresponding attachmentTicketStatus.
The corresponding VolumeAttachment loses a corresponding attachmentTicket, but still has a corresponding attachmentTicketStatus.
The corresponding VolumeAttachment loses a corresponding attachmentTicketStatus.
The Snapshot is deleted.
The corresponding VolumeAttachment gets a corresponding attachmentTicket.
The corresponding VolumeAttachment gets a corresponding attachmentTicketStatus.

I just did a manual cleanup this morning, removing snapshot-controller-* tickets from 20 of my 42 volumes. Multiple volumes had more than one of those tickets though. I also removed three backup-controller-* tickets. They were all at least nine hours old.
One thing I noticed: the status for the majority of those tickets was saying "the volume is currently attached to different node "

OK, found the root cause by running the recreate script while using a longhorn-manager with a bunch of additional logging.
time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshotName: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2 while finalizer exists: true" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Am reconciling on: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:500290d4-e383-4e50-b871-04ebfa7731e2 GenerateName: Namespace:longhorn-system SelfLink: UID:257ceb79-b140-457b-9b4e-9cdc03f7bc0f ResourceVersion:50705090 Generation:2 CreationTimestamp:2023-09-13 17:11:48 +0000 UTC DeletionTimestamp:2023-09-13 17:12:26 +0000 UTC DeletionGracePeriodSeconds:0xc008fe8698 Labels:map[longhornvolume:test] Annotations:map[] OwnerReferences:[{APIVersion:longhorn.io/v1beta2 Kind:Volume Name:test UID:84218a8f-a6d9-44b9-ad68-401dedee5fb2 Controller:<nil> BlockOwnerDeletion:<nil>}] Finalizers:[longhorn.io] ManagedFields:[{Manager:longhorn-manager Operation:Update APIVersion:longhorn.io/v1beta2 Time:2023-09-13 17:11:48 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:spec\":{\".\":{},\"f:createSnapshot\":{},\"f:labels\":{},\"f:volume\":{}}} Subresource:} {Manager:longhorn-manager Operation:Update APIVersion:longhorn.io/v1beta2 Time:2023-09-13 17:12:22 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:status\":{\".\":{},\"f:checksum\":{},\"f:children\":{\".\":{},\"f:f413016d-fe19-4a59-b868-e3cbe2fc56f0\":{}},\"f:creationTime\":{},\"f:labels\":{},\"f:markRemoved\":{},\"f:ownerID\":{},\"f:parent\":{},\"f:readyToUse\":{},\"f:restoreSize\":{},\"f:size\":{},\"f:userCreated\":{}}} Subresource:status}]} Spec:{Volume:test CreateSnapshot:false Labels:map[]} Status:{Parent: Children:map[f413016d-fe19-4a59-b868-e3cbe2fc56f0:true] MarkRemoved:true UserCreated:false CreationTime:2023-09-13T17:11:43Z Size:153804800 Labels:map[] OwnerID: Error: RestoreSize:5368709120 ReadyToUse:false Checksum:}}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Updating attachmentTicket because old: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc008b44040] Volume:test} new: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc008991fc0 snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2:0xc008b440c0] Volume:test}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Decided to delete attachmentTicket for snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Deleting attachmentTicket because old: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc0091c0480 snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2:0xc0091c04c0] Volume:test} new: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc0091c02c0] Volume:test}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Decided to remove finalizer for snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2 since attachmentTicket is deleted" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="RemoveFinalizerForSnapshot called for snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2"
time="2023-09-13T17:12:26Z" level=info msg="Actually succeeded in removing finalizer for snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2"

time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshotName: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2 while finalizer exists: true" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Am reconciling on: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:500290d4-e383-4e50-b871-04ebfa7731e2 GenerateName: Namespace:longhorn-system SelfLink: UID:257ceb79-b140-457b-9b4e-9cdc03f7bc0f ResourceVersion:50705090 Generation:2 CreationTimestamp:2023-09-13 17:11:48 +0000 UTC DeletionTimestamp:2023-09-13 17:12:26 +0000 UTC DeletionGracePeriodSeconds:0xc0094b2750 Labels:map[longhornvolume:test] Annotations:map[] OwnerReferences:[{APIVersion:longhorn.io/v1beta2 Kind:Volume Name:test UID:84218a8f-a6d9-44b9-ad68-401dedee5fb2 Controller:<nil> BlockOwnerDeletion:<nil>}] Finalizers:[longhorn.io] ManagedFields:[{Manager:longhorn-manager Operation:Update APIVersion:longhorn.io/v1beta2 Time:2023-09-13 17:11:48 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:spec\":{\".\":{},\"f:createSnapshot\":{},\"f:labels\":{},\"f:volume\":{}}} Subresource:} {Manager:longhorn-manager Operation:Update APIVersion:longhorn.io/v1beta2 Time:2023-09-13 17:12:22 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:status\":{\".\":{},\"f:checksum\":{},\"f:children\":{\".\":{},\"f:f413016d-fe19-4a59-b868-e3cbe2fc56f0\":{}},\"f:creationTime\":{},\"f:labels\":{},\"f:markRemoved\":{},\"f:ownerID\":{},\"f:parent\":{},\"f:readyToUse\":{},\"f:restoreSize\":{},\"f:size\":{},\"f:userCreated\":{}}} Subresource:status}]} Spec:{Volume:test CreateSnapshot:false Labels:map[]} Status:{Parent: Children:map[f413016d-fe19-4a59-b868-e3cbe2fc56f0:true] MarkRemoved:true UserCreated:false CreationTime:2023-09-13T17:11:43Z Size:153804800 Labels:map[] OwnerID: Error: RestoreSize:5368709120 ReadyToUse:false Checksum:}}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Updating attachmentTicket because old: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc008afb340] Volume:test} new: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc008afb280 snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2:0xc008afb400] Volume:test}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Request (user: system:serviceaccount:longhorn-system:longhorn-service-account, longhorn.io/v1beta2, Kind=VolumeAttachment, namespace: longhorn-system, name: test, operation: UPDATE) patchOps: [{\"op\": \"replace\", \"path\": \"/spec/attachmentTickets\", \"value\": {\"csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d\":{\"id\":\"csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d\",\"type\":\"csi-attacher\",\"nodeID\":\"eweber-v125-worker-e472db53-xjmzr\",\"parameters\":{\"disableFrontend\":\"false\",\"lastAttachedBy\":\"\"},\"generation\":0},\"snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2\":{\"id\":\"snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2\",\"type\":\"snapshot-controller\",\"nodeID\":\"eweber-v125-worker-e472db53-xjmzr\",\"parameters\":{\"disableFrontend\":\"any\"},\"generation\":1}}}]" service=admissionWebhook
time="2023-09-13T17:12:26Z" level=info msg="Decided to delete attachmentTicket for snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Deleting attachmentTicket because old: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc009211740 snapshot-controller-500290d4-e383-4e50-b871-04ebfa7731e2:0xc009211780] Volume:test} new: {AttachmentTickets:map[csi-9fdc3a3f8765082a1ff2c0d6cd1cba8c4e3f806bc12c02249440f87e2e67738d:0xc009211640] Volume:test}" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=error msg="Failed to sync Longhorn snapshot longhorn-system/500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot error="longhorn-snapshot: failed to sync snapshot longhorn-system/500290d4-e383-4e50-b871-04ebfa7731e2: handleAttachmentTicketDeletion: failed to clean up attachment: Operation cannot be fulfilled on volumeattachments.longhorn.io \"test\": the object has been modified; please apply your changes to the latest version and try again" node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshotName: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Did not find snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Reconciling snapshotName: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr
time="2023-09-13T17:12:26Z" level=info msg="Did not find snapshot: 500290d4-e383-4e50-b871-04ebfa7731e2" controller=longhorn-snapshot node=eweber-v125-worker-e472db53-xjmzr

Our code assumes that once we remove the finalizer on a snapshot, the snapshot controller will never reconcile it again. Before removing the finalizer, we ensure any attachmentTickets we created have been deleted. However, it is possible that we DO reconcile again. In the example above, the first reconciliation creates an attachmentTicket, successfully cleans it up, and removes the finalizer. The second reconciliation (which we don't expect to do) is on the SAME resource version. It creates an attachmentTicket, fails to clean it up (usually no big deal, since we can just get it on the next reconciliation), and errors out. We don't reconcile again (the snapshot object is gone), so the snapshot attachmentTicket remains.
It is known that we may reconcile an object that is previously deleted. We either need to:

Deal with that issue (is there a way to wait for the cache to sync before continuing to reconcile?), or
Rewrite the logic here.

@longhorn/dev-control-plane, any ideas on the first one? I am assuming this is just an aspect of the Kubernetes controller machinery that we have to live with, but I may be missing something.

Recreate logs:
https://mysuse-my.sharepoint.com/:f:/g/personal/eric_weber_suse_com/EnY4MIWFt7FAuM2ov5K0AUcBPClZpE67FGZexLQXWIklxQ?e=G3mWss

Great to hear that you were able to reproduce the problem on your end.
Just a quick update from today: I had exactly one snapshot ticket still dangling around this morning. Based on the timestamps yesterday and the timestamp of the snapshot left today, most of them seem to be the result of the snapshot recurring job I am running daily at midnight UTC.

There might be a good chance that there is a transactional-update running close to the same time, eventually triggering a reboot of the system. For example, I had one node updating at 0:11 UTC, then restarting at 0:16. Unfortunately, I have already deleted the snapshot controller ticket. Will try to look at those numbers more closely tomorrow morning.
Do you think what you've described above could also be caused by concurrent node restarts?

I have renamed this ticket to reflect my intention to use it specifically for the orphan snapshot attachmentTicket issue moving forward. @james-munson is handling the orphan backup attachmentTicket issue in https://github.com/longhorn/longhorn/issues/6654.

@docbobo, I think this issue is generally more likely to occur when there is churn on the cluster, especially related to snapshots being created/deleted in quick succession. However, given the root cause, I do not think there is anything in particular you can do to avoid it for now. (After it occurs, you can clean up attachmentTickets as you have been doing.) I will backport the fix to v1.5.2 and add upgrade logic to remove orphaned tickets, so hopefully users won't hit the issue after upgrading.

Thanks for the update, @ejweber. I will just keep cleaning up the snapshot until hopefully 1.5.2 fixes those issues.

@ejweber I've updated to 1.5.2 do days ago. I've checked the attachment tickets this morning and I can see two volumes with attached snapshot-controller tickets.
However...
There were some weird issues last night - probably load related - and I had nodes reporting as down for a while. Maybe that's related.

Verified pass on longhorn master (longhorn-manager dd6638) with test steps

Run below script for 6 hours in longhorn master, did not observe an snapshot-controller type ticket in volumeattachments.longhorn.io

for i in {1..1000}; do kubectl get replica -n longhorn-system -o custom-columns=Name:.metadata.name --no-headers | head -n 1 | xargs kubectl delete -n longhorn-system replica; sleep 30; done

In Longhorn v1.5.x(v1.5.4-dev), add false attachmentTickets into volumeattachments.longhorn.io then upgrade Longhorn to master(v1.6.0-dev), after upgrade complete, the false attachmentTickets in volumeattachments.longhorn.io are gone(did not see in both spec and status field).


避免多账号同步zstack region异常

这个 PR 实现什么功能/修复什么问题:
避免多账号同步zstack region异常
是否需要 backport 到之前的 release 分支:
None
/cc @swordqiu

Codecov Report

Merging #897 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #897      +/-   ##
=========================================
+ Coverage    7.62%   7.64%   +0.01%
=========================================
  Files         355     355
  Lines       55105   55105
=========================================
+ Hits         4204    4213       +9
+ Misses      50504   50495       -9
  Partials      397     397

Flag
Coverage Δ

#aFlag
7.64% <ø> (+0.01%)
:arrow_up:

Impacted Files
Coverage Δ

pkg/util/seclib2/seclib.go
91.48% <0%> (-4.26%)
:arrow_down:

pkg/util/rbacutils/rabc.go
73.86% <0%> (+4.16%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7fb6731...50f4531. Read the comment docs.

/lgtm
/approve


Connector: Trello

Trello is a visual collaboration tool that creates a shared perspective for teams to organize, prioritize, and manage projects through customizable boards, lists, and cards. Inspired by the kanban system, Trello facilitates task tracking and project management with its drag-and-drop interface, allowing users to easily move tasks (cards) through different stages, assign responsibilities, set deadlines, and attach files. It is part of Atlassian now.
Trello does have APIs: https://developer.atlassian.com/cloud/trello/rest/api-group-actions/#api-group-actions
The search API is here: https://developer.atlassian.com/cloud/trello/rest/api-group-search/#api-group-search
This looks like a typical REST API that returns JSON, so it may be easy to make a SearchProvider that configures RequestsGet to search Trello content. Look at the Atlassian connectors as a starting point: https://github.com/swirlai/swirl-search/blob/main/SearchProviders/atlassian.json

I would like to work on it @sid-swirl

@neelshah2409, thanks for taking up the issue. You've been assigned this issue. Feel free to join our Slack  and ask the team about any issues you're facing.

i would to like to work on this @sid-swirl

@sid-swirl I would also like to work on this issue please assign these to me.

Accepted PR!


Add parameter for exporting all runs to Platform.export_timeseries_data

Add parameter for exporting all model+scenario runs to Platform.export_timeseries_data.

Bugfix (via updated ixmp.jar): Platform.export_timeseries_data should not export data of other models. This happened because ixmp_source DbDAO.exportTimeseriesDataexported the timeseries of all runs if the run_ids list provided via ixmp was empty, see https://github.com/iiasa/ixmp_source/pull/331/commits/fa18af29b5532b2fb7c1bc68f90e5d5623c1c58a - so if no timeseries existed for the provided model/scenario in Platform.export_timeseries_data, all runs were exported instead of none.
Add a parameter to  Platform.export_timeseries_data to allow exporting all model/scenario run combinations

How to review
Check that tests pass and the used naming is compliant with ixmp: runs are not used often in ixmp, maybe there's a more suitable name for the export_all_runs parameter
PR checklist

[x] Add or expand tests.
[x] Add, expand, or update documentation.
[x] Update release notes.

Platform.exportTimeseriesData method contract is to return all runs if empty (or null) run ID list is provided as parameter (same applies to other parameters) and the Python code doesn't fulfill this contract passing empty list where is should not.

@khaeru
I changed the default method behavior of Platform.exportTimeseriesData in ixmp_source, to not return all runs in case of an empty runid list, and added a parameter to still obtain the old behavior. With these changes the currently failing test succeeds.
If you want you can have a quick look at https://github.com/iiasa/ixmp_source/pull/331 - otherwise I'll just merge it and add the updated ixmp.jar to this PR.

If you want you can have a quick look at iiasa/ixmp_source#331 - otherwise I'll just merge it and add the updated ixmp.jar to this PR.

I can't review that PR in detail, so please go ahead. This will also need a rebase, since I had to make some CI fixes on master since your last changes.

I can review and merge this only once:

the title and description of the PR are up to date (they seem not to be),
the PR checklist is complete, and
the PR is no longer in draft state.

I had left the PR in the draft state on intention because I planned to further update it - it's now ready to be reviewed, sorry that it took a while!


ZAV1

Use the preview tab to see the example given below. Edit the relevant information and preview the changes before submitting. You may delete these instructions.

Key
Value

ID
ZAV1

Source
Vaaran Bhai Gurdas Ji

Page
24

Line
hir goivMdu gurU, guru BwieAw ]24]

Correction
THIS ≠ THAT

PROOF (EDITION)
Image:

Correction:
hir goivMdu ≠ hirgoivMdu
Proof:
Vaaran Bhai Gurdaas Ji | November 2011 | Shamsher Singh Ashok | SGPC, Sri Amritsar
Image:


GH-239 consistent argument passthrough

addresses #239 -- removes 'help option' from generate-options to allow newly generated or updated module commands to passthrough arguments with double dashes
this means to view the 'help text' for a module command, one should use the rerun <modulename>: command
see below for examples of identical output sans for allowing passthrough of double-dash arguments
creating module
before / after change (identical output)
$ RERUN_MODULES=./modules rerun stubbs: add-module -m my-module --description "my module"
Created module structure: ./modules/my-module.

creating command
before / after change (identical output)
$ RERUN_MODULES=./modules rerun stubbs: add-command -m my-module -c my-command --description "my command"
Wrote command script: ./modules/my-module/commands/my-command/script
Wrote options parser: ./modules/my-module/commands/my-command/options.sh
Wrote test script: ./modules/my-module/tests/my-command-1-test.sh

creating option (cannot specify an empty default value on the cmd line -- bug?)
before / after change (identical output)
$ RERUN_MODULES=./modules rerun stubbs: add-option -m my-module -c my-command -o rerun-option --required true --description "a required rerun option" --export false --default ""
Default:

Wrote option metadata: ./modules/my-module/options/rerun-option/metadata
Updated command metadata:  ./modules/my-module/commands/my-command/metadata
Updated command script header: ./modules/my-module/commands/my-command/script

my-module/commands/my-command/script implementation
# Command implementation
# ----------------------

# - - -
echo "wrapped-exe $_CMD_LINE"
# - - -

executing without required argument
before / after change (identical output)
$ RERUN_MODULES=./modules rerun my-module: my-command
missing required option: --rerun-option

executing with required argument (with value)
before / after change (identical output)
$ RERUN_MODULES=./modules rerun my-module: my-command --rerun-option blah
wrapped-exe

executing with required argument (with value), and single-dash passthrough argument
before / after change (identical output)
$ RERUN_MODULES=./modules rerun my-module: my-command --rerun-option blah -pass-me-through
wrapped-exe -pass-me-through

executing with required argument (with value), and double-dash passthrough argument
before change
$ RERUN_MODULES=./modules rerun my-module: my-command --rerun-option blah --pass-me-through
unrecognized option: --pass-me-through
usage: rerun my-module:my-command  --rerun-option <>

after change
$ RERUN_MODULES=./modules rerun my-module: my-command --rerun-option blah --pass-me-through
wrapped-exe --pass-me-through

also here's a comparison diff of options.sh before and after:
2c2
< # Created: Wed Sep 27 15:59:11 CDT 2017
---
> # Created: Wed Sep 27 16:07:51 CDT 2017
27,31d26
<             # help option
<             -|--*?) echo >&2 "unrecognized option: $OPT"
<                 rerun_option_usage
<                 exit 2
<                 ;;

Need a test that parses correctly something like these.  The last one should error.  The rest should succeed.
$ rerun module:command --recognized1 val --unrecognized1 val --recognized2 val --unrecognized2 val
$ rerun module:command --recognized1 val --unrecognized1 --recognized2 val --unrecognized2 val
$ rerun module:command --recognized1 val --unrecognized1 --unrecognized2 val --recognized2 val --unrecognized3
$ rerun module:command --recognized1 val --unrecognized1 --unrecognized2 val --recognized2 val --unrecognized3
$ rerun module:command --recognized1 --unrecognized1 val --recognized2 val --unrecognized2 val

@stagrlee I modeled those tests but appears that behavior wouldn't be supported, due to a bug
In addition, the last case wouldn't actually error, because the option --unrecognized becomes the value for --recognized1
On the bright side, I fixed the bug with commit on #244 PR
Basically rather than shifting at the end every time and breaking on unrecognized like this...
    while (( "$#" > 0 ))
    do
        OPT="$1"
        case "$OPT" in
            --recognized1) rerun_option_check $# $1; RECOGNIZED1=$2 ; shift ;;
            --recognized2) rerun_option_check $# $1; RECOGNIZED2=$2 ; shift ;;
            # end of options, just arguments left
            *)
              break
        esac
        shift
    done

Instead, we want to shift twice on known args with values, and shift once on anything unrecognized, and not break.
    while (( "$#" > 0 ))
    do
        OPT="$1"
        case "$OPT" in
            --recognized1) rerun_option_check $# $1; RECOGNIZED1=$2 ; shift 2 ;;
            --recognized2) rerun_option_check $# $1; RECOGNIZED2=$2 ; shift 2 ;;
            # end of options, just arguments left
            *)
              shift
              ;;
        esac
    done

Before this change, you couldn't put unrecognized options in between recognized options, after the change you can (the new tests pass!)

Apologize for delay.  I'm trying to figure out the parsing mechanism.  As far as the double dash syntax, rerun does
'--option value'
and gnu https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html does
'--option=value'
I would imagine the rerun syntax would not be parse-able  by getopt() and argp_parse().
So I think you have it right, if its a rerun recognized option, skip twice, otherwise skip once.  If you are trying to passthrough options to something that is more like rerun (or to another rerun call) you might be out of luck.  In rerun in this case, you would go ahead and need to define the options that will chain into another rerun call.  Interesting...

If you are trying to passthrough options to something that is more like rerun (or to another rerun call) you might be out of luck

Should still be able to passthrough to anything else, I think, as long as you're satisfying the upstream's arguments properly.
e.g. the only real 'problem' I ran into is with your test case example, where you supply:
--recognized1 --unrecognized

Where recognized1 expects an argument, because in that case, rerun is going to assign the value of --unrecognized to it. However, I think this is expected\normal behavior, because in this case, the user is failing to satisfy the argument properly.
As long as the user satisfies all arguments for the rerun command, as expected, then all the remaining arguments should safely pass through.
e.g. where recognized1 expects an argument and recognized2 is a switch:
--recognized1 upstreamvalue --unrecognized downstreamvalue --recognized2

rerun would first see --recognized1 and know to shift out both --recognized1 and upstreamvalue, would get to --unrecognized and would leave it alone, would get to downstreamvalue and would leave it alone, and finally get to --recognized2 and would again recognize it and shift it out.

@stagrlee
Okay, I was stupid!
I was shifting out the unrecognized arguments, but they weren't making their way down to $_CMD_LINE. I've addressed it with a commit on #244 -- see this comment.
I'm going to close this in favor of #242 since I've already merged these changes into it and #242 relies on the updated generate-options behavior, so I don't think it makes sense to merge just this small change without the surrounding tests and context.


login required error

We need to display an appropriate message if user requires to login. currently it's the returned message from server.

displaying a message will be handled on server using message.


Hotfix modal window images background issues

Solution: Add a white background in every modal window images.
Preview on some cases:

Cylinder schema:

before: https://www.cyberbotics.com/doc/reference/cylinder
after: https://www.cyberbotics.com/doc/reference/cylinder?version=hotfix-modal-window-background-color

Hinge2Joint schema:

before: https://www.cyberbotics.com/doc/reference/hinge2joint
after: https://www.cyberbotics.com/doc/reference/hinge2joint?version=hotfix-modal-window-background-color

generated images with shadows:

before: https://www.cyberbotics.com/doc/guide/lidar-sensors
after: https://www.cyberbotics.com/doc/guide/lidar-sensors?version=hotfix-modal-window-background-color

👍


Hide tasks within specific columns in dashboard

If you have a column in your board, lets say backlog, with tasks not
started yet, you can hide the tasks in this column from the dashboard
overview with this PR.
For details see https://github.com/fguillot/kanboard/pull/2378

@fguillot
I changed it to boolean.
Thanks a lot!!


Bug: Initialization Failed

System.InvalidOperationException: Failed to get API key.
---> System.AggregateException: One or more errors occurred. (No such host is known. (raw.githubusercontent.com:443))
---> System.Net.Http.HttpRequestException: No such host is known. (raw.githubusercontent.com:443)
---> System.Net.Sockets.SocketException (11001): No such host is known.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)    --- End of inner exception stack trace ---    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)    at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification)
at Community.PowerToys.Run.Plugin.Scoop.Scoop.GetApiKey() in F:\Programming\C#\PowerToys\src\modules\launcher\Plugins\Community.PowerToys.Run.Plugin.Scoop\Scoop.cs:line 170
--- End of inner exception stack trace ---
at Community.PowerToys.Run.Plugin.Scoop.Scoop.GetApiKey() in F:\Programming\C#\PowerToys\src\modules\launcher\Plugins\Community.PowerToys.Run.Plugin.Scoop\Scoop.cs:line 188
at Community.PowerToys.Run.Plugin.Scoop.Scoop.Init() in F:\Programming\C#\PowerToys\src\modules\launcher\Plugins\Community.PowerToys.Run.Plugin.Scoop\Scoop.cs:line 132
at Community.PowerToys.Run.Plugin.Scoop.Main.InitScoop() in F:\Programming\C#\PowerToys\src\modules\launcher\Plugins\Community.PowerToys.Run.Plugin.Scoop\Main.cs:line 80

The plugin tries to get the API key 3 times on startup. If you don't have a internet connection during this time, you'll get this error.


Add details in top level readme

[ ] building
[ ] dependencies
[ ] testing
[ ] usage
[ ] deployment

Need to clarify what compilers will work and what level of DMD.


Update to Pydantic 2

This is a mostly-straightforward port to Pydantic 2. It takes advantage of Pydantic's new JSON schema mode feature, which can be set to "validation" or "serialization". Tests added and updated and documentation updated.

Hi
Great work here! ... any plans to release this in the short term?
Thanks!

I moved this work to a fork of this project. It's now being maintained:
https://pypi.org/project/openapi-pydantic/
That project now supports both versions of Pydantic.

@hathawsh since you're maintaining it, would you want to get a name transfer on pypi so that other projects relying on this could install it? https://peps.python.org/pep-0541/#how-to-request-a-name-transfer

Any plans to merge this and surface pydantic 2 changes?

@kuimono has not been responsive, so it's probably best for everyone to move to the newer project.


Fix the UglifyJS2 README link

Markdown doesn't support having markdown inside of HTML, so the UglifyJS link wasn't rendering properly.  Now it will.

Thanks. 👍


Add initial prototype

Overview
This is (with very few changes) the state of the application provided to our local LOCKSS network for firewall troubleshooting purposes.
As things stand, the following bits of functionality are available:

cmd/hayoh

small CLI app which attempts to automatically locate LOCKSS peer
nodes and check them for 9729/tcp access
supports explicit config server URL
supports explicit on-disk config server XML file

mostly for testing

Recent changes / current status:

cmd/hayoh is functional for its intended purpose

note: reduced port connect timeout from 10s to 1s to reduce wait
time before results are displayed

several guides I've found recommend a value in milliseconds, so
presumably even 1s is overkill, though much better than 10s
(which gives the impression that the application has hung)

cmd/hayoh has at least one bug

the --port flag supports multiple values via CLI, but the
current code only uses the one value provided by the parsing
results from the main LOCKSS configuration/properties XML
(usually lockss.xml) file

Documentation is very thin

README

missing examples
stub sections

CHANGELOG not yet added
GoDoc coverage is missing content

cmd/n2n is stubbed out, but incomplete

the final form is as of yet unknown
this may become the Nagios plugin I intend to write for this
project, or it may be folded into the cmd/hayoh app

References

https://lockss.github.io/administrators/admin/setting-up/starter-config.html

https://github.com/apex/log

https://github.com/antchfx/xmlquery

https://github.com/antchfx/xpath

CI isn't setup yet, but local linting checks are passing.


[BEAM-9640] Sketching watermark tracking on FnApiRunner

r: @robertwb
This change adds an initial 'sketch' of watermark tracking to the batch runner. Watermark tracking is done like so:

If a PCollection has a delayed application, its watermark will be held at MIN_WATERMARK.
If a PTransform has a channel split, its input PCollection's watermark will be held at MIN_WATERMARK
For timers, an 'input pcollection' node is created for each timer family to be consumed by a transform. If a bundle execution returns a timer set at time X, the input PCollection for that timer family will be held at X. This will hold back the downstream watermarks from the stage.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

unable to repro these errors locally.

Run Python PreCommit

r: @robertwb

Sorry it took so long to get to this. Most of my questions are around watermark advancement.

no worries. This is a critical component, and I have other work to do, so I'm glad to get a thoughtful review. I'll address your comments soon.

Run Python 3.8 PostCommit

@robertwb sorry about the long delay on this, but I've rebased this, and addressed some of your older comments.

cc: @tvalentyn FYI I am now focusing on this change

@robertwb PTAL

@robertwb PTAL

also rebased against master

@robertwb PTAL

@robertwb PTAL

@robertwb PTAL

@robertwb PTAL

@y1chi PTAL?

@y1chi PTAL?

Will do.

@y1chi PTAL

What is the next step on this PR?

it LGTM, I only have one more question on https://github.com/apache/beam/pull/11296/files#r635450347


The new logo is so ugly

The new logo is so ugly, could you use the old logo?

#35783

Dupl #35683

Duplicate of #35783

Hi, Thanks a lot for change icon. But new icon need some stroke line.

Like this:


[Feature] Security Model

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Add document to describes the Apache DolphinScheduler security model from the perspective of the user.  It is intended to help users understand the security model and make informed decisions about how to deploy and manage Apache DolphinScheduler. This document also describes the scope of security vulnerabilities for Apache DolphinScheduler. Developers are advised to read this article before reporting security vulnerabilities.
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Great job!


Use prepend to patch Action Cable classes

To preserve the original functionality a better monkey-patching technique than copying the code should be used, e.g.,Module#prepend.
NOTE: The original Action Cable code for Connection class has been already changed, thus Lograge breaks the core functionality:
  rescue ActionCable::Connection::Authorization::UnauthorizedError
-  respond_to_invalid_request
+  close(reason: ActionCable::INTERNAL[:disconnect_reasons][:unauthorized], reconnect: false) if websocket.alive?
  end

Fixes #304.
Related to #294.

bump! we are seeing this issue too

We're able to reproduce this as well. Good fix!

Yes, this issue still makes Lograge incompatible with AnyCable; we're currently using this patch.

Bump! Now using a patched version of lograge to work around this. Thanks @palkan!

We are seeing builds failing now

Oops, sorry 🙄 (Highly recommend to fork all source-based dependencies to avoid such problems in the future)

@palkan  Awesome change! Mind submitting another PR with a changelog entry?
Hoping to cut a new release soon.


⚠ SceneTime has degraded performance

In edf9f9b, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 1227 ms

Resolved: SceneTime performance has improved in 9a5e4b8 after 8 minutes.


Feature/fixissues

Description
To cater the new requirement the message was changed to show a counter that is ascending rather than descending.
Overview of Changes
UI Message changed
N/A
Type of Change

N/A
Test Details

N/A
Additional Notes

N/A

You did it @sandeepmantha!
Thank you for signing the Anthem Opensource Software CLA.

This goes to 1.1.9


Permissions and Workout data?

We are having issues with the permissions on this to get both step data AND workout data at the same time.  Also the plugin for getting workout data takes no parameters and returns every workout ever?  Is there no alternative/a better way to do this?
The functions we are using...
Workouts - findWorkouts()
The call we want to use to get Workout permission as well is called requestAuthorization().  It feels like we cant find the correct identifier to get workouts here as well as step data.
Our current requestAuthorization call is as below:
requestOptions: HealthKitOptions = {
readTypes: ['HKQuantityTypeIdentifierStepCount'],
writeTypes: ['HKQuantityTypeIdentifierStepCount']
};
this.healthKit.requestAuthorization(this.requestOptions).then(success => {
console.log('auth success', success);
this.checkAppleHealthPermissions();
}, error => {
console.log('auth error', error);
});
The below call also always hits the error, never the success if they have allowed or denied access:
this.healthKit.findWorkouts().then(success1 => {
// for some reason never hits here
console.log('workouts1', success1);
}, success2 => {
console.log('workouts2', success2);
});

Were you ever manage to resolve this? I'm facing the same error...

Does anyone get a solution for this?


⚠ Bofrost.de has degraded performance

In 33a608d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2013 ms

Resolved: Bofrost.de performance has improved in bff134a after 20 minutes.


Producer defaults are not applied?

Description
I'm using the following producer:
            producerConfig = new ProducerConfig
            {
                BootstrapServers = _kafkaServerURL,
                Acks = Acks.All,
                EnableIdempotence = true,
                StatisticsIntervalMs = 60 * 1000,
                 };

with those settings I'm getting delays in producing:

but if I add   LingerMs = 0.5 there is no delay:

Why is that? I assumed defaults are applied without the need to specify them in code (it even says default value 0.5 for Linger)..
I'm using V1.3.0

thanks for letting us know - this is odd, i don't know how specifying the default explicitly could have this effect. would be interested to know if you see this with v1.5.3, and if so, would be interesting to see some Debug=all logs covering a period with the delay. definitely recommend 1.5.3 over 1.3.0 (bug fixes).

we had to revert back to 1.3 due to bad performance with 1.5 (https://github.com/confluentinc/confluent-kafka-dotnet/issues/1444). I'll try again with 1.5 by setting back the defaults that were not good for us in V1.5 and then check the linger value if needed or not.
I'll update..

we aren't aware of any performance issues with v1.5. if you provide a small code snippet demonstrating the issue, we can either comment on the code of investigate further.

we aren't aware of any performance issues with v1.5. if you provide a small code snippet demonstrating the issue, we can either comment on the code of investigate further.


Wrong whisper

requirements.txt installs the whisper package, not the openai-whisper package.

Fixed in 825aabf395c0ae2c6b2d6b7db52799672c8fbc06


Examples of Overengineering

https://github.com/nesterone/sf-eng-init-gl-spring-2016/blob/master/00-bootcamp/03-few-simple-principles/few-simple-principles.md#exercise-2

Overengineering
Overengineering is the designing of a product to be more robust or complicated than is necessary for its application, either (charitably) to ensure sufficient factor of safety, sufficient functionality, or because of design errors.

40 Watt Light bulb lamp that is able to work in 30 kVolt system.

Titanium spoon
Durable children shoes, that you can use for 10+ years
A cup made of bulletproof glass
And the last one is the real example of over-engineering:

#3 under question


Hey there, Could I get some help?

So hi there, basically what I want to do while creating servers, is just print out the {"errors":[{"code":"NoViableAllocationException","status":"400","detail":"No allocations satisfying the requirements for automatic deployment were found."}]} and just that. How could I do that? For now I am using
`    }catch(\HCGCloud\Pterodactyl\Exceptions\ValidationException $e){
print_r($e->errors());
}`

Hi, make sure you have enabled Allow Automatic Allocation for nodes in that location.

Hi, make sure that you have enabled Allow Automatic Allocation for nodes in this location.

Hi. I already know what to do, I am just asking how to print only that part of the code

Also how could I get the ip from an allocation?

Basic PHP knowledge, take a look here https://www.php.net/manual/en/function.json-decode.php


[Feature request] A delay differential equations solver

Dear Patrick,
Thanks for this library, it's pretty neat!
I'm currently supervising a bachelor's students thesis on NeuralODEs, and we've been meaning to use diffrax to study delay differential equations. I'm currently trying to implement a delay differential equations solver inside diffrax, and I would like to know if I'm in the right track.
For context, let me give a brief overview of how delay differential equations work, and how such a solver could be implemented. In it's simplest form, a (constant) Delay Differential Equation (DDE) has a vector field $f$ that depends not only on the current state $y(t)$, but also on $y(t-\tau)$ where $\tau\in\mathbb{R}_{>0}$. In other words
$$y'(t) = f(t, y(t), y(t-\tau)).$$
Initial value problems involving DDEs provide a history instead of a single initial value ($y(t) = \phi(t)$ for $t \leq 0$, for example), and are solved in chunks using the "method of steps". Shortly put, one solves an IVP in intervals of the form $[t_0 + k\tau, t_0 + (k+1)\tau]$. (More details in Chap. 9 of this reference).
In practice, solving a DDE numerically can be done by selecting the right step-size such that $y(t-\tau)$ is always in the grid. To predict y_{t+1} we need to evaluate the vector field in y_{t} and in some y_{t-k} corresponding to $y(t-\tau)$. Other ways of doing it would involve e.g. building an Hermite interpolation between the two relevant points in the grid if $y(t-\tau)$ happens to lie outside of the grid, but I plan to focus on the first alternative.
How could I adapt diffrax to let me pass terms of the form f(t, y(t), y(t-tau))dt? I imagine I have to implement a new DelayTerm that inherits from AbstractTerm with a different vf method; since solvers need to evaluate these vector fields, I imagine I would also need to modify/create a new one in which vf is called with the right signature, right?
I'm of course happy to contribute the implementation to diffrax once it's up and running.

So support for DDEs is something we've been noodling over in Diffrax for a while, see #169.
The main reason that PR stalled is that solving general DDEs requires solving several of nonlinear optimisation problems, and at the time Optimistix did not exist yet.
Now that it does we have been meaning to revisit that PR, fix it up to use the new root-finding functionality that is now available in Optimistix.
I must acknowledge that this is (a) fairly technical code, but also conversely (b) that the hard parts are already written.
If you'd be interested in reviving that PR then this is still a feature I'd be happy to see in Diffrax.

Hello there,
As mentionned by @patrick-kidger, most of the code itself is there (I would say 90%) and functional but some jax related bugs are still there. Happy to discuss if you are interested in giving a hand.

Hi both,
Thanks for the implementation, @thibmonsel! My student and I have been using your dde.ipynb example on a different system of DDEs, and it worked almost out-of-the-box. If I understand correctly, the neuraldde.ipynb example is not finished yet, right?
I would be happy to help, but I fear this is above my skill level (I'm only now starting to use jax). If you give me pointers on how to get started, I could give it a try, but I can't promise much.

That's great to here !
Integrating DDEs itself should be more than robust (so dde.ipynb should work fine).
However, fitting a DDE with a neural net will shootout some Exception: Leaked trace when combined with jax.check_tracer_leaks().
A MWE for this could be :
import equinox as eqx
import jax
import jax.numpy as jnp
import jax.random as jrandom
import numpy as np

import diffrax

class Func(eqx.Module):
    linear: eqx.nn.Linear

    def __init__(self, data_size, *, key, **kwargs):
        super().__init__(**kwargs)
        self.linear = eqx.nn.Linear(2 * data_size, data_size, key=key)

    def __call__(self, t, y, args, *, history):
        return self.linear(jnp.hstack([y, *history]))

class NeuralDDE(eqx.Module):
    func: Func
    delays: diffrax.Delays

    def __init__(self, data_size, delays, *, key, **kwargs):
        super().__init__(**kwargs)
        self.func = Func(data_size, key=key)
        self.delays = delays

    def __call__(self, ts, y0):
        solution = diffrax.diffeqsolve(
            diffrax.ODETerm(self.func),
            diffrax.Euler(),
            t0=ts[0],
            t1=ts[-1],
            dt0=ts[1] - ts[0],
            y0=lambda t: y0,
            saveat=diffrax.SaveAt(ts=ts, dense=True),
            adjoint=diffrax.DirectAdjoint(),
            delays=self.delays,
        )
        return solution.ys

@eqx.filter_value_and_grad
def grad_loss(model, ti, yi):
    y_pred = model(ti, yi[0])
    return jnp.mean((yi - y_pred) ** 2)

@eqx.filter_value_and_grad
def grad_loss_batch(model, ti, yi):
    y_pred = jax.vmap(model, (None, 0))(ti, yi[:, 0])
    return jnp.mean((yi - y_pred) ** 2)

if __name__ == "__main__":
    seed = np.random.randint(0, 1000)
    key = jrandom.PRNGKey(seed)
    ts = jnp.linspace(0.0, 1.0, 10)
    ys = jnp.ones_like(ts)[..., None]
    length_size, datasize = ys.shape

    delays = diffrax.Delays(delays=[lambda t, y, args: 1.0])
    model_dde = NeuralDDE(datasize, delays, key=key)

    with jax.check_tracer_leaks():
        loss, grads = grad_loss(model_dde, ts, ys)

    # Batched version
    ys = jnp.concatenate([2 * jnp.ones((1, 10, 1)), 3 * jnp.ones((1, 10, 1))], axis=0)

    # Silently side-effecting, no error ?
    loss, grads = grad_loss_batch(model_dde, ts, ys)

    # Batch leaked tracer or reporting false positive from Notes in link :
    # https://jax.readthedocs.io/en/latest/errors.html#jax.errors.UnexpectedTracerError
    with jax.check_tracer_leaks():
        loss, grads = grad_loss_batch(model_dde, ts, ys)

Happy to discuss on this thread or more in depth via email (or other medium).


Test download from tab test, upload to tab debug

Create reports in the Debug tab.
Copy them, then clear them.
Go to tab test.
Download the reports.
Go to tab debug. Check that there are still no reports.
Upload the reports downloaded from tab test.
Check that the reports are back.

There are other tests that test uploading two reports. I will do this test with downloading and uploading one report.


Credential offer through out-of-band protocol is not supported yet

When I want to offer-credentials using the out-of-band protocol, I call the /out-of-band/create-invitation endpoint with an attachment from the issuer point of view.
But when I call the /out-of-band/receive-invitation endpoint, using as body the invitation object created from the above request, from the holder point of view, the following error appears --> "Unsupported request~attach type, only request-presentation is supported".
From the code point of view this seems correct, since the receive_invitation function of the out-of-band manager (in aries_cloudagent/protocols/out_of_band/v1_0/manager.py, line 531) checks only if the request_attach[@type] is request-presentation and if not the above error occurs.
Is this functionality not yet implemented or am I doing something wrong??

Hi Stephen, thanks for the clarification about connection-less issuance of credentials.
Basically, I have tried this because, from the Open-API it seems that you are allowed to have as attachments either credential-offer or present-proof to create an invitation with OOB. So by using the /out-of-band/create-invitation you can sucessfully create an invitation using a credential-offer as an attachment. But after your comment, if I am get it right, you can not then use the OOB(from holder point of view) to receive this offer, because it is much better to first establish a connection with the entity to whom a credential is issued.
So you would prefer for example to use OOB to establish the connection and then use this connection and the issue-credential protocol to send the credential?? Am I get it right??

Yes.  There is nothing to technically to stop someone from building a connection-less issue in ACA-Py.  We haven't because we are concerned about the data model -- issuing on a channel where you don't really know to whom you are connected.
We've implemented it on the proof request side because you are just doing a receipt of data from a holder, not issuing them something.

At least the API shouldn't propose offer-credential as an option then. I can take that out in minutes.

Should I remove support for credential-offer attachments on out-of-band invitation creation then? It does seem like a three-card-monte.

I'd say yes.  If/when someone wants to add full support, this is the least of their worries.

Thank you very much for your clarifications guys!!


Ignore unversioned references

Implicit SDK references do not have version numbers and are resolved by the platform. Currently causes an Exception for AspNetCore.App projects (2.2).

@MattLavalleeMA thankbyou for your contribution!


IMAP StartTLS - unclear error message

I can only connect to IMAP Server with StartTLS when setting "TLSAcceptAllCerts": false
Error message is
INFO Mattermail	09:37:16 Using Mattermost Api Version 3
DEBG Mattermail	09:37:16 Debug mode on
INFO Mattermail	09:37:16 Checking new emails
DEBG Mattermail	09:37:16 MailProviderImap.CheckNewMessage
DEBG Mattermail	09:37:16 MailProviderImap.CheckConnection: Dial
DEBG Mattermail	09:37:16 MailProviderImap.CheckConnection:StartTLS
DEBG Mattermail	09:37:16 s0WLmA STARTTLS

EROR Mattermail	09:37:16 MatterMail.InitMatterMail Error on check new messsage: checkConnection with imap server: enable StartTLS: tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
DEBG Mattermail	09:37:16 check new message: checkConnection with imap server: enable StartTLS: tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config
INFO Mattermail	09:37:16 Try again in 30s

Does this error message mean that mattermail cannot verifiy the server certificate?
The server is a public Server (mail.your-server.de, hosted by Hetzner in Germany) and has a valid certificate, openssl s_client -connect mail.your-server.de:143 -starttls imap works.
Version is Version: 4.0-dev, with same account mattermail was working fine (I think is was a 3.0-dev) before.
Andreas

Hi Andreas !
I can't reproduce this error, have you tried to set StartTLS: false in the config file?

It works with "StartTLS": false (resulting in no encryption I think) or with "TLSAcceptAllCerts": false (no checking of the server certificate, being vulnerable to M2M and risking exposing the password).
The server is: mail.your-server.de:143. It seems to be (at least on the web server) a wildcard cert for *.your-server.de

I just ran the test with 3.5 - no error on StartTLS true works with this configuration:
[
	{
		"Name":              "Mattermail",
		"Server":            "https://mm.familie-berger.net",
		"Team":              "familie-berger",
		"Channel"<EMAIL_ADDRESS>		"MattermostUser"<EMAIL_ADDRESS>		"MattermostPass":    "xxxxxx",
		"ImapServer":        "mail.your-server.de:143",
		"Email"<EMAIL_ADDRESS>		"EmailPass":         "yyyyy",
		"MailTemplate":      ":incoming_envelope: _From: **%v**_\n>_%v_\n\n%v",
		"StartTLS":          true,
		"Disabled":          false,
		"Debug":             true,
		"LinesToPreview":    10,
		"NoRedirectChannel": false,
		"NoAttachment":      false,
		"Filter":            []
	}
]

I think this issue was solved, I'll close, feel free to reopen it
Thank you and sorry for delay ;)


Add test suite for CableReady operations

Type of PR
Enhancement
Description
Adds a basic test suite for some selected CableReady operations.
Why should this be added
To ensure the CableReady operations work as expected and to note breaking changes.
Checklist

[x] My code follows the style guidelines of this project
[x] Checks (StandardRB & Prettier-Standard) are passing
[x] This is not a documentation update

Looks great to me!


It's working without jailbreak in IOS?

Will it work on a non-jailbroken(Jailed)? thank you so much

It is not possible to hook without bypassing certain security methods. Apple publishes a white paper every year which describes in detail which protections would prevent you from modifying the executable region of a page - if you defeat these protections this hooking method will work for you.
https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf


Question - JWT Token Expiring Error When Querying Supabase Directly

First off thank you for the application, I am using it as a reference for a solution I am creating.
I am running into a problem where I am getting an error saying
[1] { message: 'JWT expired', code: 'PGRST301', details: null, hint: null }

when doing queries in my application. I am confused because the call to getAuthSession is returning a valid session but when get the supabase client
    const supabaseClient = supabase(authSession?.accessToken as string);

and make a database call, it errors out

Hello, thanks for your kind words :)
The best way to get accessToken (in my stack) is to use requireAuthSession instead of getAuthSession directly (I guess it's what you did ?).
requireAuthSession internally calls getAuthSession but do so much more like refreshing tokens and saving new tokens in session cookie (for HTTP GET only, like loader function).
It throws if It's not possible to refresh.
Example : https://github.com/rphlmr/supa-fly-stack/blob/868edd7c783b846415e2a50b9c4182b272099933/app/routes/rls/notes.tsx#L11-L15
That's for  loader function.

Example : https://github.com/rphlmr/supa-fly-stack/blob/868edd7c783b846415e2a50b9c4182b272099933/app/routes/rls/notes/new.tsx#L59-L75
That's because, requireAuthSession :

could return new accessToken and refrechToken when token expires or expires soon (10 minutes threshold)
For HTTP GET, requireAuthSession handles a redirect (like a reload) and saves new tokens for you with a redirect to request.url (because GET is safe to redirect).
For HTTP POST/PATCH/PUT/DELETE, it's not safe / possible to redirect / replay, so, developer have to commit sessions when returns from action function.

Starting point : https://github.com/rphlmr/supa-fly-stack/blob/main/app/core/auth/guards/require-auth-session.server.ts
Then : https://github.com/rphlmr/supa-fly-stack/blob/main/app/core/auth/mutations/refresh-auth-session.server.ts
Let me know if it solves your issue or if I'm totally wrong ;)

A question on this: isn't it enough to use the requireAuthSession in the loader?

I think he is just ensuring that you have a valid token before you attempt
to take some action on the database. That I what I got out of his comment
and reading through the code.
On Thu, Aug 4, 2022 at 7:28 PM Michael Sevestre @.***>
wrote:

A question on this: isn't it enough to use the requireAuthSession in the
loader?
—
Reply to this email directly, view it on GitHub
https://github.com/rphlmr/supa-fly-stack/issues/39#issuecomment-1205874768,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEAFGNXLTEJFKZ4KIFWD53VXRGZ7ANCNFSM55TFOOVQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
--
Aaron K. Saunders
CEO Clearly Innovative Inc
@.***
www.clearlyinnovative.com
This email message and any attachment(s) are for the sole use of the
intended recipient(s) and may contain proprietary and/or confidential
information which may be privileged or otherwise protected from disclosure.
Any unauthorized review, use, disclosure or distribution is prohibited. If
you are not the intended recipient(s), please contact the sender by reply
email and destroy the original message and any copies of the message as
well as any attachment(s) to the original message.

Yes but wouldn't the loader kick in before?

A question on this: isn't it enough to use the requireAuthSession in the loader?
If you need the userId, you could have a method that can retrieve it from the request and simply throw if it's not available?

Hi, loader and action have to be treated independently since anything can post directly on your action without triggering loader (like a fetcher from useFetcher, external fetch, ...).
They are independent endpoints in the end.

Sure. But in the case of the notes, you could throw if the authSession was invalid no? I understand what the aim is but I am wondering if this is required

I'll post a code snippet tomorrow

ok so this is what I meant:
what about having two sets of methods:
for Loader
We use requireAuthSession. This ensures that we actually have a session and that we can refresh the session if it is expiring etc
for Actions (POST, PUT,etc...)
We use something relying on  getAuthSession but that would throw an error if it's not valid.
something like this
export async function getCurrentAuthSession(request: Request) {
  const authSession = await getAuthSession(request);
  // If there is no user session, Fly, You Fools! 🧙♂
  if (authSession?.accessToken && authSession?.refreshToken) {
    return authSession;
  }
  throw new Error("no_user_session");
}

Problem with this: This does not refresh the session on action. But with your latest PR, where the  token is refreshed a few minutes before expiring, this should prevent issues with token expiring while the action is performed.
This way, the action code is much leaner and you do not have to deal with sessions etc in in your action code.
Thoughts?

Your action can be called by something else, outside your app (curl, malicious app, etc.).
You can do that, but getAuthtSession doesn't verify your accessToken with Supabase.
If you use Prisma and not supabase client with RLS (Prisma bypass RLS security), it's a security issue since it's possible to steal network requests (corp proxy, malicious add-on, public Wi-Fi, ...).

Example :
First, to not wait 50 min before refresh, set JWT expiry limit to 10 in your supabase dashboard
 and change REFRESH_THRESHOLD to 0.
What we want to test here is how the code reacts when accessToken is expired.
Iin routes/notes/new.tsx I have this action :
export async function action({ request }: LoaderArgs) {
  assertIsPost(request);

  const authSession = await getAuthSession(request);
  if (!authSession) throw new Error("fail");

  console.log(authSession);

  const formValidation = await getFormData(request, NewNoteFormSchema);

 ...
}

Open network inspector and make some new notes.
Take one of the POST requests, right click, copy  as cURL

Logout.

At this time, 10s later, accessToken in cURL should be expired.

Run the curl in a terminal.
Log in and you should see unwanted notes.

Retry this process with requireAuthSession, no unwanted notes.
👇 (record)
https://user-images.githubusercontent.com/20722140/183120776-1532bc47-ea53-4b18-9266-b25d24b4c177.mp4
To avoid this, you should verify accesToken with Supabase, but it's like using requireAuthSession and do not commit its result.
I think that if it's the "commit session" part that you want to remove, keep requireAuthSession or rewrite a version without refresh :
export async function action({ request }: LoaderArgs) {
  assertIsPost(request);

  const authSession = await requireAuthSession(request);
  const formValidation = await getFormData(request, NewNoteFormSchema);

  if (!formValidation.success) {
    return json(
      {
        errors: {
          title: formValidation.errors.title,
          body: formValidation.errors.body,
        },
      },
      {
        status: 400,
      }
    );
  }

  const { title, body } = formValidation.data;

  const note = await createNote({ title, body, userId: authSession.userId });

  return redirect(`/notes/${note.id}`);
}

If you use supabase client with RLS, then, you should not have a security issue because it'll reject your accessToken

The only cons I can see by removing "refresh on action "and "commit session" is the case where user is idle (going to lunch :D), come back and submit to your action without refreshing before => logout and form content lost

Hey @rphlmr
Yes you are right. The verify session with supabase needs to happen indeed. I think this is a good compromise
Thanks so much for the amazing answer 😊

Feel free to reopen as I made some changes since the day you open this issue 😬


steps to add a custom hand

Nice work! Would to please share the steps about how to add a custom hand and perform retargeting? Thanks!

@zhongxiangzju
Creating a config file for a new robot hand is not so hard, but the tricky thing is that you need a hand and with good quality. Which hand do you want to add? Do you have a link to the manufacture provided urdf?

Thanks for quick reply! I have added urdf of the custom dexterous gripper and run position retargeting. The results seem reasonable but with serious piercing problems.

Any suggestions to improve the results? Thanks.

And could you please introduce the difference between position retargeting and vector retargeting?

To put it simply, vector/DexPilot retargeting is mainly for teleoperation while position retargeting is for retargeting from video

To alleviate the penetration, you can try to modify the retargeting config and add more position on the robot hand

@zhongxiangzju Hi bro, how to create a config file on custom hand ? Thank you~


Chrysalis Phase 2 completion

Successor of https://github.com/iotaledger/identity.rs/pull/155
Includes https://github.com/iotaledger/identity.rs/pull/160

Looks like CI is failing because the wasm bindings don't compile after these changes - just need to handle the change to async here:
https://github.com/iotaledger/identity.rs/blob/dev/bindings/wasm/src/iota.rs#L45

Currently blocked by lack of wasm support in iota.rs
https://github.com/iotaledger/iota.rs/issues/444


Validation for storage uri in webhook

What this PR does / why we need it: The webbook will validate the storageURI passed in the trained model
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

None

/assign @yuzliu @yuzisun

/lgtm
/approve


Migrate the emoji sample app to use Redwood Layout Row/Column.

This is a WIP as iOS builds, but doesn't render properly. Android is pretty solid:
video.webm

Okay this should be good to review now. It's definitely not production ready, but both samples run with only a couple (common) issues.