[BUG] Failed rebuilding a large replica of 3TB

Describe the bug
I have a large PV (2 replicas) of 3TB of which data I just rsynced from another storage system; the size of data in the PV is 2.4TB. When I rebooted one node, and replica in the rebooted node always failed and another replica was created but never managed to finish. So I tested with reducing replica count to 1 and increasing again to 2, however rebuilding also failed. The CPU load in the source node was around 1 with a longhorn process running, but no data was actually transfer in the network during this time. When data was originally rsynced from another storage, I assume this snapshot is 2.4TB with no holes in the sparse file. Is the reason it fails because it's too large? I have other PVs of 50GB size and have no issue rebuilding.
To Reproduce
Steps to reproduce the behavior:

Create a PV with 2 replicas of 3TB
Rsync-ed a 2.4TB data
Reduce replica count to 1, increase it again to 2
Rebuilding ran for less than 1 hour, than failed

Expected behavior
Replica should rebuild successfully.
Log
If applicable, add the Longhorn managers' log when the issue happens.
time="2021-04-18T06:00:54Z" level=error msg="Failed rebuilding of replica <IP_ADDRESS>:10075" controller=longhorn-engine error="failed to add replica address='tcp://<IP_ADDRESS>:10075' to controller 'pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.1.0/longhorn [--url <IP_ADDRESS>:10000 add tcp://<IP_ADDRESS>:10075], output , stderr, time=\"2021-04-18T05:13:17Z\" level=info msg=\"Adding replica tcp://<IP_ADDRESS>:10075 in WO mode\"\ntime=\"2021-04-18T05:13:17Z\" level=info msg=\"Using replica tcp://<IP_ADDRESS>:10075 as the source for rebuild \"\ntime=\"2021-04-18T05:13:17Z\" level=info msg=\"Using replica tcp://<IP_ADDRESS>:10075 as the target for rebuild \"\ntime=\"2021-04-18T06:00:54Z\" level=fatal msg=\"Error running add replica command: failed to sync files [{FromFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img ToFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img ActualSize:2593904701440} {FromFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img.meta ToFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img.meta ActualSize:0} {FromFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img ToFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img ActualSize:18315870208} {FromFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img.meta ToFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img.meta ActualSize:0} {FromFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img ToFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img ActualSize:34347081728} {FromFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img.meta ToFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img.meta ActualSize:0} {FromFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img ToFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img ActualSize:0} {FromFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img.meta ToFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img.meta ActualSize:0}] from tcp://<IP_ADDRESS>:10075: rpc error: code = Unavailable desc = transport is closing\"\n, error exit status 1" node=slave-sys2 volume=pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0
You can also attach a Support Bundle here. You can generate a Support Bundle using the link at the footer of the Longhorn UI.
longhorn-support-bundle_a34f5b3a-be96-4afd-8271-6de8b0ebaa01_2021-04-18T06-33-04Z.zip
Environment:

Longhorn version: 1.1.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Rancher Catalog App
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2 1.20.5+rke2r1

Number of management node in the cluster: 1
Number of worker node in the cluster: 2

Node config

OS type and version: Ubuntu 20.04.2
CPU per node: 32
Memory per node: 256GB
Disk type(e.g. SSD/NVMe): NVMe
Network bandwidth between the nodes: 10Gbps

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: ~20

Additional context
Add any other context about the problem here.

Hi! Can you provide information on disk size (NVME)?  From log, I saw repeated messages like;
2021-04-18T05:36:21.216620804+04:00 time="2021-04-18T01:36:21Z" level=error msg="There's no available disk for replica pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-r-4815d394, size<PHONE_NUMBER>328"

and all the PVC is degraded, which mean cluster doesn't meet the minimum resource requirement.
NODE        NAME                                      STATE     STATUS    SIZE           REPLICA
master3     pvc-1492d039-409b-42f2-8809-25bd4cac8601  attached  degraded  209715200      3
master3     pvc-2e8dc1b4-504f-439c-8e89-3ddba7d1d814  attached  degraded  104857600      3
master3     pvc-8f76f7ba-1204-4d13-a102-a69378a2ba54  attached  degraded<PHONE_NUMBER>0    3
master3     pvc-bea32350-c1fe-4757-93f8-127d9c750da3  attached  degraded<PHONE_NUMBER>0    3
            pvc-4abcdf2b-f1cf-4aca-861f-190d0c1d06c2  detached  unknown   10485760       3
            pvc-85bd81b2-7380-4834-8165-b480a2b26686  detached  unknown   10485760       3
            pvc-d76290b8-32bc-45c0-b700-3f4459739efc  detached  unknown   20971520       3
            pvc-df38c94f-ae33-428d-a72a-77a0613e01de  detached  unknown   10485760       3
slave-sys2  pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0  attached  degraded<PHONE_NUMBER>328  2
slave-sys2  pvc-da8b91d4-ef94-4e21-8a90-2048ec44fef4  attached  degraded  10485760       3
slave-sys3  pvc-3a30f709-1e87-4fb2-be26-ca151dbb0bcf  attached  degraded  10485760       3
slave-sys3  pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7  attached  degraded<PHONE_NUMBER>0    3
slave-sys3  pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be  attached  degraded<PHONE_NUMBER>0    3
slave-sys3  pvc-ecf5db8c-9091-4705-b664-df17e7c1375b  attached  degraded  10485760       3

Notice that some of PVC are on /var/lib/longhorn/nvme and others are in /var/lib/longhorn/ssd.

Hi. They're degraded because default replica count is 3, but I only have 2 nodes on the slave side at the moment (slave-sys2/3). Most of the PVs are in ssd disks and have no issues because most PVs have size of 50G. The one failing rebuilding is on NVMe. NVMe size is 6TB with default overprovisioning of 200%.

From longhorn-manager.log, we can see the schedule backup cannot find an available disk where pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0 reside on slave-sys2 and clean up the process and eventually lead to the error which provide in ticket description.
2021-04-18T09:12:43.148701782+04:00 time="2021-04-18T05:12:43Z" level=error msg="There's no available disk for replica pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-r-4815d394, size<PHONE_NUMBER>328"
2021-04-18T09:12:43.148729555+04:00 time="2021-04-18T05:12:43Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev node=slave-sys2 owner=slave-sys2 replica=pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-r-4815d394 state=attached volume=pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0
2021-04-18T09:12:43.148741673+04:00 time="2021-04-18T05:12:43Z" level=info msg="Cleaning up failed to scheduled replica pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-r-4815d394"

Replicas will take up space residing on host /var/lib/longhorn/nvme/replica/pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0, which includes snapshot data. Seems like there are already data there. Overprovision works if only volume didn't take up most request space.  The actual size for pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0 is<PHONE_NUMBER>968, which is more than half of requested space.

@cclhsu The original replica is in slave-sys2. It's trying to schedule a replica in slave-sys3 of which space in /var/lib/longhorn/nvme is almost empty (6TB space).
root@slave-sys2:~# df -h /var/lib/longhorn/nvme/
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/vg--nvme-k8s  5.8T  2.5T  3.1T  44% /var/lib/longhorn/nvme
root@slave-sys2:~# ssh slave-sys3
Last login: Mon Apr 19 10:14:34 2021 from <IP_ADDRESS>
root@slave-sys3:~# df -h /var/lib/longhorn/nvme/
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/vg--nvme-k8s  5.8T   98M  5.5T   1% /var/lib/longhorn/nvme

I have also tried merging 2 snapshots in the single healthy replica by deleting 17GB snapshot so it merges to the larger 2.4TB snapshot. It has been going on for 24 hours. The original creation of 2.4TB data only took 7 hours.

For the 1st rebuilding failure, which is also what you reported in the beginning, we saw a HTTP call detach. If users try to detach a volume during the rebuilding, the rebuilding will be interrupted then gets failed:
2021-04-18T10:00:54.783660865+04:00 time="2021-04-18T06:00:54Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514\", UID:\"60bdf9fc-5dfb-4999-9567-c12d923974d7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"2376654\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with Address <IP_ADDRESS>:10075: failed to add replica address='tcp://<IP_ADDRESS>:10075' to controller 'pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0': failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.1.0/longhorn [--url <IP_ADDRESS>:10000 add tcp://<IP_ADDRESS>:10075], output , stderr, time=\"2021-04-18T05:13:17Z\" level=info msg=\"Adding replica tcp://<IP_ADDRESS>:10075 in WO mode\"\ntime=\"2021-04-18T05:13:17Z\" level=info msg=\"Using replica tcp://<IP_ADDRESS>:10075 as the source for rebuild \"\ntime=\"2021-04-18T05:13:17Z\" level=info msg=\"Using replica tcp://<IP_ADDRESS>:10075 as the target for rebuild \"\ntime=\"2021-04-18T06:00:54Z\" level=fatal msg=\"Error running add replica command: failed to sync files [{FromFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img ToFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img ActualSize:2593904701440} {FromFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img.meta ToFileName:volume-snap-1dfe2ff5-eb10-41be-a686-1661013710be.img.meta ActualSize:0} {FromFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img ToFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img ActualSize:18315870208} {FromFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img.meta ToFileName:volume-snap-51bf61cc-0f68-4a03-848c-07b8c35a51d5.img.meta ActualSize:0} {FromFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img ToFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img ActualSize:34347081728} {FromFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img.meta ToFileName:volume-snap-5e304d7a-bec7-4ec1-9ec0-598443e46306.img.meta ActualSize:0} {FromFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img ToFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img ActualSize:0} {FromFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img.meta ToFileName:volume-snap-6dcea017-1fe4-45be-9f3f-88207a3de8cb.img.meta ActualSize:0}] from tcp://<IP_ADDRESS>:10075: rpc error: code = Unavailable desc = transport is closing\"\n, error exit status 1"
2021-04-18T10:00:54.789161648+04:00 time="2021-04-18T06:00:54Z" level=info msg="Removed failed rebuilding replica <IP_ADDRESS>:10075" controller=longhorn-engine node=slave-sys2 volume=pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0
2021-04-18T10:00:54.793428137+04:00 time="2021-04-18T06:00:54Z" level=info msg="Marked failed rebuild on replica pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-r-73048188, backoff period is now 10 seconds" controller=longhorn-engine node=slave-sys2 volume=pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0
2021-04-18T10:00:54.862331401+04:00 time="2021-04-18T06:00:54Z" level=debug msg="Detaching volume pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0 from slave-sys2"
2021-04-18T10:00:54.862842280+04:00 <IP_ADDRESS> - - [18/Apr/2021:06:00:54 +0000] "POST /v1/volumes/pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0?action=detach HTTP/1.1" 200 3818 "" "Go-http-client/1.1"

For the 2nd failure, which is the failure you reported later, it's caused by the read timeout from a healthy replica, I am not sure if it is caused by network fluctuation or CPU/Disk surge.
2021-04-18T10:19:58.450458764+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Connecting to remote: <IP_ADDRESS>:10075"
2021-04-18T10:19:58.451551566+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Opening: <IP_ADDRESS>:10075"
2021-04-18T10:19:58.466642742+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Snapshot: <IP_ADDRESS>:10075 018bc2f4-1c08-41d0-a66c-1897c779d5ff UserCreated false Created at 2021-04-18T06:19:58Z, Labels map[]"
2021-04-18T10:19:58.467845149+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Snapshot: <IP_ADDRESS>:10075 018bc2f4-1c08-41d0-a66c-1897c779d5ff UserCreated false Created at 2021-04-18T06:19:58Z, Labels map[]"
2021-04-18T10:19:58.468976176+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Adding backend: tcp://<IP_ADDRESS>:10075"
2021-04-18T10:19:58.469172387+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Start monitoring tcp://<IP_ADDRESS>:10075"
2021-04-18T10:19:58.472966877+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Set revision counter of <IP_ADDRESS>:10075 to : 0"
2021-04-18T10:19:58.473814013+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Set backend tcp://<IP_ADDRESS>:10075 revision counter to 0"
2021-04-18T10:19:58.476513942+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Synchronizing volume-head-005.img.meta to volume-head-001.img.meta@<IP_ADDRESS>:10078"
2021-04-18T10:19:58.478880042+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:19:58Z" level=info msg="Done synchronizing volume-head-005.img.meta to volume-head-001.img.meta@<IP_ADDRESS>:10078"
2021-04-18T10:21:38.264829875+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:21:38Z" level=error msg="Read timeout on replica <IP_ADDRESS>:10076 seq= 147 size= 512 (kB)"
2021-04-18T10:21:38.264862736+04:00 time="2021-04-18T06:21:38Z" level=error msg="Retry  1 on replica <IP_ADDRESS>:10076 seq= 147 size= 512 (kB)"
2021-04-18T10:21:38.265714795+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:21:38Z" level=error msg="Read timeout on replica <IP_ADDRESS>:10076 seq= 148 size= 1024 (kB)"
2021-04-18T10:21:38.265724133+04:00 time="2021-04-18T06:21:38Z" level=error msg="Retry  1 on replica <IP_ADDRESS>:10076 seq= 148 size= 1024 (kB)"

As for the stuck snapshot deletion/merge, can you point out which volume the issue happened on? It's better to tell when. Since I saw there are lots of snapshot deletion & merge in the logs, most of them spent several seconds only. e.g.,
2021-04-15T14:00:06.142707573+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Prepare removing disk: snap-31e6835b"
2021-04-15T14:00:06.142889261+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Coalescing volume-snap-snap-7d066325.img to volume-snap-snap-31e6835b.img"
2021-04-15T14:00:06.183703922+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Replace volume-snap-snap-7d066325.img with volume-snap-snap-31e6835b.img"
2021-04-15T14:00:06.184191754+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Replacing disk volume-snap-snap-31e6835b.img with volume-snap-snap-7d066325.img"
2021-04-15T14:00:06.184508342+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Old file volume-snap-snap-7d066325.img exists, deleting"
2021-04-15T14:00:06.203081121+04:00 [pvc-41d1fe78-9ea2-4307-8337-e9ea16f722e7-r-0c61c297] time="2021-04-15T10:00:06Z" level=info msg="Done replacing volume-snap-snap-7d066325.img with volume-snap-snap-31e6835b.img"

2021-04-15T14:00:10.707822206+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Prepare removing disk: snap-589c357f"
2021-04-15T14:00:10.708123054+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Coalescing volume-snap-snap-bd32d492.img to volume-snap-snap-589c357f.img"
2021-04-15T14:00:10.710401947+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Replace volume-snap-snap-bd32d492.img with volume-snap-snap-589c357f.img"
2021-04-15T14:00:10.710763397+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Replacing disk volume-snap-snap-589c357f.img with volume-snap-snap-bd32d492.img"
2021-04-15T14:00:10.711024983+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Old file volume-snap-snap-bd32d492.img exists, deleting"
2021-04-15T14:00:10.729899264+04:00 [pvc-dd15ce97-8276-4300-b26b-9bb0373ae9be-r-0fce5843] time="2021-04-15T10:00:10Z" level=info msg="Done replacing volume-snap-snap-bd32d492.img with volume-snap-snap-589c357f.img"
......

The detach you mentioned is because I had to reboot the node. The status is rebuilding, but the longhorn CPU process in slave-sys2 only spins at load of 1-2 and did nothing else. There's also zero data transfer to slave-sys3 for the large snapshot. When I cancel rebuilding the longhorn process got stuck. top(1) shows longhorn process running at load 1 with zero VIRT/RSS memory (literally ZERO).
The snapshot merging was done after I submitted support bundle. It finished successfully after 30 hours. But merging 17GB for 30 hours is a bit too long.
I tried creating a new PV, recopied the data from original source, and repeat the process of deleting one replica and rebuilding to no avail. As far as I can see, when processing a snapshot this large, longhorn engine does something on the snapshot source (maybe checksumming or something) but never transfer any data across network (literally zero data transfer).
I've managed to use PV as large as 400GB with no issues.

I do find that stopping replicas then detaching takes more than 90s (from 2021-04-18T06:17:03Z to 2021-04-18T06:18:35Z). I guess the replica needs some time to complete the in-fly data processing. But I don't understand what the zero data transfer is.
The snapshot merging you mentioned is abnormal. Can you provide more info about this part? I mean the related volume name/ snapshot name/timestamp.
BTW. I don't know why the instance manager pod instance-manager-r-35918346 only contains the log after 2021-04-18T06:17:03Z. Regradless the pod is not crashed recently.

Before snapshot creation/merge, the engine will flush the data into replicas via sync before v1.1.1, or freeze the volume in v1.1.1 to avoid unexpected data async/loss. Then each replicas is responsible for operating the actual snapshots by themselves and the engine won't transfer data for them.
The error I mentioned means somehow the engine encounters the read timeout from a healthy replica when the engine receive the io request from applications. There is no nothing Longhorn can do to prevent this.
2021-04-18T10:21:38.264829875+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:21:38Z" level=error msg="Read timeout on replica <IP_ADDRESS>:10076 seq= 147 size= 512 (kB)"
2021-04-18T10:21:38.264862736+04:00 time="2021-04-18T06:21:38Z" level=error msg="Retry  1 on replica <IP_ADDRESS>:10076 seq= 147 size= 512 (kB)"
2021-04-18T10:21:38.265714795+04:00 [pvc-a9bb1886-22a7-42d5-b4dc-3ecfa4fa21b0-e-f333a514] time="2021-04-18T06:21:38Z" level=error msg="Read timeout on replica <IP_ADDRESS>:10076 seq= 148 size= 1024 (kB)"
2021-04-18T10:21:38.265724133+04:00 time="2021-04-18T06:21:38Z" level=error msg="Retry  1 on replica <IP_ADDRESS>:10076 seq= 148 size= 1024 (kB)"

Normally for a normal snapshot replication, longhorn in the source node would start processing in a few seconds and show progress transfer to the corresponding snapshot in the target node. "zero data transfer" just means longhorn in the source node starts processing the source snapshot and the size of the snapshot in the target node is 0.
However I now know what's the bottleneck after testing with a smaller set of data. It's not the size of the data in the snapshot but the number of files (and correspondingly the number of holes in the snapshot sparse file). This was also most likely the reasons the merging of snapshots took so long (there was only 1 replica at the time, so data transfer in the network is irrelevant). The merging was 17GB of data in 1 snapshot to 2.4TB data in another snapshot, but the number of files in the 2.4TB snapshot is around 21 millions.
I reran the test with a smaller set, a PV snapshot with 1.9TB data, 20 million files in the snapshot. As you can see below, longhorn does processing for around 8 hours before starting the data transfer to the corresponding snapshot in the the target node. The data transfer after this intial processing is around 6 hours.
Source node:
[pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-edc988a8] time="2021-04-28T18:23:21Z" level=info msg="Sending file volume-snap-38d221f9-eabf-4aff-8803-5d3444062443.img.meta to <IP_ADDRESS>:10112" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-edc988a8] time="2021-04-28T18:23:21Z" level=info msg="source file size: 178, setting up directIo: false" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-edc988a8] time="2021-04-28T18:23:21Z" level=info msg="Done sending file volume-snap-38d221f9-eabf-4aff-8803-5d3444062443.img.meta to <IP_ADDRESS>:10112" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-edc988a8] time="2021-04-28T18:23:21Z" level=info msg="Sending file volume-snap-6087797b-f04a-4f6c-adb0-d67cc9a3306c.img to <IP_ADDRESS>:10113" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-edc988a8] time="2021-04-28T18:23:21Z" level=info msg="source file size:<PHONE_NUMBER>328, setting up directIo: true"
Target node:
[pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-28T18:23:21Z" level=info msg="open: receiving fileSize: 178, setting up directIo: false" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-28T18:23:21Z" level=info msg="Ssync server opened and ready" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-28T18:23:21Z" level=info msg="Running ssync server for file volume-snap-6087797b-f04a-4f6c-adb0-d67cc9a3306c.img at port 10113" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-28T18:23:21Z" level=info msg="Closing ssync server" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-29T02:37:23Z" level=info msg="open: receiving fileSize:<PHONE_NUMBER>328, setting up directIo: true" [pvc-11313189-67a6-4589-af8d-3ecee3c76604-r-dca34737] time="2021-04-29T02:37:23Z" level=info msg="Ssync server opened and ready"
I then tested creating a new PV, write 1 file of size 1.9TB using:
dd if=/dev/urandom of=test1 bs=1G count=1900 iflag=fullblock
longhorn took only a few seconds to process before starting the data transfer across the network. The data transfer took around the same time to finish as the above case, in the order of 6 hours.
The difference of above 2 cases are both have the same data usage of 1.9TB in the source snapshot, but 1 has 20 million files, the other one 1 file. The initial processing of 8 hours (before starting any data transfer across the network) is a function of the number of files. And after reading the issue and longhorn source code mentioned here github.com/longhorn/longhorn/issues/1948 hence a function of the number of holes in the sparse file which in my case probably in the order of a million.
At least now I know longhorn does the replica rebuild, it's just because the initial processing was so long I didn't know whether it was just stuck or not. So my question is, is this initial processing delay a result of longhorn inefficiently does extents/finding holes in the sparse file, or it's inherent ext4 sparse file processing for large number of holes?

Thank you for your reporting! The case that massive holes in a large snapshot file greatly degrades the rebuilding speed is what Longhorn team has not paid attention to.
For the rebuilding progress, I think there is a tiny misleading. Let me explain the actual progress then answer the questions:

the engine list all snapshot file and meta files in the source replica.
launch the target(new) replica process.
The target replica establishes connections (1 connection for 1 snapshot) with the source replica then starts file transferring.

Step1 & 2, which are the initial processing of the rebuilding, don't involve the internal instruction(the hole numbers) of the snapshot files. The snapshot metafiles mainly record the info like parent snapshot and the file size. Hence the metafile size is almost empty.
In step3, when transferring the actual snapshot file, Longhorn will check the fiemap and get data chunk/extent locations between holes, then compare (calculate checksum) and transfer the chunks one by one. Each time when a chunked is transferred, the rebuilding progress will be updated. As you can see here:

Longhorn won't calculate the checksum for all chunks first then start the transferring. Instead, it will process chunks sequentially.
Longhorn relies on the stat of inherent ext4 sparse files to process/figure out chunks & holes. Here is the implementation: https://github.com/longhorn/sparse-tools/blob/5c51de69e0ebc2baf20d412863d416d88a2c4384/sparse/client.go#L71 (the client in the code means the source & healthy replica, which is the sender)

To minimize the checksum cost, Longhorn can check the target replica file status first. But this implicitly means more extra communication, which will make the case worse if there are massive small extents in a snapshot file.
For the 1st step, Longhorn team should do some benchmarking based on the case you report. Hope we can fix this in the next release.

OK, so somehow I misunderstood the steps. Then I'm still not sure what took so long after finishing transfering the small prior snapshot at time="2021-04-28T18:23:21Z", and starting transfer for the next big snapshot at time="2021-04-29T02:37:23Z". I always see longhorn process just spinned at load 1 and no transfer for the big snapshot happening (until 8 hours later for this case). This always happened for this big snapshot no matter whether I reran the rebuild or recreated a new PV and recopied the data and rebuilt again.

Rebuilding and snapshot merge showed different behaviour. For snapshot merge, I saw progress at 20%, hours later at 40%, etc. For rebuilding, the UI show "Rebuilding ..." as in pic in previous comment, but there's no network transfer for this snapshot, so it's effectively at 0% for 8 hours.
The issue is at GetFiemapExtents function in the client.go. I downloaded this fiemap.c, and run this against the source snapshot. It really took 7.5h, as well as load at 1cpu, and the process is hard to kill (the same behaviour when longhorn was running during initial 8h):
File volume-snap-7432db6c-6aef-4bd7-901d-a4cd5ccea236.img has 1216060 extents: 0.00user 26854.01system 7:28:05elapsed 99%CPU (0avgtext+0avgdata 134160maxresident)k 50368inputs+0outputs (0major+33621minor)pagefaults 0swaps
Nvme disk rating with dbench is:
Random Read/Write IOPS: 260k/233k. BW: 3210MiB/s / 2675MiB/s Average Latency (usec) Read/Write: 99.53/25.30 Sequential Read/Write: 3394MiB/s / 2660MiB/s Mixed Random Read/Write IOPS: 166k/55.4k
To show it's not the structure of the extents that is an issue, I created a sparse file using your method above (but with 4k data size), alternating between data and holes, so 4k data, 4k hole, 4k data, 4k hole, etc. Then I ran the fiemap on it, it took 6h.
To show it's not an OS/disk issue, I did same on CentOS 8.2 and different ssd disk (the above ran on Ubuntu 20.04.2 and nvme disk), it took 7h.
Then I did the same on XFS filesystem. It took 1s!!
So, the ext4 based sparse file probably has to traverse all the trees of the extents while xfs seems to keep this info somewhere. I don't know if mkfs.ext4 has options so it can do like xfs. But now I retested the 1.9T production data using xfs. Initial write from outside source is 30% slower, but snapshot merging now takes less than 2h. Rebuilding doesn't have that 8h lead time. Normal read/write performance is similar to ext4. Parallel checksum for source and target snapshot will also be helpful to reduce rebuilding when target snapshot exists.
Thanks for the help with this troubleshooting, the rebuilding steps you mentioned above is really helpful.

Hi @hendra-2020 , this is an interesting finding and a brilliant troubleshooting! We're interested in hearing more about it. When you have time, would it be possible for you to reach out to me at<EMAIL_ADDRESS>? Looking forward to talking with you.

Some more comments from @hendra-2020 (if you don't mind):

The interesting thing about ext4, when you do complete rebuild of a replica, the number of extents between source and target snapshots don't change too much (1.2m extents in source/target), but in XFS sometimes they're completely different after snapshot merge or replica resync.
For latest XFS test I went like this:

Created a brand-new PV of 3TB of 2 replicas

rsync from external source; created a snapshot when 17G data is already copied

rsync completed; created another snapshot of roughly 1.9T; the number of extents in this snapshot is 2.2 millions (as opposed to 1.2 millions in ext4)

deleted 17G snapshot; longhorn seemed to start copying data from 1.9T snapshot into 17G snapshot and when finished, renamed the 17G snapshot as 1.9T snapshot name. But now the number of extents in this merged 1.9T snapshot is only 53k (it's like XFS is defragmenting the extents).

1 node is rebooted, a rebuild is running. When finished, the target snapshot now has 5k extents, instead of the original 53k; only around 10G data goes over the network.

checked using qemu-nbd to mount (read-only) the 1.9T snapshots on both nodes and run 'rsync -aHAXS --delete --dry-run'; they're identical.

I know XFS does try to merge extents when updating files, and also it uses aggregation groups so extents can be in different groups; but now that source/target snapshots show completely different extents I don't know how checksum on blocks based on holes when resyncing work in this regard.
Btw, I turned off revision counter in our setup (wasn't sure whether it would affect our SSD/NVME life).

@hendra-2020 @shuo-wu thanks for the brilliant analysis and testing.
Can you guys share the kernel versions of your worker nodes, where you did the testing/benchmarking.
I am currently working on a solution to make the extents processing faster and also increase the rebuild speed at the same time.

Here is the kernel version in which I can reproduce the issue with ext4 filesystem:
$ cat /proc/version
Linux version 4.15.0-118-generic (buildd@lgw01-amd64-039) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020

@joshimoo Thanks for working on this. I use Ubuntu 20.04.2
~# cat /proc/version Linux version 5.4.0-65-generic (buildd@lcy01-amd64-018) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021

Benchmark / Test instructions:

Create a  of size  can be small or large
Attach the  to a node in the cluster via the longhorn ui
Populate the volume via FIO job  (only needs to be done once, for comparison between old/new implementation for that block size)
For different Block sizes you need to create a new Volume and populate it with a FIO job examples above
Exec into the instance-manager-r on a different node
ps -aux to indentify the correct replica process then kill

Example process, we are always looking for the replica process, there will also be a sync-agent process
root       22852  0.0  0.9 1386484 39876 ?       Sl   Jun01   7:52 /host/var/lib/longhorn/engine-binaries/joshimoo-longhorn-engine-1948-replica-rebuild-no-chksum/longhorn replica

Before killing the process you might want to attach kubetail otherwise you can retrieve the logs later.
kubetail instance-manager-r -n longhorn-system
You can use the UI to get a rough idea of the time required for the rebuild.
For accurate numbers use the log example below:
# time 8 minutes - sender
[rebuild-perf-test-new-r-12baf306] time="2021-05-28T17:38:13Z" level=info msg="Sending file volume-snap-39f71d7b-e5de-4bd0-af0b-5c97e8835138.img to <IP_ADDRESS>:10004"
[rebuild-perf-test-new-r-12baf306] time="2021-05-28T17:38:13Z" level=info msg="source file size:<PHONE_NUMBER>0, setting up directIO: true"
[rebuild-perf-test-new-r-12baf306] time="2021-05-28T17:45:52Z" level=info msg="Done sending file volume-snap-39f71d7b-e5de-4bd0-af0b-5c97e8835138.img to <IP_ADDRESS>:10004"

# time 8 minutes - receiver
[rebuild-perf-test-new-r-c7bcb713] time="2021-05-28T17:38:13Z" level=info msg="Running ssync server for file volume-snap-39f71d7b-e5de-4bd0-af0b-5c97e8835138.img at port 10004"
time="2021-05-28T17:38:13Z" level=info msg="Creating Ssync service"
[rebuild-perf-test-new-r-c7bcb713] time="2021-05-28T17:38:13Z" level=info msg="open: receiving fileSize:<PHONE_NUMBER>0, setting up directIO: true"
[rebuild-perf-test-new-r-c7bcb713] time="2021-05-28T17:45:52Z" level=info msg="Done running ssync server for file volume-snap-39f71d7b-e5de-4bd0-af0b-5c97e8835138.img at port 10004"

For testing scenarios you can test the

worst case 4k data 4k holes
best case 128k data 4k holes (since we sent a single interval in batches of 128k bytes)
other cases 512k, 1MB, etc

For less precise benchmarks create a filesystem on top of the volume then create a bunch of test files inside of the workload filesystem. This will give you less control over the blocks/extents but would more likely represent a users use case.

You can create test cases for:
many small files (4k - 1MB)
couple of big files (128MB - 1GB)

cc @khushboo-rancher @kaxing

Verified with v1.1.2-rc1
Validation - In Progress
Cluster Config : 1 Control plane, 3 worker nodes (4 vcpus, 8GB RAM)
The performance of data with holes has been increased in v1.1.2-rc1

Data / Version
v1.1.1
v1.1.2-rc1

2 Gi (2 files of 1 Gi) with 10K holes after each block
~1 min 55 sec
~1 min 42 sec

2 Gi (2000 files of 1M) with 10K holes after each block
~2 min 30 sec
~1 min 9 sec

10 Gi ( 10000 files of 1M) with 100K holes after each block
~12 min
~ 3 min 40 sec

@khushboo-rancher can you add the

host os (ubuntu XYZ)
host filesystem where the replica data is stored. (xfs/ext4)

Also what do you mean with 10K holes after each block?
Can you describe the testing, I am guessing you are creating files inside of the workload fs, which is fine.
But just want to make sure we understand what and how it was tested :)

@joshimoo Updated the above comment https://github.com/longhorn/longhorn/issues/2507#issuecomment-866550145 with os and fs details.
I used the command like dd if=/dev/urandom of=file1.txt count=100 bs=1M conv=fsync skip=10k to introduce the holes.

Thanks for adding the additional information.

On the old kernel < 5.8 ext4 retrieval for the worst case 4k data / 4k holes will take approximately 3 minutes per 1024 extends.
So for 100mb of data the prepass of the old engine will take around 75 minutes. before the first transfer.

@joshimoo Thanks for explaining the expected behavior and testing steps in details.
The result looks good as per validation comment above https://github.com/longhorn/longhorn/issues/2507#issuecomment-866550145

Closing this issue as the performance has enhanced and we can confirm that the data after rebuild is fine from manual and integration automation tests as there is no integration test failure related to rebuild and restore because of data mismatch. https://ci.longhorn.io/job/public/job/v1.1.2/job/v1.1.2-longhorn-tests-ubuntu-amd64/38/testReport/

I did a quick test of this on Longhorn:master-head 06/25/2021 with 10 Gi, it took around 3 min 51 sec to rebuild which is enhanced from v1.1.1.


RFP - povray

Checklist

[x] The package I am requesting does not already exist on https://chocolatey.org/packages;
[x] There is no open issue for this package;
[x] The issue title starts 'RFP - ';
[x] The download URL is public and not locked behind a paywall / login;

Package Details
Software project URL: http://povray.org/
Direct download URL for the software / installer: http://www.povray.org/redirect/www.povray.org/ftp/pub/povray/Official/povwin-3.7-agpl3-setup.exe
Note:

POV-Ray for Windows requires Windows XP or later, and has been tested on versions up to and including Windows 8. We recommend having at least 100mb or so of free disk space (for temporary file storage during renders) and at least 1GB of available memory, though less is OK if you are not doing complicated scenes. Since the POV-Ray for Windows distribution package includes both the 32 and 64-bit binaries, the setup file determines which one to provide as the main program link (all versions are installed). Full documentation and sample scene files are also included.

Software summary / short description:
FOSS tool for creating stunning three-dimensional graphics.

is this the package you are requesting?
https://chocolatey.org/packages/pov-ray

Yes! sorry it is spelled differently in other places.


Update "Writing DS Plugins" Intro

Basic Plugin construction, mininal zip, ppk contents
DS plugin creation tool
external (java) plugin editors

AlanH — 01/10/2023 11:40
Not sure if it’s what you want, this url is how to create a plug-in with AIDE
https://groups.google.com/d/msg/androidscript/UdFza30tYEY/ELBOIbFTBQAJ
Jumar — 02/10/2023 03:07
If you are creating plugins as pure javascript plugin you can use the Web IDE to code. But if you are creating a native/java plugin for DS you should use AIDE or Android Studio to code which you can build to APK too using AIDE or Android Studio.
For javascript plugin you can compile it to .ppk or just zip the folder containing the "MyPlugin.js" or "MyPlugin.inc", "MyPlugin.html" and "Version.txt".
To install, copy the .zip file or .ppk file into the Plugins folder of DroidScript and restart DroidScript app. You should be able to see a popup saying "Plugin installed" upon opening DroidScript.
d_a_v_e — 11/10/2023 02:58
@Eric Zollman If you are creating a pure JavaScript plugin then you can use the ide.MakePlugin method like this -
If you create a fresh project in DS and include the ide.MakePlugin call at the start, it will generate the necessary javascript and html files required for a simple DS plugin. You just need to reload your project to see the new generated files.  For example the code below will add 3 new files to your project - DemoPlugin.js, DemoPlugin.html and DemoPlugin.png
function OnStart()
{
  //Package this project into a plugin and install it.
  ide.MakePlugin( "DemoPlugin" )

   //Test the plugin functionality.
   app.LoadPlugin( "DemoPlugin" )
   plg = app.CreateDemoPlugin()
   alert( plg.Test( "World" ) )
}

You can examine those files to see the function pattern/style you need to use to add new features to your plugin.  These files will be re-generated every time you run the project and a DemoPlugin.ppk will also be generated every time (in the DroidScript/PPKs folder) which contains the packaged plugin, ready for deployment or uploading to the DS Store.
If you want to make a Java plugin, I recommend using Android Studio and this template - https://github.com/DroidScript/Plugin-Java


Documenting use of shortmess for search count virtual text

Describe the bug
Virtual text of search count doesn't appear when shortmess is set to certain flags.
I had my shortmess set to "filnxtToOFIAS".
When commenting out this line in my config, the virtual text appears as expected.
Which version of Neovim are you using?
Nightly 0.9
To Reproduce
Steps to reproduce the behavior:

Set shortmess to "filnxtToOFIAS"
Make a search
Notice search count doesn't appear as virtual text.

Expected Behavior

Show a warning
Or show configuration problem in checkhealth.

Good point. Added a health check for this.


Update jfact to 5.0.3

Updates net.sourceforge.owlapi:jfact from 4.0.4 to 5.0.3.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Need to stick with OWL API v4 versions of JFact.


@swc/css supports AST API

Describe the feature
Background
I'm the author of the postcss-media-minmax plugin, and as the CSS @media syntax gets more and more complex, we've run into some maintenance bottlenecks. I'm going to try to refactor the new plugin using swc's AST parsing capabilities.
https://github.com/swc-project/bindings/tree/main/crates/css_node
Is it possible to support some AST APIs for my scenario?
Babel plugin or link to the feature description
No response
Additional context
No response

Are you asking some rust-based APIs, like wasm plugin?

@kdy1 I expect to use it in Node.js. Currently I have a wasm binding. It would be nice if it was officially supported directly.

I'm not asking about your usecase. I 'm asking if you are requesting js-side api or rust-side api like ES

@kdy1 I need something similar to minifySync() in @swc/css. I'm not sure if this is JS-side API or Rust-side API.


Bug: Docker - migration error

What happened?
Since the latest update of the trunk image, my Zipline container can't start. It seems like it's a database-migration error.
Here is my docker-compose
zipline:
    container_name: zipline
    image: ghcr.io/diced/zipline:trunk
    networks:
      - web
      - database
    restart: always
    environment:
      - CORE_HTTPS=false
      - 'CORE_SECRET=******************'
      - CORE_HOST=<IP_ADDRESS>
      - CORE_PORT=3000
      - CORE_DATABASE_URL=postgresql://postgres:***************@postgres/zipline
      - CORE_LOGGER=true
      - DATASOURCE_TYPE=local
      - DATASOURCE_DIRECTORY=./uploads
      - UPLOADER_ROUTE=/u
      - UPLOADER_EMBED_ROUTE=/a
      - UPLOADER_LENGTH=6
      - UPLOADER_ADMIN_LIMIT=5245000000
      - UPLOADER_USER_LIMIT=1049000000
      - UPLOADER_DISABLED_EXTS=
      - URLS_ROUTE=/go
      - URLS_LENGTH=6
    volumes:
      - '/home/*****/docker/useful-apps/zipline/uploads:/zipline/uploads'
      - '/home/*****/docker/useful-apps/zipline/public:/zipline/public'

Please let me know if you need any additional information !
Version
upstream (ghcr.io/diced/zipline:trunk)
What browser(s) are you seeing the problem on?
No response
Zipline Logs
Node.js v19.4.0 |
-- | --
  |   | } |
  |   | batchRequestIdx: undefined |
  |   | meta: { column: 'Invite.expiresAt' }, |
  |   | clientVersion: '4.8.1', |
  |   | code: 'P2022', |
  |   | at start (/zipline/src/server/index.ts:107:12) { |
  |   | at Object.clearInvites (/zipline/src/server/index.ts:122:7) |
  |   | at async PrismaClient._request (/zipline/node_modules/@prisma/client/runtime/index.js:36082:16) |
  |   | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) |
  |   | at RequestHandler.request (/zipline/node_modules/@prisma/client/runtime/index.js:34991:12) |
  |   | at RequestHandler.handleAndLogRequestError (/zipline/node_modules/@prisma/client/runtime/index.js:34996:12) |
  |   | at RequestHandler.handleRequestError (/zipline/node_modules/@prisma/client/runtime/index.js:35024:13) |
  |   | The column `Invite.expiresAt` does not exist in the current database. |
  |   | Invalid `prisma.invite.deleteMany()` invocation: |
  |   | PrismaClientKnownRequestError: |
  |   | ^ |
  |   | throw new PrismaClientKnownRequestError(message, { |
  |   | /zipline/node_modules/@prisma/client/runtime/index.js:35024 |
  |   | 2023-01-16 10:30:56,178 AM info  [server] started production<EMAIL_ADDRESS>server |
  |   | 2023-01-16 10:30:56,178 AM info  [server] listening on <IP_ADDRESS>:3000 |
  |   | 2023-01-16 10:30:55,176 AM info  [datasource] using Local(./uploads) datasource

### Browser Logs

_No response_

### Additional Info

_No response_

Are there any errors before this?

No, this is only looping on this error. Do you want me to enable debug mode ?

Could you run docker-compose exec postgres psql -U postgres -d postgres -tc "select json_agg(json_build_object('id', id, 'checksum', checksum, 'finished_at', finished_at, 'migration_name', migration_name, 'logs', logs, 'rolled_back_at', rolled_back_at, 'started_at', started_at, 'applied_steps_count', applied_steps_count )) from _prisma_migrations;" and post the result here? The output is very big so if you would like to upload it a text uploader somewhere, that's fine.

The command doesn't work, but there is a .csv export of the _prisma_migrations table
_prisma_migrations.csv

Hey there! There's been a recent commit addressing potential problems with migrations, could you try pulling the latest changes and seeing if migration works? Or were you able to resolve this issue?

No response from the opener, closing as stale.


sierra-gtk-theme: init at 2018-09-14

Motivation for this change
Add sierra-gtk-theme. Sierra is a Mac OSX like theme for GTK 3, GTK 2 and Gnome-Shell which supports GTK 3 and GTK 2 based desktop environments like Gnome, Pantheon, XFCE, Mate, etc.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

Success on x86_64-linux (full log)
Attempted: sierra-gtk-theme
Partial log (click to expand)
Installing '/nix/store/aanmp1nw8nb6lm78h4m49idzys1a9nqi-sierra-gtk-theme-2018-09-14/share/themes/Sierra-compact-dark-solid-thin'...

Done.
glibPreFixupPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/aanmp1nw8nb6lm78h4m49idzys1a9nqi-sierra-gtk-theme-2018-09-14
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/aanmp1nw8nb6lm78h4m49idzys1a9nqi-sierra-gtk-theme-2018-09-14
checking for references to /build in /nix/store/aanmp1nw8nb6lm78h4m49idzys1a9nqi-sierra-gtk-theme-2018-09-14...
/nix/store/aanmp1nw8nb6lm78h4m49idzys1a9nqi-sierra-gtk-theme-2018-09-14

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: sierra-gtk-theme
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on aarch64-linux (full log)
Attempted: sierra-gtk-theme
Partial log (click to expand)
Installing '/nix/store/ysp9707iqkz38awblla7jdzsciwypsxj-sierra-gtk-theme-2018-09-14/share/themes/Sierra-compact-dark-solid-thin'...

Done.
glibPreFixupPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ysp9707iqkz38awblla7jdzsciwypsxj-sierra-gtk-theme-2018-09-14
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/ysp9707iqkz38awblla7jdzsciwypsxj-sierra-gtk-theme-2018-09-14
checking for references to /build in /nix/store/ysp9707iqkz38awblla7jdzsciwypsxj-sierra-gtk-theme-2018-09-14...
/nix/store/ysp9707iqkz38awblla7jdzsciwypsxj-sierra-gtk-theme-2018-09-14


Fix CUDA-enabled HYPRE finalization order issue

Resolves #4503.
This ensures that Hypre is finalized when mfem::Device goes out of scope in the main function.

PR
Author
Editor
Reviewers
Assignment
Approval
Merge

#4504
@helloworld922
@tzanio
@pazner + @v-dobrev
9/16/24
⌛due 9/30/24
⌛due 10/7/24

PR Checklist

[ ] Code builds.
[ ] Code passes make style.
[ ] Update CHANGELOG:

[ ] Is this a new feature users need to be aware of? New or updated example or miniapp?
[ ] Does it make sense to create a new section in the CHANGELOG to group with other related features?

[ ] Update INSTALL:

[ ] Had a new optional library been added? If so, what range of versions of this library are required? (Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!)
[ ] Have the version ranges for any required or optional libraries changed?
[ ] Does make or cmake have a new target?
[ ] Did the requirements or the installation process change? (rare)

[ ] Update continuous integration server configurations if necessary (e.g. with new version requirements for each of MFEM's dependencies)

[ ] .github
[ ] .appveyor.yml

[ ] Update .gitignore:

[ ] Check if make distclean; git status shows any files that were generated from the source by the project (not an IDE) but we don't want to track in the repository.
[ ] Add new patterns (just for the new files above) and re-run the above test.

[ ] New examples:

[ ] All sample runs at the top of the example source file work.
[ ] Update examples/makefile:

[ ] Add the example code to the appropriate SEQ_EXAMPLES and PAR_EXAMPLES variables.
[ ] Add any files generated by it to the clean target.
[ ] Add the example binary and any files generated by it to the top-level .gitignore file.

[ ] Update examples/CMakeLists.txt:

[ ] Add the example code to the ALL_EXE_SRCS variable.
[ ] Make sure THIS_TEST_OPTIONS is set correctly for the new example.

[ ] List the new example in doc/CodeDocumentation.dox.
[ ] If new examples directory (e.g.examples/pumi), list it in doc/CodeDocumentation.conf.in
[ ] Companion pull request for documentation in mfem/web repo:

[ ] Update or add example-specific documentation, see e.g. the src/examples.md.
[ ] Add the description, labels and screenshots in src/examples.md and src/img.
[ ] In examples.md, list the example under the appropriate categories, add new categories if necessary.
[ ] Add a short description of the example in the "Extensive Examples" section of features.md.

[ ] New miniapps:

[ ] All sample runs at the top of the miniapp source file work.
[ ] Update top-level makefile and makefile in corresponding miniapp directory.
[ ] Add the miniapp binary and any files generated by it to the top-level .gitignore file.
[ ] Update CMake build system:

[ ] Update the CMakeLists.txt file in the miniapps directory, if the new miniapp is in a new directory.
[ ] Add/update the CMakeLists.txt file in the new miniapp directory.
[ ] Consider adding a new test for the new miniapp.

[ ] List the new miniapp in doc/CodeDocumentation.dox
[ ] If new miniapps directory (e.g.miniapps/nurbs), add it to MINIAPP_SUBDIRS in the makefile.
[ ] If new miniapps directory (e.g.miniapps/nurbs), list it in doc/CodeDocumentation.conf.in
[ ] Companion pull request for documentation in mfem/web repo:

[ ] Update or add miniapp-specific documentation, see e.g. the src/meshing.md and src/electromagnetics.md files.
[ ] Add the description, labels and screenshots in src/examples.md and src/img.
[ ] The miniapps go at the end of the page, and are usually listed only under a specific "Application (PDE)" category.
[ ] Add a short description of the miniapp in the "Extensive Examples" section of features.md.

[ ] New capability:

[ ] All new public, protected, and private classes, methods, data members, and functions have full Doxygen-style documentation in source comments. Documentation should include descriptions of member data, function arguments and return values, template parameters, and prerequisites for calling new functions.
[ ] Pointer arguments and return values must specify whether ownership is being transferred or lent with the call.
[ ] Any new functions should include descriptions of their intended use e.g. for internal use only, user-facing, etc., along with references to example code whenever possible/appropriate.
[ ] Consider adding new sample runs in existing examples to highlight the new capability.
[ ] Consider saving cool simulation pictures with the new capability in the Confluence gallery (LLNL only) or submitting them, via pull request, to the gallery section of the mfem/web repo.
[ ] If this is a major new feature, consider mentioning it in the short summary inside README (rare).
[ ] List major new classes in doc/CodeDocumentation.dox (rare).

[ ] Update this checklist, if the new pull request affects it.
[ ] Run make unittest to make sure all unit tests pass.
[ ] Run the tests in tests/scripts.
[ ] (LLNL only) After merging:

[ ] Update internal tests to include the new features.

@helloworld922, is this ready-for-review?

Yes, it should be ready for review.

This PR is now under review (see the table in the PR description). To help with the review process, please do not force push to the branch.

@v-dobrev this change will work in all cases for hypre version < 2.19.0 or >= 2.29.0.
It's inbetween where I can't 100% say it will work or fail. I did try manually calling different permutations of init/finalize both using mfem's singleton as well as hypre's native API and it at least appears to run. However, when I dig into hypre's source it looks like there are some state changes which might be unexpected for the rest of hypre with repeat calls to HYPRE_Init or HYPRE_Finalize (in particular when configuring hypre with umpire).
I think the safest course of action is to add a flag in mfem::Hypre which tracks if Hypre was initialized by mfem or not, then move the call to HYPRE_Init out of the constructor to mfem::Hypre and call it in mfem::Hypre::Init if needed. Similarly, in mfem::Hypre::Finalize we can finalize only if hypre was initialized by mfem and hasn't been finalized by mfem yet.
Unfortunately, this does not handle the case where the user calls mfem::Hypre::Init then calls HYPRE_Finalize before mfem::Hypre::Finalize. The only way to handle this case for versions between the above listed ones is to inspect the internal hypre state, which while possible I would prefer to avoid and consider this case to be user error/problem in HYPRE, just like trying to call HYPRE_Finalize after mfem::Hypre::Finalize.

Good catch, I made the changes minus the logging to std::cout.

ping: @v-dobrev

I think I understand we run into this issue. The issue is the order in which the exit hooks for mfem::Hypre and CUDA are registered. We first call mfem::Hypre::Instance (which registers an exit handler to destroy the singleton mfem::Hypre object defined inside the function mfem::Hypre::Instance); after that we call HYPRE_Init which makes the first call to CUDA and that first call probably registers an exit handler to finalize CUDA. The exit handlers are executed in reverse order, so CUDA is finalized before mfem::Hypre::~Hypre is called which calls HYPRE_Finalize which calls CUDA functions and they generate errors because CUDA is already finalized.
This leads me to believe that the clean way to fix the issue is not to call mfem::Hypre::Finalize from the mfem::Device destructor but to simply call HYPRE_Init before we call mfem::Hypre::Instance for the first time.
What do you think?

I tried moving the call to mfem::Hypre::Instance after the calls to HYPRE_Initialize and mfem::Hypre::SetDefaultOptions but that did not resolve the issue, so there must be something else going on.

Another observation: if I move the call to mfem::Hypre::Instance to the end of Hypre::InitDevice then I don't see the issue. Maybe HYPRE_Initialize and mfem::Hypre::SetDefaultOptions do not actually make any CUDA calls.

Personally I would prefer to ensure that all finalization actions are performed inside the body of the main function, which right now is done by the local Device object's destructor which is reclaimed before main returns. Ideally we would have some other mfem RAII object which manages all singleton-like states like MPI, HYPRE, and device-specific things instead of using multiple static singletons, but I think that would be a big breaking change so the next best thing is to use the Device RAII object for this purpose.
Officially from Nvidia they say that trying to call CUDA calls before/after the main function is undefined behavior, and HYPRE_Finalize does make CUDA calls.
https://forums.developer.nvidia.com/t/cuda-run-time-library-unload/51956

I don't think HYPRE_Initialize makes any CUDA calls itself (maybe with the memory tracker option enabled it does?), but HYPRE_DeviceInitialize does and it is expected that HYPRE_Finalize will also cleanup objects created by HYPRE_DeviceInitialize.

Okay, I'm not sure I see a much better way to handle this issue than what's currently done here -- calling mfem::Hypre::Finalize from the mfem::Device destructor.
There's one thing, however, I think we should change: Hypre::Finalize now calls Hypre::Instance (just to check its state) and that call can actually be the first call to Hypre::Instance (e.g. if the user does not use Hypre::Init for HYPRE initialization). This redundant instantiation/destruction can be avoided by simply making Hypre::state a static variable and removing the call to Hypre::Instance from Hypre::Finalize.

The default constructor of Hypre no longer calls Hypre::Initialize, so even if mfem::Device's destructor is the first to call Hypre::Finalize which creates the instance, it will just be effectively a no-op since Finalize() will detect that the default state is State::UNINITIALIZED for HYPRE < 2.29.0, and will use HYPRE's internal state management for HYPRE >= 2.29.0.
I suppose it could be unexpected in someone's code if doesn't they don't ever call Hypre::Initialize and expect HYPRE to still be functional when mfem::Device is destroyed, I can change it back so it will only finalize if the internal state is INITIALIZED.

I'm testing the PR with HYPRE v2.29.0 (built with CUDA v11.1.1) and I see crashes in ex0p at the end of the run.
I'm investigating.

I think I know why it's crashing: ex0p doesn't ever create a device object, so Hypre::InitDevice is never called, which in turn it looks like HYPRE then defaults to solving on the GPU if built with GPU support.
Then HYPRE will internally call HYPRE_DeviceInitialize() on the first API call requesting GPU compute/data transfer.
However, since we don't have a Device object destruction at main end, Hypre::Finalize is called during static variable cleanup, which is the original bug this PR is trying to fix.
I think one possible fix is during Hypre::Init also call HYPRE_SetMemoryLocation(HYPRE_MEMORY_HOST) and HYPRE_SetExecutionPolicy(HYPRE_EXEC_HOST) (if configure_runtime_policy_from_mfem == true), then this can be overridden when the Device object is created and calles Hypre::InitDevice.
This does leave a hole if someone has configure_runtime_policy_from_mfem == false, never creates a Device object, and expects mfem to finalize hypre for them via Hypre::Finalize.

There is also a question of what versions of hypre we want to support, since these two functions only exist in HYPRE >= 2.26.0, and from the comments in Hypre::InitDevice there are potential problems for HYPRE < 2.31.0 initializing vendor libraries.

Here's a patch (for commit 9bee2fca) that works for me:
diff --git a/general/device.cpp b/general/device.cpp
index 2d600955f..3ed11e916 100644
--- a/general/device.cpp
+++ b/general/device.cpp
@@ -258,7 +258,15 @@ void Device::Configure(const std::string &device, const int device_id)
    destroy_mm = true;

 #ifdef MFEM_USE_MPI
-   Hypre::InitDevice();
+#if defined(HYPRE_USING_GPU) && (MFEM_HYPRE_VERSION >= 23100)
+   // Skip the call to Hypre::InitDevice() if HYPRE is not initialized, e.g.
+   // * if running a serial code
+   // * if running with the environment variable MFEM_DEVICE set.
+   if (HYPRE_Initialized())
+   {
+      Hypre::InitDevice();
+   }
+#endif
 #endif
 }

diff --git a/linalg/hypre.cpp b/linalg/hypre.cpp
index 39c466a25..3ffc1ada8 100644
--- a/linalg/hypre.cpp
+++ b/linalg/hypre.cpp
@@ -28,16 +28,23 @@ namespace mfem
 {

 bool Hypre::configure_runtime_policy_from_mfem = true;
+Hypre::State Hypre::state = Hypre::State::UNINITIALIZED;

 void Hypre::Init()
 {
-   Hypre &hypre = Instance();
-   if (hypre.state != State::INITIALIZED)
+   if (state != State::INITIALIZED)
    {
+#if MFEM_HYPRE_VERSION >= 21900
       HYPRE_Init();
-      hypre.SetDefaultOptions();
+#endif
+      SetDefaultOptions();
+      // Apply the setting of 'configure_runtime_policy_from_mfem' according to
+      // the current configuration of the mfem::Device (HYPRE >= 2.31.0):
+      InitDevice();
+      // Create the singleton Hypre object AFTER initializing HYPRE:
+      Instance();
    }
-   hypre.state = State::INITIALIZED;
+   state = State::INITIALIZED;
 }

 void Hypre::InitDevice()
@@ -48,6 +55,8 @@ void Hypre::InitDevice()
 #if defined(HYPRE_USING_GPU) && (MFEM_HYPRE_VERSION >= 23100)
    if (configure_runtime_policy_from_mfem)
    {
+      MFEM_VERIFY(HYPRE_Initialized(), "HYPRE must be initialized before"
+                  " calling Hypre::InitDevice()");
       if (Device::Allows(Backend::DEVICE_MASK & ~Backend::DEBUG_DEVICE))
       {
          HYPRE_SetMemoryLocation(HYPRE_MEMORY_DEVICE);
@@ -65,14 +74,13 @@ void Hypre::InitDevice()

 void Hypre::Finalize()
 {
-   Hypre &hypre = Instance();
-   if (hypre.state != State::UNINITIALIZED)
+   if (state != State::UNINITIALIZED)
    {
 #if MFEM_HYPRE_VERSION >= 21900
       HYPRE_Finalize();
 #endif
    }
-   hypre.state = State::UNINITIALIZED;
+   state = State::UNINITIALIZED;
 }

 void Hypre::SetDefaultOptions()
diff --git a/linalg/hypre.hpp b/linalg/hypre.hpp
index 48df67bde..1435c36b2 100644
--- a/linalg/hypre.hpp
+++ b/linalg/hypre.hpp
@@ -121,7 +121,7 @@ private:
    ~Hypre() { Finalize(); }

    /// Set the default hypre global options (mostly GPU-relevant).
-   void SetDefaultOptions();
+   static void SetDefaultOptions();

    /// Create and return the Hypre singleton object.
    static Hypre &Instance()
@@ -133,7 +133,7 @@ private:
    enum class State { UNINITIALIZED, INITIALIZED };

    /// Tracks whether Hypre was initialized or finalized by this class.
-   State state = State::UNINITIALIZED;
+   static State state;
 };

I committed the patch and so far everything appears to be working fine for me with HYPRE 2.31.0.

Merged in next for testing...


[Model Analysis] Based on generated model op tests, create summary for all ops

This summary should contain an overview of where specific op is failing. More precisely, in which part of the compiler stack, and why.
Purpose of this issue is to get:

Single file (best to be markdown) which contains list of all ops and their unique configurations, with exact details:

Where op is failing (TVM, FFE, MLIR, TTNN)
Why op is failing (error message)

Part of this issue:

https://github.com/tenstorrent/tt-forge-fe/issues/661


Exception during Job handling will never log an error

Hello,
The following implementation https://github.com/stackkit/laravel-google-cloud-tasks-queue/blob/52452fc2648eb2ce9413afedc7770812270e2bef/src/TaskHandler.php#L44C9-L44C15 will never throw an Exception, correctly.
The down side of this is that a Stack trace or simply a log of the error will never be printed.
Following this pattern, any error logging method (as Sentry or Mezmo) will never catch the error.
public function handle(?string $task = null): void
    {
        $task = IncomingTask::fromJson($task ?: request()->getContent());

        event(new TaskIncoming($task));

        if ($task->isInvalid()) {
            abort(422, 'Invalid task payload');
        }

        if (! CloudTasksApi::exists($task->fullyQualifiedTaskName())) {
            abort(404);
        }

        $config = config('queue.connections.'.$task->connection());

        $this->config = is_array($config) ? $config : [];

        // We want to catch any errors so we have more fine-grained control over
        // how tasks are retried. Cloud Tasks will retry the task if a 5xx status
        // is returned. Because we manually manage retries by releaseing jobs,
        // we never want to return a 5xx status as that will result in duplicate
        // job attempts.
        rescue(fn () => $this->run($task), report: false);
    }

Solution: just ignore report field or set it as true

Thanks! I think I got report confused with actually throwing the exception 🙃, but I've released a v4.2.1 with a fix!


DEC-630: Migrate enable_browse

-Created migrations to add enable_browse text column to Collections and copy existing data from the exhibits. enable_browse into collections. enable_browse
-Changed all references to exhibits. enable_browse to collections. enable_browse

Make new PR against DEC-376-Merge-collection-and-exhibit-models


foreman vs pure python command

i have a strange issue trying to load new plugins.
seems if i do:
 foreman run python manage.py syncdb --migrate

i do not have the same behavior as :
 source env_vars && python manage.py syncdb --migrate

env_vars contains same as conf/production.env, except the fact it has 'export' at first of every variables like:
'export CABOT_PLUGINS_ENABLED=cabot_alert_hipchat==1.7.0,cabot_alert_twilio==1.6.1,cabot_alert_email==1.3.1,cabot_alert_ovh==1.0.0b1" '
using foreman, i can use the 'cabot_alert_ovh', but using the second one, the plugin isn't available.
is there anything special that foreman does ?

seems like my nginx reverse proxy was the wrong guy.
i'll check deeper


result of .concat is not a stopified array

Needs to be fixed fast.

Fixed
https://github.com/plasma-umass/Stopify/pull/421/commits/016f93f6b60ee671978eacfc104ee59db5b8fe25


Order types in alphabetical order

Order types in alphabetical order

Closing this because I just noticed that I build off the wrong branch (we moved from master to main). Making another one...


All Godot windows completely blank after reopening laptop lid.

Tested versions

Reproducible in v4.2.2.stable.arch_linux

System information
Godot v4.2.2.stable unknown - Arch Linux #1 SMP PREEMPT_DYNAMIC Fri, 28 Jun 2024 04:32:50 +0000 - Wayland - GLES3 (Compatibility) - zink Vulkan 1.3(NVIDIA GeForce RTX 3050 Laptop GPU (NVIDIA_PROPRIETARY)) () - 11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz (12 Threads)
Issue description
Only effects Godot windows. Has happened to the Project Manager, Editor, and running projects.
When a game is running the log file ends with-
ERROR: 1 RID allocations of type 'N5GLES37TextureE' were leaked at exit.
USER ERROR: Texture with GL ID of 13: leaked 4 bytes.
   at: ~Utilities (drivers/gles3/storage/utilities.cpp:79)

-but I'm uncertain if it's related.
If there is any better information I could give, please let me know.
Steps to reproduce

Open Godot
Close laptop lid
Reopen laptop
Blank windows

Minimal reproduction project (MRP)
Doesn't even need a project, it happens outside of them as well.

Thanks for the report! Consolidating in https://github.com/godotengine/godot/issues/52738. This is a NVIDIA issue we can't fix on our end without large sweeping changes. All other applications using Vulkan on Linux run into the same issue unless they can restart their rendering context entirely in a clean manner (preferably without losing any data).
You may try following the advice on this page at your own risk: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Preserve_video_memory_after_suspend


Added , and / as separators for leader skill searching.

what am i even doing
feat regex
This assumes forward slash and quote will never be used as part of card search.

closing on author's request, will re-open


add organization to the data schema for hibp

Added the organization to the data schema to facilitate querying for report generation

🗣 Description
This PR only adds one field to the hibp_exposed_credentials table and the hibp view
💭 Motivation and context
Previously there was no way to easily query all credential breaches for a specific organization. adding this field to the table and view will allow us to pull all values for a specific organization in a specified time frame.
🧪 Testing
Should pass with the same tests as before
✅ Checklist

[X ] This PR has an informative and human-readable title.
[X ] Changes are limited to a single goal - eschew scope creep!
[X ] All future TODOs are captured in issues, which are referenced
in code comments.
[ ] All relevant type-of-change labels have been added.
[X ] I have read the CONTRIBUTING document.
[ X] These code changes follow cisagov code standards.
[X ] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[ X] Tests have been added and/or modified to cover the changes in this PR.
[ ] All new and existing tests pass.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 76.987%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
174

Relevant Lines:
217

💛  - Coveralls


uid_entrypoint.sh renamed to entrypoint.sh, breaks using v2.1.0

Describe the bug
argocd-repo-server-6dc987d959-q75xr   0/1     CreateContainerError

because of hardcoded uid_entrypoint.sh which was renamed here

https://github.com/argoproj/argo-cd/commit/4ae1d876fd6c66ae665db996a0d2e9c43fdac7e6
https://github.com/argoproj/argo-cd/blob/master/entrypoint.sh
To Reproduce
install version 2.1.0 argocd

This has been fixed in v2.1.1 of Argo CD, there now is a symlink uid_entrypoint.sh -> entrypoint.sh. Since upstream operator issue is closed, can we close this as well?

ah yes.


i18n(fr): Update layouts.mdx translation

What kind of changes does this PR include?

updated content
Translated content

Sorry, The job is already done #4687


Some questions about the code and equations in paper

An excellent work! I've several questions.

Why does the par appear in out_grad, while par_pos in out_data? https://github.com/Megvii-BaseDetection/TreeFilter-Torch/blob/4eed6b61f0f8688618a8563b98a1f21a5c1f9440/furnace/kernels/lib_tree_filter/src/refine/refine.cu#L184
delta S(Ei,j)/delta wi,j  in eq.9 seems to disappear in the code. https://github.com/Megvii-BaseDetection/TreeFilter-Torch/blob/4eed6b61f0f8688618a8563b98a1f21a5c1f9440/furnace/kernels/lib_tree_filter/src/refine/refine.cu#L367
The paper says "uniformly sample a vertex as the root and perform breadth-first sorting (BFS) algorithm to obtain the topological order of tree G“. But it seems that the code always start from the index 0.
https://github.com/Megvii-BaseDetection/TreeFilter-Torch/blob/4eed6b61f0f8688618a8563b98a1f21a5c1f9440/furnace/kernels/lib_tree_filter/src/bfs/bfs.cu#L79

Thanks for your interest in our work.

Because out_data is ordered as the input feature, but out_grad is ordered in the breadth-first-search (BFS) manner.
The "weight" in "refine.cu" is already calculated by S(Ei, j), whose gradient is obtained by using the native PyTorch code. Please refer to the distance_func https://github.com/Megvii-BaseDetection/TreeFilter-Torch/blob/4eed6b61f0f8688618a8563b98a1f21a5c1f9440/furnace/kernels/lib_tree_filter/modules/tree_filter.py#L29
In the BFS algorithm, different vertexes as the root, the algorithm will output a consistent topology. For convenience, we choose the first vertex as the root.

Got It! Thank you very much!


Changes for clap defaults

Fixed issue with default values for clap command-line parameters (#15).
Added a new setting use_defaults to cmd parser. Use defaults from clap arguments. Default is false. IMPORTANT: Once that setting will be set to true then all defined command-line parameters will have values which will override values with same key(s) from parsers which was added to ConfigBuilder after this parser.

Thanks!


Questions about the developer manual

Hi, may I ask why I can't find a developer manual about Cesium Native? It's hard to fully understand it only from the source code.

The individual classes in cesium-native are reasonably well documented. I studied cesium-unreal hard in order to write vsgCs.

@kring

Since we have #832 to actually publish our Doxygen-generated documentation, I'd like to repurpose this issue for the actual organization.
Our Doxygen docs will be on the website regardless, but it will take more resources to actually help users understand what's going on. For instance, see #982 , #981 , #980.
We should figure out the best place for these resources to live. Should we our resources:

Be markdown files stored in the cesium-native repo?
Live on the website somewhere?
Both?

It could even be worth taking advantage of our learning platform at https://cesium.com/learn/ to create an entirely separate category for Cesium Native. The "tutorials" could walk through how a simple integration with Cesium Native could look like, and host other written documentation for concepts that users would need to know.

I'd like to advocate for keeping the cesium-native docs in markdown and in the same repo as the code. Because:

cesium-native is inherently a developer product. The polish of the web site isn't as essential.
Managing content in the web site CMS adds a ton of friction for us.
Keeping docs in the repo makes versioning basically free. On the web site, it's a nightmare.

Closing this issue. We have decided to keep put these docs in doxygen, and these docs will be published every release to https://cesium.com/learn/cesium-native/ref-doc/.


[BUG] - cloud provider initialized without required inputs

Describe the bug
While working on #2056, I discovered that when a provider is given but the corresponding provider field is not defined, we are trying to initialize provider objects in https://github.com/nebari-dev/nebari/blob/16866933dcbe9cd1ca08247f0cb245c94f11c6a7/src/_nebari/stages/infrastructure/init.py#L527-L546
It works for local and existing providers because they don't require user inputs. But for all cloud providers, user inputs are mandatory so the current implementation doesn't work.
We haven't seen this before due to the lack of unit test. And, when user run nebari initialize, all the required fields for cloud provider got rendered so it won't reach the object initialization code in the root_validator.
The the rendering is here:
https://github.com/nebari-dev/nebari/blob/16866933dcbe9cd1ca08247f0cb245c94f11c6a7/src/_nebari/initialize.py#L112-L176
Expected behavior
We need to align the render and initialization code to clarify where and how the rendering/information filling is done.
OS and architecture in which you are running Nebari
macOS M1
How to Reproduce the problem?
@pytest.fixture
def config_schema():
    return nebari_plugin_manager.config_schema

def test_provider_validation(config_schema):
    config_dict = {
        "project_name": "test",
        "provider": "gcp",
    }
    config = config_schema(**config_dict)

After #2056, the initialization of cloud provider object is skipped so the code snippet won't raises an error.
Command output
No response
Versions and dependencies used.
No response
Compute environment
None
Integrations
No response
Anything else?
No response

Thanks for digging into this @fangchenli!
A summary of this would be: we need to decide how we initialize the nebari-config.yaml, using the code in initialize.py or relying on the pydantic models within the stages themselves.

The affected code is gone.  I'm going to close this since it may no longer be relevant.


Fluent progress style

Have you considered making the fluent style look more like the current "Progress" style than the legacy? That would be something familiar to most users.

I considered it, but I ended up basing it off the Vista style because it already looked pretty good and it took up little space in comparison to the Progress style. I may make an alternate style based off it, if it's something that some people might prefer.


⚠ Bofrost.de has degraded performance

In 38bc41a, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2547 ms

Resolved: Bofrost.de performance has improved in de69058.


merge-friendly encryption

hi, onunu.
Currently, leml changes all encrypted value when I change some value with leml:edit.
But I need some merge-friendly encryption.
for example, I have some encrypted key-value like this.
---
development:
  aaa: Q3Jpc0Z2cWdycmlTSXk4V3FWcHVtUT09LS1RNHBkdlowOUJoRE4ranp5Sm0ydnRBPT0=--9477269966b8680370c4f49ca06bb2d9b33eeddc
  bbb: RmRDb2pPNTh0UGNjeVFLMWQvRHkrQT09LS04RG55dFdybmhna0Y2THhiSmZDeHdnPT0=--b13587bc82b8f527f59fe906ac24ad661800bc73

and then I change a value of "aaa". that re-encrypted all value like this.
development:
  aaa: b2tHaVBvdG1qRE8xK0JwdjJXVUo1Zz09LS1TT0d3TFl0ODg3NnpoMGIyL2tPOXpnPT0=--97645be11032602c6f1e7b43609ce1c256112456
  bbb: OVlld2pDOE9yNnBtblRXSzhRNkQ0UT09LS1GQ2RqMjZvMCtVUjFLNndkeFNMV2dRPT0=--79ffef2e01464d919d9543e0de17f0512cda0026

So, I change value of "aaa" but "bbb" was changed too.
That is so hard to merge, check diff and so on.
I need same encrypted "bbb" value like this.
---
development:
  aaa: cTJ1UDc2NzJVZ0tYTHUxekRuenBGdz09LS1ZT1lESlRncU1POXdlZUlxUU5taHBRPT0=--1e05f9795704696a5a0a443c5ffea72b91b2c234
  bbb: RmRDb2pPNTh0UGNjeVFLMWQvRHkrQT09LS04RG55dFdybmhna0Y2THhiSmZDeHdnPT0=--b13587bc82b8f527f59fe906ac24ad661800bc73

Here is changing only "aaa" value and exactly same encrypted "bbb" value before edit.
If you like it, merge this PR please.
Thanks.

Hi, kyohsuke.
Thank you for your suggestion and pull-req.
That is exactly same I thought ( but I couldn't do it well..)
Also i thank you for adding test. It's awesome help for me.
well, I' ll merge this pull-req and release new version.


bits_and_bytes not support windows, how to fine-tune whisper on windows?

I find that peft + lora + dnd cannot train on windows, even worse, can not run on windows. So how to solve the problem?
If there a method support windows or choose other 8 bit train?

Solved by this pip But the peft+lora reference is still slow, so it is not that useful

https://huggingface.co/docs/bitsandbytes/main/en/installation?OS+system=Windows

starting v0.43 there's the official wheel for windows


Feature request: support file relocation/rename

I would like vendir to strip/rename paths
- path: vendor/k8s
  contents:
  - path: eduk8s
    git:
      url: https://github.com/eduk8s/eduk8s
      ref: <IP_ADDRESS>
    includePaths:
    - resources/crds-v1/**/*
    - resources/operator/**/*

and this ends up creating:
vendor/k8s/eduk8s/resources/crds-v1
vendor/k8s/eduk8s/resources/operator

but I would want:
vendor/k8s/eduk8s/resources/

everything just in this dir.
The use case is:
I’m collecting the k8s resource definitions from multiple sources to then process them with ytt and apply them with kapp, and have a consistent directory structure. When I get stuff from github releases I just get the files I need in the root of the path, when I get it this other way it ends in subdirectories, and hence dir layout is not really consistent

one of the reasons why its is like that at this moment is because its not clear what should happen when there are conflicts between file names, directory name vs file name, etc. any suggestions?

It would maybe make sense to add a relocation/fail strategy. An example would be:
- path: vendor/k8s
  contents:
  - path: eduk8s
    git:
      url: https://github.com/eduk8s/eduk8s
      ref: <IP_ADDRESS>
    includePaths:
    - resources/crds-v1/**/*
    - resources/crds-v1alpha1/**/*
    - resources/operator/**/*
    relocatePath:  # Apply this relocation rules in order, fail if one don't succeed
    - resources/crds-v1/**/*=/**/*                          # To strip path
    - resources/crds-v1alpha1/=/                            # To strip path (as it ends with / is a dir. Fail if a file)
    - resources/operator/**/*=resources/**/*       # To relocate everything under resources folder (rename folder)
    - resources/operator/example.yaml=resources/my-example.yaml   # rename file

We can definitely see how this need is really valid!
For further discussion, what aspects of directory shaping fit within vendir's responsibilities?
How much of that is intrinsic to vendoring in software?

https://github.com/k14s/vendir/releases/tag/v0.11.0 includes newRootPath configuration that changes "root" directory for the source.

Example (and discussion) demonstrating why this could be helpful: https://kubernetes.slack.com/archives/CH8KCCKA5/p1625843954329600?thread_ts=1625811418.324700&cid=CH8KCCKA5


some issues about the code in pytorch_embeding.py

Hello, I'd like to know , in line 38, why the stride is set to sum(latent_dim) as well? when I run the code, I've met some problems:
RuntimeError: Calculated padded input size per channel: (1). Kernel size: (5). Kernel size can't be greater than actual input size.

Hi, I set it the sum(latent_dim) because in this line I flatten sorted node embeddings to 1 * k(sum_latent_dim). If the kernel size is larger than actual input size in your case, this should not happen in the first conv1d layer (since you have at least k times kernel width sequence), and could possibly happen in later conv1d layers. You may reduce the kernel width of later conv1d layers, or increase k to solve the issue.


Email notifications have errors

The email subject has errors. Example:

"Adnan Fatayerji commented post" as well as the apostrophe being displayed as <& # 0 3 9 ;>

or "Andreas just wrote post"

Weekly Summary also says "likes post" and "likes comment"

Also within freeflowpages' notifications, the same errors appear:

In addition:
Would be nice to add Instagam to the Social Bookmarks section for Profiles.
In the profile view, it looks like the spacing for the text for the "About" section is misaligned.
When you go to add a friend, it brings you to their profile, but I think it should just keep you on the page you are on and instead say "Request Sent" or something along those lines.
I got an email notification that said "Adnan Fatayerji commented post "Are you interested..." It should be "commented on a post" or "commented on the post" so I would recommend those notification emails get double-checked.
Also on FreeFlow Pages itself, in the Latest Activities section, the notification says "Sabrina Sadik likes post "Thank you everyone for attenting..." but it should be "Sabrina Sadik likes the post "Thank you everyone..."


Different embeddings obtained when running with different batch size

Thanks for sharing this awesome work.
I'm trying a simple symmetric text retrieval demo, which involves computing text embeddings for text retrieval. What I don't understand is why I get different embeddings when I run l2v with one caption or multiple captions:
sentences = [
    "how much protein should a female eat",
    "summit define",
    "As a general guideline",
    "Definition of summit for English Language Learners"]
print(l2v.encode(sentences[0:1])[0:1, :10])
print(l2v.encode(sentences[0:2])[0:1, :10])
print(l2v.encode(sentences[0:3])[0:1, :10])
print(l2v.encode(sentences[0:4])[0:1, :10])

The print statements would always output the first 10 out of 4096 values of the first caption's embedding. I expect all print statements outputing the same values but actually they don't. Am I misunderstanding something here?
Thanks for your help!

Hi @wufeim,
Thank you for your interest in our work and raising this issue. While exploring this issue we actually uncovered a bug in our code (#74 ).
Firstly, the output of batch size 1 is very different from output of batch size >1 because of a bug in the implementation of bidirectional_llama.py. I have pushed a fix (#75 ). If you plan to use batch size of 1 while using encode, you should build lm2vec package from source using the latest changes. The bug does not impact when the batch size is greater than 1.
Regarding the output being different for batch size 2,3,4 - this is a know issue of transformers library. Basically, this happens due to accumulation of matrix multiplication errors, which is more pronounced in lower precision like bf16. Here is a detailed explanation by one of the maintainers of transformers library.
By changing the precision of fp32, the variability with batch size will reduce quite a lot but will never be zero (see tests run in the above detailed explanation)
Here are more related issues on the transformers library for reference:
https://github.com/huggingface/transformers/issues/26869
https://github.com/huggingface/transformers/issues/27626
Another on llama.cpp
https://github.com/ggerganov/llama.cpp/issues/3014
Hope this answers your question, let me know if you have any more queries.

Closing as it is stale, feel free to re-open if the issue persists


Change Makefile so /regression and /unit executables are not built by

default. That's because currently those executables take way too much space and if you have other stuff in your $HOME, your diskspace can easily fill up and crash the build. Now one has to build these on demand with, for example,
make cuda-build/regression
make cuda-build/unit
or specific tests with
make cuda-build/regression/rt_weibel_4d
make cuda-build/unit/ctest_dg_bin_ops
Additionally, change -j in scripts that build dependencies to
-j 32 because Stellar and Perlmutter admins don't like that we hog
all the cores on the login node.
Adapt config and make files to allow passing location of cuda math
libraries, because on Perlmutter they are in some other nonstandard
location. Update Perlmutter machine files with this (also had to unload
darshan module for some reason, recommended by admins).

This is probably ok, but it seems that target names should be just make unit and make regression. It’s ok to only build the lib by default. Make this change and we can merge the PR
Sent from my iPhone

On Oct 29, 2022, at 6:54 PM, Manaure Francisquez @.***> wrote:

@manauref requested your review on: #108 Change Makefile so /regression and /unit executables are not built by.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because your review was requested.


[New Port Request] D3D12MemAlloc

Library name: D3D12 Memory Allocator
Library description: Simple-to-use GPU memory allocator for the D3D12 GPU API
Source repository URL: https://github.com/GPUOpen-LibrariesAndSDKs/D3D12MemoryAllocator/
Project homepage (if different from the source repository): https://gpuopen.com/d3d12-memory-allocator/
Anything else that is useful to know when adding (such as optional features the library may have that should be included): Nothing else I'm aware of

I copied src/D3D12MemAlloc.h and src/D3D12MemAlloc.cpp into my build tree and added them to my VS solution

If you wish to add this port in the future, you can reopen this issue, but we're closing it for now.


Enforce proper commit workflow

Right now we don't really have a workflow for commits. People make commits as they wish and we pull it all in. We should probably enforce that commits are made in such a way that it's easier to review the changes.

Let's compile some guidelines and thoughts here that we can put into DEVGUIDE as well, please.

Does this include determining a branching workflow (cf. what we discussed a while ago in Discord) as well?

That's separate IMO. You can make another issue for that I suppose.

I'd like to bump this, and also add that the release cycle/criteria should be communicated in the README. master isn't passing CI right now, and staging is 135 commits ahead, 100 commits behind master.


merge from template-sync/template-arlac77-github

package.json

merge from template package.json

Pull Request Test Coverage Report for Build<PHONE_NUMBER>0
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.842%

Totals

Change from base Build<PHONE_NUMBER>9:
0.0%

Covered Lines:
416

Relevant Lines:
426

💛  - Coveralls


Error: SourceMapGenerator.prototype.applySourceMap requires either an explicit source file, or the source map's "file" property.

Hi guys,
After this https://github.com/jakemmarsh/angularjs-gulp-browserify-boilerplate/pull/124 I've started getting this error.
07 12 2015 11:29:58.524:ERROR [karma]: [Error:     SourceMapGenerator.prototype.applySourceMap requires either an explicit source file, or the     source map's "file" property. Both were omitted.]
Error: SourceMapGenerator.prototype.applySourceMap requires either an explicit source file, or the source map's "file" property. Both were omitted.
    at SourceMapGenerator_applySourceMap [as applySourceMap] (...\node_modules\browserify-    ngannotate\node_modules\ng-annotate\node_modules\source-map\lib\source-map\source-map-    generator.js:175:17)
    at SourceMapper.applySourceMap (...\node_modules\browserify-    ngannotate\node_modules\ng-annotate\build\es5\generate-sourcemap.js:92:20)
    at generateSourcemap (...\node_modules\browserify-ngannotate\node_modules\ng-    annotate\build\es5\generate-sourcemap.js:115:20)
    at ngAnnotate (...\node_modules\browserify-ngannotate\node_modules\ng-    annotate\build\es5\ng-annotate-main.js:1153:9)
    at DestroyableTransform.flush [as _flush] (...\node_modules\browserify-    ngannotate\index.js:45:24)
    at DestroyableTransform.<anonymous> (...\node_modules\browserify-ngannotate\node_modules\through2\node_modules\readable-    stream\lib\_stream_transform.js:123:12)
    at DestroyableTransform.g (events.js:260:16)
    at emitNone (events.js:67:13)
    at DestroyableTransform.emit (events.js:166:7)
    at prefinish (...\node_modules\browserify-    ngannotate\node_modules\through2\node_modules\readable-    stream\lib\_stream_writable.js:499:12)
    at finishMaybe (...\node_modules\browserify-    ngannotate\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:507:7)
    at endWritable (...\node_modules\browserify-    ngannotate\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:519:3)
    at DestroyableTransform.Writable.end (...\node_modules\browserify-    ngannotate\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:484:5)
    at DuplexWrapper.onend (...\node_modules\bulkify\node_modules\static-    module\node_modules\duplexer2\node_modules\readable-stream\lib\_stream_readable.js:537:10)
    at DuplexWrapper.g (events.js:260:16)
    at emitNone (events.js:72:20)

Any way of fixing this?
Thanks

The main.js.map file is being generated, and it's located in the build folder right next to the main.js.
For some reason the configuration from the https://github.com/DrewML/angularjs-gulp-browserify-boilerplate/commit/a91fb0ac93a998908aeb87b2d488d277234b4e9c didn't work for me in the manner that the sourcemap file hasn't been generated on the correct path.
const sourceMapLocation = global.isProd ? './' : '';

When I've changed it from '' to './', as it is for the production, the file got generated.
Although I'm able to generate the sourcemap file, the error still persists.
Do we perhaps need something like this? https://www.npmjs.com/package/karma-sourcemap-loader

Problem in the karma configuration. Duplicated 'browserify-ngannotate',... Ma bad :)
I'll move the sourceMapLocation problem to the corresponding place.


[branch-2.9][cherry-pick] Close TransactionBuffer when MessageDeduplication#checkStatus failed

Motivation
Cherry-pick https://github.com/apache/pulsar/pull/19157 into branch-2.9
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

@tjiuming Could you please help check the failed test? It looks like introduced by this PR.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


⚠ Bofrost.es has degraded performance

In 7961c63, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1520 ms

Resolved: Bofrost.es performance has improved in 7a08756 after 10 minutes.


Bump PHP version to 7.1, clean some things up

@hungneox @crisu83 what's your opinion on this? Should we do it?


Bug report

The problem I'm experiencing is
    After executing:
git clone https://github.com/InverseIntegral/gdbmeter.git
    cd gdbmeter
    ./gradlew clean assemble
    cd build/libs
    java -jar gdbmeter-*.jar -db janus -o partition
It caused:
    java.lang.IllegalArgumentException: Could not find implementation class: org.janusgraph.diskstorage.lucene.LuceneIndex
Then I add a dependency:
    implementation("org.janusgraph:janusgraph-lucene:0.6.3")
Now the exception turns into:
    java.lang.IllegalArgumentException: Unsupported class file major version 61
Here are some details about my setup:

Operating System: windows11 wsl2
Java Version: jdk 17

Thank you for your attention to this matter.

Hi @choeoe,
Not sure if @InverseIntegral still actively maintains the project. You might also want to check https://matteokamm.ch/ on how to contact him, as I'm unsure if he still uses his ETH email address.
Perhaps @YuanchengJiang has an idea of how to build the project, as he recently successfully compared against GDBMeter in his evaluation. In case you work on graph database systems testing, you might also be interested in his ICSE '24 paper and project (see https://github.com/YuanchengJiang/GraphGenie and https://yuanchengjiang.github.io/docs/GraphGenie-ICSE24.pdf).

Thank you for your reply!

Hi @choeoe, please try on the Ubuntu environment.

@choeoe sorry for my delayed answer. You are correct, a dependency on lucene index is missing. I do have a working fix that makes it possible to test the JanusGraph database again. The problem seems related to #17 (both occurred after switching from Maven to Gradle). I will provide the fix in the next few days after testing it thoroughly :slightly_smiling_face:

@choeoe the newest version should fix your problem. It adds the missing dependency. Note, however, that we now only support JDK 11 due to JanusGraph not supporting anything above that :slightly_frowning_face: I will consider dropping support for JanusGraph so that a higher JDK version can be used.

@choeoe With the newest version the problem has been fixed. The missing dependency has been added. Note, however, that we currently only support JDK 11 since JanusGraph only supports up to that version :slightly_frowning_face:


Asset change is sent to the unknown address

After a successful asset transfer using the cli and descriptor wallet, the change was sent to an unknown address.
Description
We followed the RGB v0.10 release video demo, generated all our wallets using descriptor wallet, managed to issue contracts, perform a transfer and update our stash. However when we broadcast the final PSBT to the blockchain the change UTXO was sent to an address that the sending wallet did not control and this effectively resulted in us transfer 100 of an RGB20 asset but having all the change (so the remaining supply) sent to a random address. We can recreate the issue every time we try transfer. No errors are produced.
The bug is also present when calling “rgb utxos” after broadcasting the transfer as the RGB CLI can’t detect the UTXO either.
In the video Maxim uses this TPUB: tpubDCNiWHaiSkgnQjuhsg9kjwaUzaxQjUcmhagvYzqQ3TYJTgFGJstVaqnu4yhtFktBhCVFmBNLQ5sN53qKzZbMksm3XEyGJsEhQPfVZdWmTE2
Then the following steps are made:

create an RGB wallet for with the descriptor tapret(tpubDCNiWHaiSkgnQjuhsg9kjwaUzaxQjUcmhagvYzqQ3TYJTgFGJstVaqnu4yhtFktBhCVFmBNLQ5sN53qKzZbMksm3XEyGJsEhQPfVZdWmTE2, ...)
issue<PHONE_NUMBER>000000 of an RGB20 asset called TST to a UTXO associated with this wallet.
perform a transfer action where he transfers 100 TST to a new UTXO, sign the transfer PSBT and then broadcast it.

This is the transaction he broadcast in the video: https://www.blockstream.info/testnet/tx/85f9b8ce3b495a206de334928c89ce8fc93f02304b634641ac76e4008948e2a3
The transaction has 1 input (The UTXO which sealed the TST) and 1 output, which is the change from the transaction. The change also is the UTXO where the remaining TST is bound. The address the change is sent to is: tb1pmv024fd3f62peg5rg5ex8m3zl4n9qkze5xsrw58fknh5987hkmusflwcpu
The tpub from earlier is unable to detect the change utxo (85f9b8ce3b495a206de334928c89ce8fc93f02304b634641ac76e4008948e2a3:0) when calling "rgb utxos" OR when calling "btc-cold check " furthermore the change utxo is unspent (meaning the problem is not that it has been spent).
The result in the video and when we performed tests is that when you do a RGB transfer call the correct amount of assets get transferred to the person you want to send them to but the change gets sent to a random address meaning the original owner loses their RGB20 assets (or loses the ability to spend them). From what I can tell this looks like when the RGB CLI is manipulating the PSBT (through the transfer call) it is deriving the change address incorrectly from the tpub.

Hi @UkolovaOlga,
I think this is not necessary a bug, only missing steps in rgb-cli + adjust compatibilities between tools, to facilitate use by people unfamiliar with the projects.

We followed the RGB v0.10 release video demo, generated all our wallets using descriptor wallet, managed to issue contracts, perform a transfer and update our stash. However when we broadcast the final PSBT to the blockchain the change UTXO was sent to an address that the sending wallet did not control and this effectively resulted in us transfer 100 of an RGB20 asset but having all the change (so the remaining supply) sent to a random address. We can recreate the issue every time we try transfer. No errors are produced.

There is not "random address", just an output script pubkey + tweak.
In the bitmask project I created a few more steps to allow "spending change contracts", below:

I created a way to extract the tweak from the PSBT file, so that I can use it to generate a new PSBT.
I store the tweak (tapret) in RgbDesc, which is a descriptor Maxim created exclusively for RGB CLI
Customized the descriptor-wallet to support a input + tweak.

I opened an issue on bitmask-core to track all my changes and create "official" fixes or at least help Maxim with testing, hehehe!
I plan to address some in the next few days, I'm reviewing the implementations to remove "customization overkill" =)
See here: https://github.com/diba-io/bitmask-core/issues/204
I hope I've helped.

This is duplicate of https://github.com/RGB-WG/rgb/issues/16
@crisdut is right. Pls also check my comment in https://github.com/RGB-WG/rgb/issues/16#issuecomment-1564038954