Leading provisioer randomly deadlock and fail on PVCs provision

Describe the bug
After some period of time, Ceph CSI provisioner will stuck
Environment details

Image/version of Ceph CSI driver : quay.io/cephcsi/cephcsi:v3.2.0
Helm chart version : Based off Rook
Kernel version : dependent
Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its
krbd or rbd-nbd) : cephfs/default (most likely cephfs/kernel)
Kubernetes cluster version : v1.20.4+k3s1
Ceph cluster version : min_mon_release 15 (octopus)

Steps to reproduce
Steps to reproduce the behavior:

Set up a Ceph CSI cluster connection. Either directly create it by manually writing manifests for ceph-csi, or import an external cluster from Rook
(if taken Rook approach) Enable CephFS/RBD CSI provisioner
Make a storage class that request persistent volume from the CSI provisioner of your own choice
Try to provision a large number of /PVC?s/ using that storage class at the same time

Actual results
Some /PVC?s?/ creation will stuck in a pending state. (fun fact: my own lottery shows it has a 1-0.636=0.363=36.3% chance of failing -- I provisioned 11 volumes at the same time, only 7 fulfilled)
Expected behavior
All the /PVC?s?/ fulfillment to be committed without having to worry about any of them pending
Logs
I don't think I need any logs at the moment, the symptoms are identical to what this issue is: https://github.com/ceph/ceph-csi/issues/1828
Workaround
You can just kill the lease holder, or "leader" of the CSI provisioner, so when the new leader is elected, the /PVC?/ fulfillment will immediately turnaround, probably due to checking the /PVC?s?/ for any state refresh by examining any need to modify the stats, for example volume expansion.
Another good workaround is, the leader needs to (in)voluntarily release the lease once a certain failure distribution met (for example, over the last 5 minutes we observed 90% of the fulfillment failed) (and better yet, disallow second term election for itself), so a new leader can be elected, and we can use the previously mentioned workaround behavior to refresh all /PVC?/ states. If more than half of the historical terms have any pending PVC, we consider the CSI provisioners total failure, and kill them all together in order to hope for a new state.
Additional context
That said, if the workaround behavior is consistent, i.e. leader is "unresponsive" after a certain time, we can either use an internal K8S helper to listen for any PVC stream satisfying pending time > certain threshold (also means if creation time + threshold > current time), then we can assume the leader is dead, and we can issue a kill pod command to the leader and it will somewhat automatically recover the next time a new leader prevails. This however is very resource intensive and disruptive and I do not recommend it as the official solution.
Strictly speaking, the leader isn't technically dead since maybe it's just waiting for Ceph to turnaround the creation result (not confirmed)?  So it doesn't really satisfy the requirement for being a livelock either. There is already an internal deadline check but it just keep retrying unbounded, this may also indicate the connection to the Ceph /monitors?/ is/are not stable, as some people do get stuck when issuing Ceph related operational command manually by attaching to the pod (I happened to not be stuck, so I have no idea what is going on).

@stevefan1999-personal it could be kind of a throttling issue or ceph cluster command delayed or hung for sometime. To understand some more on this issue can you please let me know what kind ( cephfs or rbd  OR both ? ) of PVCs were provisioned?

@stevefan1999-personal it could be kind of a throttling issue or ceph cluster command delayed or hung for sometime. To understand some more on this issue can you please let me know what kind ( cephfs or rbd  OR both ? ) of PVCs were provisioned?

I provisioned cephfs most of the time, but using rbd had the same effect too

@stevefan1999-personal it could be kind of a throttling issue or ceph cluster command delayed or hung for sometime. To understand some more on this issue can you please let me know what kind ( cephfs or rbd  OR both ? ) of PVCs were provisioned?

I provisioned cephfs most of the time, but using rbd had the same effect too

We have seen an issue with CephFS core where a deadlock cause the cephfs command to hung in a cluster.  If its common issue between cephfs and rbd , it could be something else like throttling requests or slow operations in ceph cluster..etc. The logs captured at time of the issue could help .

@stevefan1999-personal it could be kind of a throttling issue or ceph cluster command delayed or hung for sometime. To understand some more on this issue can you please let me know what kind ( cephfs or rbd  OR both ? ) of PVCs were provisioned?

I provisioned cephfs most of the time, but using rbd had the same effect too

We have seen an issue with CephFS core where a deadlock cause the cephfs command to hung in a cluster.  If its common issue between cephfs and rbd , it could be something else like throttling requests or slow operations in ceph cluster..etc. The logs captured at time of the issue could help .

ok, i will reproduce the situtaion and capture the log

Here it is

I0324 07:48:54.166395       1 controller.go:1420] provision "minio/zero-minio" class "rook-cephfs": volume "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c" provisioned
I0324 07:48:54.166426       1 controller.go:1437] provision "minio/zero-minio" class "rook-cephfs": succeeded
E0324 07:48:54.213448       1 controller.go:1443] couldn't create key for object pvc-c230f8ed-124b-4b75-bd59-65baa68b120c: object has no meta: object does not implement the Object interfaces
I0324 07:48:54.213519       1 controller.go:1317] provision "minio/zero-minio" class "rook-cephfs": started
I0324 07:48:54.214330       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"minio", Name:"zero-minio", UID:"c230f8ed-124b-4b75-bd59-65baa68b120c", APIVersion:"v1", ResourceVersion:"11803243", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-c230f8ed-124b-4b75-bd59-65baa68b120c
I0324 07:48:54.214352       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"minio", Name:"zero-minio", UID:"c230f8ed-124b-4b75-bd59-65baa68b120c", APIVersion:"v1", ResourceVersion:"11803243", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "minio/zero-minio"
I0324 07:48:54.534570       1 controller.go:1420] provision "minio/zero-minio" class "rook-cephfs": volume "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c" provisioned
I0324 07:48:54.534609       1 controller.go:1437] provision "minio/zero-minio" class "rook-cephfs": succeeded
E0324 07:48:54.577812       1 controller.go:1443] couldn't create key for object pvc-c230f8ed-124b-4b75-bd59-65baa68b120c: object has no meta: object does not implement the Object interfaces
I0324 07:48:54.578557       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"minio", Name:"zero-minio", UID:"c230f8ed-124b-4b75-bd59-65baa68b120c", APIVersion:"v1", ResourceVersion:"11803277", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-c230f8ed-124b-4b75-bd59-65baa68b120c
I0324 07:49:09.240156       1 controller.go:1453] delete "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c": started
I0324 07:49:11.072794       1 controller.go:1468] delete "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c": volume deleted
I0324 07:49:11.130254       1 controller.go:1518] delete "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c": persistentvolume deleted
E0324 07:49:11.130450       1 controller.go:1521] couldn't create key for object pvc-c230f8ed-124b-4b75-bd59-65baa68b120c: object has no meta: object does not implement the Object interfaces
I0324 07:49:11.130522       1 controller.go:1523] delete "pvc-c230f8ed-124b-4b75-bd59-65baa68b120c": succeeded
I0324 07:49:19.243908       1 controller.go:1453] delete "pvc-0b58177e-75f7-404f-b90c-628b0277f0fb": started
I0324 07:49:20.983336       1 controller.go:1468] delete "pvc-0b58177e-75f7-404f-b90c-628b0277f0fb": volume deleted
I0324 07:49:21.038917       1 controller.go:1518] delete "pvc-0b58177e-75f7-404f-b90c-628b0277f0fb": persistentvolume deleted
E0324 07:49:21.039115       1 controller.go:1521] couldn't create key for object pvc-0b58177e-75f7-404f-b90c-628b0277f0fb: object has no meta: object does not implement the Object interfaces
I0324 07:49:21.039188       1 controller.go:1523] delete "pvc-0b58177e-75f7-404f-b90c-628b0277f0fb": succeeded
I0324 07:49:22.630687       1 controller.go:1453] delete "pvc-89956e8e-7585-4b14-86f3-0496289a9dd8": started
I0324 07:49:24.619837       1 controller.go:1468] delete "pvc-89956e8e-7585-4b14-86f3-0496289a9dd8": volume deleted
I0324 07:49:24.673737       1 controller.go:1518] delete "pvc-89956e8e-7585-4b14-86f3-0496289a9dd8": persistentvolume deleted
E0324 07:49:24.673930       1 controller.go:1521] couldn't create key for object pvc-89956e8e-7585-4b14-86f3-0496289a9dd8: object has no meta: object does not implement the Object interfaces
I0324 07:49:24.674134       1 controller.go:1523] delete "pvc-89956e8e-7585-4b14-86f3-0496289a9dd8": succeeded
I0324 07:49:25.647468       1 controller.go:1453] delete "pvc-c7e33d4f-7062-43dd-956c-d0aa0923a81b": started
I0324 07:49:26.048131       1 controller.go:1453] delete "pvc-2015690f-53df-4132-a2b8-c35b0d397394": started
I0324 07:49:27.456828       1 controller.go:1468] delete "pvc-c7e33d4f-7062-43dd-956c-d0aa0923a81b": volume deleted
I0324 07:49:27.510795       1 controller.go:1518] delete "pvc-c7e33d4f-7062-43dd-956c-d0aa0923a81b": persistentvolume deleted
E0324 07:49:27.511064       1 controller.go:1521] couldn't create key for object pvc-c7e33d4f-7062-43dd-956c-d0aa0923a81b: object has no meta: object does not implement the Object interfaces
I0324 07:49:27.511203       1 controller.go:1523] delete "pvc-c7e33d4f-7062-43dd-956c-d0aa0923a81b": succeeded
I0324 07:49:27.861625       1 controller.go:1468] delete "pvc-2015690f-53df-4132-a2b8-c35b0d397394": volume deleted
I0324 07:49:27.915347       1 controller.go:1518] delete "pvc-2015690f-53df-4132-a2b8-c35b0d397394": persistentvolume deleted
E0324 07:49:27.915676       1 controller.go:1521] couldn't create key for object pvc-2015690f-53df-4132-a2b8-c35b0d397394: object has no meta: object does not implement the Object interfaces
I0324 07:49:27.915837       1 controller.go:1523] delete "pvc-2015690f-53df-4132-a2b8-c35b0d397394": succeeded
I0324 07:49:29.263025       1 controller.go:1453] delete "pvc-153bc89e-1b5f-4289-8a3a-f01459b56db0": started
I0324 07:49:31.023810       1 controller.go:1468] delete "pvc-153bc89e-1b5f-4289-8a3a-f01459b56db0": volume deleted
I0324 07:49:31.079124       1 controller.go:1518] delete "pvc-153bc89e-1b5f-4289-8a3a-f01459b56db0": persistentvolume deleted
E0324 07:49:31.079351       1 controller.go:1521] couldn't create key for object pvc-153bc89e-1b5f-4289-8a3a-f01459b56db0: object has no meta: object does not implement the Object interfaces
I0324 07:49:31.079442       1 controller.go:1523] delete "pvc-153bc89e-1b5f-4289-8a3a-f01459b56db0": succeeded
I0324 07:49:59.242140       1 controller.go:1453] delete "pvc-b1c60184-44fb-47d8-a7c1-2d4f6a3cfe1c": started
I0324 07:50:01.028464       1 controller.go:1468] delete "pvc-b1c60184-44fb-47d8-a7c1-2d4f6a3cfe1c": volume deleted
I0324 07:50:01.085963       1 controller.go:1518] delete "pvc-b1c60184-44fb-47d8-a7c1-2d4f6a3cfe1c": persistentvolume deleted
E0324 07:50:01.086666       1 controller.go:1521] couldn't create key for object pvc-b1c60184-44fb-47d8-a7c1-2d4f6a3cfe1c: object has no meta: object does not implement the Object interfaces
I0324 07:50:01.086844       1 controller.go:1523] delete "pvc-b1c60184-44fb-47d8-a7c1-2d4f6a3cfe1c": succeeded
I0324 07:50:11.333070       1 controller.go:1317] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": started
I0324 07:50:11.336113       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-chartmuseum", UID:"bcc1c5bb-751d-4ac8-abb0-3c42371eda0f", APIVersion:"v1", ResourceVersion:"11804166", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/zero-harbor-chartmuseum"
I0324 07:50:11.344450       1 controller.go:1317] provision "harbor/zero-harbor-registry" class "rook-cephfs": started
I0324 07:50:11.348771       1 controller.go:1317] provision "harbor/zero-harbor-jobservice" class "rook-cephfs": started
I0324 07:50:11.349194       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-registry", UID:"09254db1-e8c0-4be5-bfcb-e8f01d30c2af", APIVersion:"v1", ResourceVersion:"11804167", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/zero-harbor-registry"
I0324 07:50:11.349214       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-jobservice", UID:"530e31e3-27be-4fa3-8e81-3582f8391b02", APIVersion:"v1", ResourceVersion:"11804170", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/zero-harbor-jobservice"
I0324 07:50:11.467774       1 controller.go:1317] provision "harbor/data-zero-harbor-trivy-0" class "rook-cephfs": started
I0324 07:50:11.469815       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-harbor-trivy-0", UID:"db795b56-74a0-4f75-b028-358e4472a221", APIVersion:"v1", ResourceVersion:"11804276", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-harbor-trivy-0"
I0324 07:50:11.475247       1 controller.go:1317] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": started
I0324 07:50:11.475575       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"redis-data-zero-redis-master-0", UID:"b846d8e7-2d1b-4363-a774-1e0d39dc6761", APIVersion:"v1", ResourceVersion:"11804282", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/redis-data-zero-redis-master-0"
I0324 07:50:11.476105       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:50:11.476861       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
I0324 07:50:11.730101       1 controller.go:1420] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": volume "pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f" provisioned
I0324 07:50:11.730438       1 controller.go:1437] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": succeeded
I0324 07:50:11.807351       1 controller.go:1420] provision "harbor/zero-harbor-registry" class "rook-cephfs": volume "pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af" provisioned
I0324 07:50:11.807813       1 controller.go:1437] provision "harbor/zero-harbor-registry" class "rook-cephfs": succeeded
I0324 07:50:11.830808       1 controller.go:1420] provision "harbor/zero-harbor-jobservice" class "rook-cephfs": volume "pvc-530e31e3-27be-4fa3-8e81-3582f8391b02" provisioned
I0324 07:50:11.831055       1 controller.go:1437] provision "harbor/zero-harbor-jobservice" class "rook-cephfs": succeeded
I0324 07:50:11.886988       1 controller.go:1420] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": volume "pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761" provisioned
I0324 07:50:11.887207       1 controller.go:1437] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": succeeded
I0324 07:50:11.900164       1 controller.go:1420] provision "harbor/data-zero-harbor-trivy-0" class "rook-cephfs": volume "pvc-db795b56-74a0-4f75-b028-358e4472a221" provisioned
I0324 07:50:11.900749       1 controller.go:1437] provision "harbor/data-zero-harbor-trivy-0" class "rook-cephfs": succeeded
E0324 07:50:11.981167       1 controller.go:1443] couldn't create key for object pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f: object has no meta: object does not implement the Object interfaces
I0324 07:50:11.981472       1 controller.go:1317] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": started
I0324 07:50:11.982814       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-chartmuseum", UID:"bcc1c5bb-751d-4ac8-abb0-3c42371eda0f", APIVersion:"v1", ResourceVersion:"11804166", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/zero-harbor-chartmuseum"
I0324 07:50:11.982982       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-chartmuseum", UID:"bcc1c5bb-751d-4ac8-abb0-3c42371eda0f", APIVersion:"v1", ResourceVersion:"11804166", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f
E0324 07:50:12.180800       1 controller.go:1443] couldn't create key for object pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af: object has no meta: object does not implement the Object interfaces
I0324 07:50:12.181083       1 controller.go:1317] provision "harbor/zero-harbor-registry" class "rook-cephfs": started
I0324 07:50:12.181261       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-registry", UID:"09254db1-e8c0-4be5-bfcb-e8f01d30c2af", APIVersion:"v1", ResourceVersion:"11804167", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/zero-harbor-registry"
I0324 07:50:12.181409       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-registry", UID:"09254db1-e8c0-4be5-bfcb-e8f01d30c2af", APIVersion:"v1", ResourceVersion:"11804167", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af
E0324 07:50:12.385233       1 controller.go:1443] couldn't create key for object pvc-530e31e3-27be-4fa3-8e81-3582f8391b02: object has no meta: object does not implement the Object interfaces
I0324 07:50:12.385553       1 controller.go:1317] provision "harbor/zero-harbor-jobservice" class "rook-cephfs": started
I0324 07:50:12.385660       1 controller.go:1326] provision "harbor/zero-harbor-jobservice" class "rook-cephfs": persistentvolume "pvc-530e31e3-27be-4fa3-8e81-3582f8391b02" already exists, skipping
I0324 07:50:12.385752       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-jobservice", UID:"530e31e3-27be-4fa3-8e81-3582f8391b02", APIVersion:"v1", ResourceVersion:"11804170", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-530e31e3-27be-4fa3-8e81-3582f8391b02
E0324 07:50:12.580824       1 controller.go:1443] couldn't create key for object pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761: object has no meta: object does not implement the Object interfaces
I0324 07:50:12.580902       1 controller.go:1317] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": started
I0324 07:50:12.580999       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"redis-data-zero-redis-master-0", UID:"b846d8e7-2d1b-4363-a774-1e0d39dc6761", APIVersion:"v1", ResourceVersion:"11804282", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/redis-data-zero-redis-master-0"
I0324 07:50:12.581033       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"redis-data-zero-redis-master-0", UID:"b846d8e7-2d1b-4363-a774-1e0d39dc6761", APIVersion:"v1", ResourceVersion:"11804282", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761
E0324 07:50:12.781907       1 controller.go:1443] couldn't create key for object pvc-db795b56-74a0-4f75-b028-358e4472a221: object has no meta: object does not implement the Object interfaces
I0324 07:50:12.782284       1 controller.go:1317] provision "harbor/data-zero-harbor-trivy-0" class "rook-cephfs": started
I0324 07:50:12.782409       1 controller.go:1326] provision "harbor/data-zero-harbor-trivy-0" class "rook-cephfs": persistentvolume "pvc-db795b56-74a0-4f75-b028-358e4472a221" already exists, skipping
I0324 07:50:12.782578       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-harbor-trivy-0", UID:"db795b56-74a0-4f75-b028-358e4472a221", APIVersion:"v1", ResourceVersion:"11804276", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-db795b56-74a0-4f75-b028-358e4472a221
I0324 07:50:13.133737       1 request.go:645] Throttling request took 1.150528703s, request: PATCH:https://<IP_ADDRESS>:443/api/v1/namespaces/harbor/events/zero-harbor-chartmuseum.166f37dc7a4717b3
I0324 07:50:13.254704       1 controller.go:1420] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": volume "pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f" provisioned
I0324 07:50:13.255025       1 controller.go:1437] provision "harbor/zero-harbor-chartmuseum" class "rook-cephfs": succeeded
I0324 07:50:13.656099       1 controller.go:1420] provision "harbor/zero-harbor-registry" class "rook-cephfs": volume "pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af" provisioned
I0324 07:50:13.656386       1 controller.go:1437] provision "harbor/zero-harbor-registry" class "rook-cephfs": succeeded
I0324 07:50:13.856553       1 controller.go:1420] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": volume "pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761" provisioned
I0324 07:50:13.856761       1 controller.go:1437] provision "harbor/redis-data-zero-redis-master-0" class "rook-cephfs": succeeded
E0324 07:50:13.978297       1 controller.go:1443] couldn't create key for object pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f: object has no meta: object does not implement the Object interfaces
I0324 07:50:13.978542       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-chartmuseum", UID:"bcc1c5bb-751d-4ac8-abb0-3c42371eda0f", APIVersion:"v1", ResourceVersion:"11804385", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-bcc1c5bb-751d-4ac8-abb0-3c42371eda0f
E0324 07:50:14.175276       1 controller.go:1443] couldn't create key for object pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af: object has no meta: object does not implement the Object interfaces
I0324 07:50:14.175569       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"zero-harbor-registry", UID:"09254db1-e8c0-4be5-bfcb-e8f01d30c2af", APIVersion:"v1", ResourceVersion:"11804389", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-09254db1-e8c0-4be5-bfcb-e8f01d30c2af
E0324 07:50:14.575745       1 controller.go:1443] couldn't create key for object pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761: object has no meta: object does not implement the Object interfaces
I0324 07:50:14.575826       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"redis-data-zero-redis-master-0", UID:"b846d8e7-2d1b-4363-a774-1e0d39dc6761", APIVersion:"v1", ResourceVersion:"11804402", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b846d8e7-2d1b-4363-a774-1e0d39dc6761
W0324 07:52:41.533634       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 0
E0324 07:52:41.534150       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0324 07:52:41.534217       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:52:41.539036       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0324 07:52:41.539065       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
W0324 07:52:41.583337       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 1
E0324 07:52:41.584673       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:41.584867       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:42.034087       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:52:42.036637       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
W0324 07:52:42.079089       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 2
E0324 07:52:42.079115       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:42.079142       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:44.079252       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:52:44.080737       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
W0324 07:52:44.122479       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 3
E0324 07:52:44.123784       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:44.123629       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:48.123962       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:52:48.125993       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
W0324 07:52:48.168740       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 4
E0324 07:52:48.169732       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:48.171022       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:56.172398       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:52:56.173114       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
W0324 07:52:56.215307       1 controller.go:943] Retrying syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114", failure 5
E0324 07:52:56.215345       1 controller.go:966] error syncing claim "f6fc54f3-74bd-453d-b114-79f2e7430114": failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists
I0324 07:52:56.215372       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-f6fc54f3-74bd-453d-b114-79f2e7430114 already exists

Of all the repeating deletions for reproduction of the issue, only the postgres volume failed to provision.
After killing the leader at the time of the failure, the new leader issue the volume immediately:

I0323 17:25:24.443025       1 csi-provisioner.go:121] Version: v2.0.0
I0323 17:25:24.443269       1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0323 17:25:24.454954       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
W0323 17:25:34.455000       1 connection.go:172] Still connecting to unix:///csi/csi-provisioner.sock
W0323 17:25:44.455428       1 connection.go:172] Still connecting to unix:///csi/csi-provisioner.sock
I0323 17:25:47.488464       1 common.go:111] Probing CSI driver for readiness
W0323 17:25:47.489957       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0323 17:25:47.518339       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/rook-ceph-cephfs-csi-ceph-com...
I0324 07:55:50.501778       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
I0324 07:55:50.621916       1 controller.go:820] Starting provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-8658f67749-24pt7_bc012d7c-c9b6-4d24-b28e-a89aa93bb635!
I0324 07:55:50.621931       1 clone_controller.go:66] Starting CloningProtection controller
I0324 07:55:50.621930       1 volume_store.go:97] Starting save volume queue
I0324 07:55:50.622090       1 clone_controller.go:84] Started CloningProtection controller
I0324 07:55:50.722541       1 controller.go:869] Started provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-8658f67749-24pt7_bc012d7c-c9b6-4d24-b28e-a89aa93bb635!
I0324 07:55:50.723445       1 controller.go:1317] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": started
I0324 07:55:50.723827       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "harbor/data-zero-postgresql-0"
I0324 07:55:50.845749       1 controller.go:1420] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": volume "pvc-f6fc54f3-74bd-453d-b114-79f2e7430114" provisioned
I0324 07:55:50.845777       1 controller.go:1437] provision "harbor/data-zero-postgresql-0" class "rook-cephfs": succeeded
E0324 07:55:50.851583       1 controller.go:1443] couldn't create key for object pvc-f6fc54f3-74bd-453d-b114-79f2e7430114: object has no meta: object does not implement the Object interfaces
I0324 07:55:50.851578       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"harbor", Name:"data-zero-postgresql-0", UID:"f6fc54f3-74bd-453d-b114-79f2e7430114", APIVersion:"v1", ResourceVersion:"11804291", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-f6fc54f3-74bd-453d-b114-79f2e7430114

Might be related to #1881.

I think we hit the same bug / error in our cluster:
k8s: 1.20.6
ceph: 14.2.10
ceph-csi: 3.3.1
  Warning  ProvisioningFailed    14s                rbd.csi.ceph.com_ceph-csi-rbd-provisioner-7dbd7746f6-89fpg_2f6ac140-f938-4813-b38e-aeada4d648a6  failed to provision volume with StorageClass "csi-rbd-hdd": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  11s (x6 over 74s)  persistentvolume-controller                                                                      waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
  Normal   Provisioning          7s (x5 over 74s)   rbd.csi.ceph.com_ceph-csi-rbd-provisioner-7dbd7746f6-89fpg_2f6ac140-f938-4813-b38e-aeada4d648a6  External provisioner is provisioning volume for claim "staging/elasticsearch-data-elasticsearch-es-default-1"
  Warning  ProvisioningFailed    7s (x4 over 14s)   rbd.csi.ceph.com_ceph-csi-rbd-provisioner-7dbd7746f6-89fpg_2f6ac140-f938-4813-b38e-aeada4d648a6  failed to provision volume with StorageClass "csi-rbd-hdd": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-18d72dee-bb4b-4820-9c8c-f8833a95b7c5 already exists

I think because of the DeadlineExceeded error we also get the already exists message @nixpanic mentioned in https://github.com/ceph/ceph-csi/issues/1881
The error happens from time to time if we create RBD volumes. On the ceph cluster side there are not many operations ongoing so i think its not related to any throttling...
I try to gather some provisioner / GO logs if the error happens the next time.

I could just trigger the error again and this is what i saw from csi-provisioner:
I0610 15:42:27.242664       1 controller.go:1317] provision "common/postfix-smarthost" class "csi-rbd": started
I0610 15:42:27.242755       1 controller.go:573] CreateVolumeRequest {Name:pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa CapacityRange:required_bytes:52428800  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-system csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:ceph-system csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-system imageFeatures:layering pool:pool-hdd] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0610 15:42:27.243658       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"common", Name:"postfix-smarthost", UID:"0cb653fa-5b19-4b08-9a0c-e7ffebd164aa", APIVersion:"v1", ResourceVersion:"30106384", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "common/postfix-smarthost"
I0610 15:42:27.249351       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I0610 15:42:27.249414       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":52428800},"name":"pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","csi.storage.k8s.io/pv/name":"pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa","csi.storage.k8s.io/pvc/name":"postfix-smarthost","csi.storage.k8s.io/pvc/namespace":"common","imageFeatures":"layering","pool":"pool-hdd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0610 15:42:27.251001       1 connection.go:185] GRPC response: {}
I0610 15:42:27.251028       1 connection.go:186] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa already exists
I0610 15:42:27.251056       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa already exists
I0610 15:42:27.251083       1 controller.go:1091] Temporary error received, adding PVC 0cb653fa-5b19-4b08-9a0c-e7ffebd164aa to claims in progress
W0610 15:42:27.251102       1 controller.go:943] Retrying syncing claim "0cb653fa-5b19-4b08-9a0c-e7ffebd164aa", failure 36
E0610 15:42:27.251148       1 controller.go:966] error syncing claim "0cb653fa-5b19-4b08-9a0c-e7ffebd164aa": failed to provision volume with StorageClass "csi-rbd": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa already exists
I0610 15:42:27.251194       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"common", Name:"postfix-smarthost", UID:"0cb653fa-5b19-4b08-9a0c-e7ffebd164aa", APIVersion:"v1", ResourceVersion:"30106384", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa already exists

I0610 15:42:41.734019       1 controller.go:1317] provision "staging/elasticsearch-data-elasticsearch-es-default-1" class "csi-rbd": started
I0610 15:42:41.734108       1 controller.go:573] CreateVolumeRequest {Name:pvc-e4155096-1eb8-4f9f-9978-611bf7b568c1 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[clusterID:01578d80-6c97-46ba-9327-cb2b13980916 csi.storage.k8s.io/controller-expand-secret-name:csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace:ceph-system csi.storage.k8s.io/fstype:ext4 csi.storage.k8s.io/node-stage-secret-name:csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace:ceph-system csi.storage.k8s.io/provisioner-secret-name:csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace:ceph-system imageFeatures:layering pool:pool-hdd] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0610 15:42:41.734645       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"staging", Name:"elasticsearch-data-elasticsearch-es-default-1", UID:"e4155096-1eb8-4f9f-9978-611bf7b568c1", APIVersion:"v1", ResourceVersion:"29064698", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "staging/elasticsearch-data-elasticsearch-es-default-1"
I0610 15:42:41.740575       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I0610 15:42:41.740593       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e4155096-1eb8-4f9f-9978-611bf7b568c1","parameters":{"clusterID":"01578d80-6c97-46ba-9327-cb2b13980916","csi.storage.k8s.io/pv/name":"pvc-e4155096-1eb8-4f9f-9978-611bf7b568c1","csi.storage.k8s.io/pvc/name":"elasticsearch-data-elasticsearch-es-default-1","csi.storage.k8s.io/pvc/namespace":"staging","imageFeatures":"layering","pool":"pool-hdd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":1}}]}
I0610 15:42:44.365776       1 connection.go:185] GRPC response: {}
I0610 15:42:44.365831       1 connection.go:186] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.365862       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.365929       1 controller.go:1091] Temporary error received, adding PVC 485b659a-68c9-4ff4-9b58-f47002aea29d to claims in progress
W0610 15:42:44.365949       1 controller.go:943] Retrying syncing claim "485b659a-68c9-4ff4-9b58-f47002aea29d", failure 392
E0610 15:42:44.365982       1 controller.go:966] error syncing claim "485b659a-68c9-4ff4-9b58-f47002aea29d": failed to provision volume with StorageClass "csi-rbd": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.366079       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"staging", Name:"elasticsearch-data-elasticsearch-es-default-1", UID:"485b659a-68c9-4ff4-9b58-f47002aea29d", APIVersion:"v1", ResourceVersion:"29454218", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.366138       1 connection.go:185] GRPC response: {}
I0610 15:42:44.366169       1 connection.go:186] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.366188       1 controller.go:645] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.366218       1 controller.go:1091] Temporary error received, adding PVC f243aeba-3f0b-4d5d-afca-ad3b580d6146 to claims in progress
W0610 15:42:44.366231       1 controller.go:943] Retrying syncing claim "f243aeba-3f0b-4d5d-afca-ad3b580d6146", failure 392
E0610 15:42:44.366247       1 controller.go:966] error syncing claim "f243aeba-3f0b-4d5d-afca-ad3b580d6146": failed to provision volume with StorageClass "csi-rbd": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0610 15:42:44.366301       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"staging", Name:"elasticsearch-data-elasticsearch-es-default-0", UID:"f243aeba-3f0b-4d5d-afca-ad3b580d6146", APIVersion:"v1", ResourceVersion:"29454207", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-rbd": rpc error: code = DeadlineExceeded desc = context deadline exceeded

However, the given pvc volume does not exist in my cluster:
$ kubectl get pv |grep pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa
$ kubectl get pvc -A |grep pvc-0cb653fa-5b19-4b08-9a0c-e7ffebd164aa

After killing the ceph pod and waiting for the new leader election the volume was bound immediately.

I have multiple k8s clusters running (same k8s version, same ceph-csi version) but i'm only able to reproduce the error in one cluster - the other clusters are running smooth.
However, i can reproduce the error now relatively easy by just creating a few PVs (one after the other) and then the error happens and provisioning is broken until i restart the ceph-csi pod. I did a few more tests and it seems that the error only happens if i have scale the ceph-csi-rbd-provisioner deployment to 2x pods. If i scale down to 1x pod i cannot reproduce the error ...
@nixpanic @humblec if i should test anything for you just write me ...

nope

some news: it seems like after upgrading Ceph to the latest version of today (28-8-2021) solved this problem. I have yet to confirm it however.


texPreview and Preview functionality

{texPreview} has changed some and it should make things easier on our end. There's now an S3 method actually in the package for our equations.
The only think is the new mathjax argument, which has to be set to FALSE. Wondering whether we should just write a quick wrapper on the existing S3 method to do that.
m <- lm(bill_length_mm ~ species*bill_depth_mm, palmerpenguins::penguins)
eq <- equatiomatic::extract_eq(m, mathjax = FALSE, wrap = TRUE, terms_per_line = 2)
texPreview::tex_preview(eq)

I had a bit of an issue with the margins too though. Not sure what's going on there.
One other thing that is kind of neat is there's a new texpreview knitr engine. See here

Ooh, since texPreview handles the equationmatic S3 class, do we need our own preview.equation() anymore? Especially now since there's no mathjax argument, it seems like our equations can just feed right into texPreview::tex_preview(), and there's no reason to use our own preview.equation(). Ours is really just a wrapper for texPreview::tex_preview anyway.
 So going forward we could either (a) just go ahead and declare it as a dependency and import texPreview::tex_preview, so the user wouldn't have to load it on their own, or (b) just cut it from our package altogether and keep it in Suggests, with examples in the documentation.
I like having the preview functionality in the package but I think I'd probably lean more toward (b) because of magick and the other dependencies that are part of texPreview. Thoughts?

I vote (b) and just keep it in Suggests—that way we can focus on the TeX construction and leave previewing and rendering to other packages. I'll make the changes!

Perfect!

(realized I probably should let you close it after you make the changes)


Fix: import gdnative prelude in lib.rs template

The default lib.rs template has a compiler error with newer versions of the gdnative library. Importing the Variant type is now required to run the godot_init macro. I decided to include the prelude instead of just importing Variant to hopefully prevent these API changes from breaking things in future versions.
Test Plan:
ftw new test
cd test
ftw build

Error Output:
error[E0433]: failed to resolve: use of undeclared type `Variant`
  --> rust\src\lib.rs:13:1
   |
13 | godot_init!(init);
   | ^^^^^^^^^^^^^^^^^ not found in this scope
   |
   = note: this error originates in the macro `$crate::godot_nativescript_init` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider importing this struct
   |
4  | use gdnative::core_types::Variant;
   |

For more information about this error, try `rustc --explain E0433`.
error: could not compile `example` due to 2 previous errors
 ERROR: The system cannot find the file specified. (os error 2)
Error: ()

lib.rs
mod game;
mod spinning_cube;

use gdnative::prelude::{godot_init, InitHandle};

// Function that registers all exposed classes to Godot
fn init(handle: InitHandle) {
    handle.add_class::<game::Game>();
    handle.add_class::<spinning_cube::SpinningCube>();
}

// macros that create the entry-points of the dynamic library.
godot_init!(init);

Hi @aclave1,
Thank you for filing this issue, may I know what target, rust version, ftw version and the operating system you are running ftw on? Based on my initial test, I seem to don't get the error that you are getting. Please see screenshot below.

I'm on NixOS, x86_64-unknown-linux-gnu target, rust v1.64.0 and ftw v0.10.0
Mike

I'm on windows 11, target is x86_64-pc-windows-msvc, rust 1.63.0, ftw 0.10.0. One thing I see from your logs is the usage of gdnative v0.11.0. The version of gdnative that mine is using is 0.10.1. This is a project generated by ftw new.  Maybe my local copy of godot-rust-template is cached?


Migrate article: Migrating to Aiven for PostgreSQL using Bucardo

Migrate the content from this article to fit the documentation portal format https://help.aiven.io/en/articles/5244801
(Assign this issue to yourself before you start, and remember to mention the issue number when you open your pull request)

Resolved via https://github.com/aiven/devportal/pull/1249.


Move scala_test/scala_test_suite to their own file

This PR moves scala_test/scala_test_suite to its own file, per #808
I relocated sanitize_string_for_usage from scala/scala.bzl to scala/private/common.bzl for now and loaded it in scala/scala.bzl, but as things move into private, these functions will only be used in private.
As I get more familiar with the little private library of functions rules_scala has, perhaps organizing them a bit better like how rules_haskell does for its own private lib would make sense, dunno yet.

@ittaiz Nope, I haven't touched any implementation thus far, only moves, and one of my motivations for doing such refactoring is to make it less intimidating to look at large monolithic files while trying to make real meaty improvements!
Checking the test suites, the results are the same as before and ✅.

Will merge on green

Don't think the build will be green in this case since the Windows build has some unrelated sadness it seems

I took a 10 minute glance at the Windows build and couldn't figure out what might be wrong since I don't know enough there. Windows support is unfortunately harder without a local env too.

I think we need to disable windows CI unfortunately. Between the beta status of travis CI on windows, and less support from bazel, it is making our CI very noisy.

@johnynek I stopped short of suggesting so but I think that's probably the right move for now, the resources for supporting Windows are scarce.

we can remove suites attribute on a follow up.


Add support for multiple forecast hours in input dataset

Closing because the size of the data files in doing this severely slows xarray, may revisit later.


Add trace observer/feedback + Rocket and Spike support

In this change, we added support for parsing and involving execution traces as an observer and feedback.

Thanks for the contribution! Except the 3 warnings from Clippy it looks all good to me


Support React 17

Overview
Hello! I was curious if React 17 support was on the roadmap to be added to react-codemirror? Thanks!
Package versions
react-codemirror version: @latest

Any updates?


Initial connection code refactoring and fixes

Remove unused class methods
Breakup large function in ConnectDialogContainer
Remove unused connection dialog states
Remove unused dialogs
Complete "micro:bits connecting" placeholder dialog

LGTM, I have left a few minor suggestions for further refactoring

Thanks. Will merge this and come back to those.


Convert to using CumulusCI 2.0 for CircleCI builds

And configure the repo for 2.0
Critical Changes

This is a sample Critical Change for the release notes

Changes

This is a sample Change for the release notes

Issues Closed
Fixes #8

Included in beta release 1.2 (Beta 102)


Json format in Spring Boot Logs

Is there a way to "print" logbook.format.style json with new line that json would be readable?

There is a PrettyPrintingJsonBodyFilter


Connecting to AWS Elasticache cluster using cluster endpoint

Hello,
AWS EC exposes 2 ways of connecting to multi-node cluster.
It gives us a cluster-endpoint AND it also gives us endpoints for each node.
Now, in redisson I see that there's a cluster connection config which requires each of the node endpoint address alongwith replicase i think.
And then there's singleServerConfig.
I was wondering what's the best way to handle this? If I use single server config with cluster endpoint, will it be okay?
Or I always need to use cluster connection config with all node endpoints registered.

you can use endpoint with AWS EC

thanks @mrniko for confirmation.


Research: Make the library multiplatform

Since DataStore became multiplatform library in 1.1.0, it would be great to have multiplatform solution for its encryption.
Current JVM dependencies of my library are:

security-crypto — might be dropped because of #31
Tink — has implementations in C++ (Native), Obj-C (Native/iOS) and Go (Wasm). TypeScript implementation is not going to be released anytime and Google recommends to use WebCrypto API instead, however it might be more reasonable to use Wasm module from JS.
JVM Input/Output streams — could be replaced by Okio

References

Wasm target:

In-browser HSM-backed Encryption with Tink and Wasm
Go wiki: WebAssembly

KMM bindings for Tink (outdated, iOS/Android only)

The preparation step is done:

https://github.com/osipxd/encrypted-datastore/pull/77


Wrong pod version installed

Description
today a new GoogleMLKit Version was released: 4.0.0
This breaks our build, because the latest version of this plugin has no explizit version set in the pod spec file.
It should be set like this:
https://github.com/bharat-biradar/Google-Ml-Kit-plugin/blob/5275654fa54e2d1aad027e8afc943a9d21e4261d/packages/google_mlkit_text_recognition/ios/google_mlkit_text_recognition.podspec#L18

Thanks for mentioning the issue and providing a solution!
I just published the version 4.1.1 which solves the problem.

Awesome, thanks for the quick fix! 😊


Fix: Wait until the last file is done bundling before starting Electron

In a setup with multiple entry files (for example, a preload and a main), the plugin would start Electron once the first one is done bundling. This is a problem, since the application might then not start at all / execute outdated code
With this PR, it now waits for all files to be done before launching Electron
I've had this happen in a repo with a fairly large main entry, compared to a smaller preload. The preload would finish, the plugin would try to start Electron before main is done; because main.js then didn't exist at all, Electron would immediately exit with an error, which in turn also cancelled the main build. Only manually creating the main.js file (with an utility like touch) would actually make the build work, which is rather cumbersome
Please note that this is my first time working on Vite plugins, please let me know if I've made any obvious mistakes. Thanks!

👋 Thx for your PR!
Regardless, you can customize the startup timing via the onstart option.
A simple way to determine the length of an array is inaccurate, as users can define multiple entry files in one entry.

Regardless, you can customize the startup timing via the onstart option

Right, but since the plugin aims for simplicity (and this is arguably a design flaw), I think it should do the "correct" thing by default

users can define multiple entry files in one entry

I see

Is this a supported use case? Doesn't seem to be mentioned anywhere in the README
As far as I can see it, this results in just one closeBundle event being fired (while we also just wait for one), so everything works out

There may not be any design flaws, I just want the plugin maintenance cost to be low enough.
import electron, { onstart } from 'vite-plugin-electron'

let largeReady = false

export default {
  plugins: [
    electron([
      {
        entry: 'electron/main.ts',
        onstart() { // Automatic startup can be prevented.
          largeReady && onstart()
        },
      },
      {
        entry: 'electron/preload.ts',
        onstart() { // Automatic startup can be prevented.
          largeReady && onstart()
        },
      },
      {
        entry: {
          foo: 'foo.ts',
          large: 'large.ts', // A large file here
        },
        vite: {
          plugins: [
            {
              name: 'custom-start',
              closeBundle() {
                if (!largeReady) {
                  largeReady = true
                }
                onstart() // Delayed start
              },
            },
          ],
        },
      },
    ]),
  ],
}

Oh I'm perfectly aware how you'd solve this on the user side. I'd just think that a build tool reporting that the build is done when it is, in fact, not done is a flaw of that build tool and shouldn't be up to the user to fix
Not counting the comment/whitespace, this is a 4 line patch. How much added maintenance cost are you expecting here?

Although Electron will start early before a large file (Preload) is built, when the large file is built, Electron will be restarted by default and will still work normally.
If you prevent "restart" via the onstart option, this behavior itself is uncontrollable.

when the large file is built, Electron will be restarted by default and will still work normally

That's the original problem: It isn't. If your larger files is Electron's main entrypoint (which it is in 99% of cases), your preload (or any other file that you've specified) will finish building, the plugin will then try to start Electron, which will then fail (as noted in my comment). This will then cancel the whole build, meaning no main.js will ever be built. The only way to get a project like this to start would be to manually create the output file
Once you've built a project once, it's true that Electron will restart once the build is done. However, this still leads to unnecessary "work" (starting Electron, only to restart it a couple seconds later) and potential issues (you're executing outdated code for that brief amount of time)

If you prevent "restart" via the onstart option, this behavior itself is uncontrollable.

Hm, I think I see what you mean here: Checking for equality in the condition breaks hot restart. This wasn't intentional. Should be resolved now

@CommandMC Hey 👋
I was pushed a PR here 👉 https://github.com/CommandMC/vite-plugin-electron/pull/1


Feature: The ability to fold historical events into one another (to keep the Event Store size managable)

This one is perhaps not a direct feature request but more of a concept I am thinking about and I thought I might share it here. Perhaps it doesn't really fit into the design and philosophy of Eventuous, but I'm curious what others might think about it.
I am currently building a multi-tenant cloud product that will use an Event Store (PostgreSQL) as one of the components within the system. Multiple applications will have subscriptions to this Event Store (and will also each push Aggregates). The requirements demand that the Event Store is a historical system, since other applications might be added to the product later and these will need to be able to subscribe to the Event Store and build a full read model (including Aggregates that have not been updated in years). This means that I cannot use the 'Archive' feature as it is described in this  section of the documentation, as the documentation warns against it:

To take the use case from the documentation as an illustration: certain products might not care about the history (the Booking system is not interested in Bookings from five years ago) but for my product the history is a requirement. But perhaps not the entire history (see below for my idea).
My concern is that, especially because the product will be multi-tenant, the Event Store will grow to a huge size and this will impact the performance (especially in the case described above where a new application is added and it will have to catch-up the entire history. That could eventually stretch into an import lasting days). We are already discussing mitigating strategies, such as for instance having a separate Event Store for each tenant (which is a good idea in any case), but even then I am expecting a great number of events for each tenant. I might further subdivide the data for a single tenant into multiple Event Stores (for instance, splitting the Aggregates 50-50 between two stores) but this would be difficult to configure/implement later on if the system was already taken into Production, and it would be hard to predict beforehand which domain Aggregates are likely to receive the most events (which would make for instance a 50-50 split basically a shot in the dark). And even with all mitigating strategies in place (1 Event Store for 1 tenant with only 1 of several domain sections from an application), the nature of a historical system means that twenty years from now, there will still be a huge history.
However, my requirements only demand that a new application must rebuild the current state of the Aggregates; each individual event from five years ago (configurable, of course) is not interesting, only the result (current state). So I am thinking about a concept of 'folding' old events into a single event that contains only the end result at that time.
An example (using Create, Update and Delete events and Entities instead of Aggregates since I don't (yet) use true DDD in the product):
jan 1st, 2020: Entity 1 Created (Name=Demo)
sep 1st, 2021: Entity 1 Updated (Description=Later Update)
jan 1st, 2022: Entity 1 Deleted
Execute the 'fold' over the Event Store with jan 1st, 2022 (midnight) as the parameter provided (i.e. all history before that date may be folded).
Result:
jan 1st, 2022: Entity 1 Created (Name=Demo, Description=Later Update, HistoryFolded=jan 1st, 2022)
jan 1st, 2022 (but later; timestamp excluded for brevity): Entity 1 Deleted
The HistoryFolded field (or something like that) would tell a consuming application that no historical events are known from before jan 1st, 2022. This would be enough for the needs of my product.
The reason I want to retain a Created event is twofold: one, I need a place for the HistoryFolded field, and two, one of my consuming applications is interested in retaining some fields even for deleted Entities (so the user might for instance be shown a view with deleted entries: "Entity 1, Named "Demo", was deleted on feb 1st, 2022." ). In other words, my consuming applications might still be interested to know that there once existed an Entity named Demo, but it has since been deleted.
I could implement something like this by creating an application that reads and folds events from Store 1 and write my 'folded' Entities to Store 2, but this would still require downtime (in order to switch all applications over from using Store 1 to Store 2; and you would technically have to turn the entire system off during the operation in order to avoid missing new events that were written to Store 1 after a 'fold' has already been executed. i.e. no application can append events to Store 1 while a fold is occurring). This would not be my chosen solution.
Could Eventuous possibly support something like a folding feature, or have I just pitched one of the cardinals sins of Event Sourcing (I am still learning the concept, and have not read all there is to read about the topic)?
One of the concerns I could identify would be, what would happen to a subscription that is currently (re)building a view model while a stream is being folded? You can't really shut down subscriptions at runtime. So you can't really get around the issue of requiring downtime.
I'm curious to see what others might think about this.

It might be related to logical or physical snapshots. The logical snapshotting is the easiest to implement as it's just an event. I think you can do it already now. Just fold the state to an event-like object and apply it as normal. Add a handler to the state record to apply the compacted state event to the state itself (basically, replacing all the information in the state).
The only part is the stream truncation that you'd need to do manually. I still haven't decided how to automate it, but the initial discussion happened in this old issue https://github.com/Eventuous/eventuous/issues/82

There is an issue currently with the snapshotting implementation (like the one you've described). It affects the Optimistic Concurrency. Currently, OC depends on these props:
public int OriginalVersion => Original.Length - 1;
public int CurrentVersion => OriginalVersion + Changes.Count;

This way Aggregate always requires you to have the entire event history to keep the Original version in sync. So even if we have a snapshot event with the current state, we still need to load all of the previous events to have a valid array Length.
We just got a need to implement snapshotting and stopped here.
Maybe it's a good idea to have OriginalVersion virtual or at least internally changeable? This way we can store the position in the snapshot, read only the snapshot and events after the snapshot, and restore the position during the snapshot unfold. It will keep ESDB optimistic concurrency control happy.

The versioning issue can be fixed, it's not a huge problem. I would suggest getting back to this particular issue when snapshotting is available.
About the issue itself, it's not very clear what the goal is. Is it to save disk space? Or is it for shortening streams to speed up the load operation? If history events are combined to one event, how does it play with the concept of immutable append-only log? How would replays work for new read models?


[add] webmail address : joruri mail format import

Coverage decreased (-0.02%) to 80.537% when pulling 7db8ba52be456a07ae900be1467acefc769718db on itowtips:dev/gws-export-address into 70271eb5b8c747f6b2b03c1886e475e89246e87e on shirasagi:dev/gws.

Coverage increased (+0.2%) to 80.738% when pulling 696aa8003f5fc61cb4613f5d02def126fb25030a on itowtips:dev/gws-export-address into 70271eb5b8c747f6b2b03c1886e475e89246e87e on shirasagi:dev/gws.


Focus loss with project context

Motivation
Allows memes set to the Focus Loss dismissal option to be dismissed only when focus is lost in the project it originated from.
In other words, AFK gifs from other open projects don't go away when you come back to an unrelated project.
Probably what #68 wants

Issue number 69


Inventory file for static docker containers

Like the inventory.yml file we need a file which keeps track of the docker containers hosted on the dockerhost machines. It should also keep track of the details of the container, for example its port and its name in jenkins node name

Moving to 2Q plan when its created

https://github.com/adoptium/infrastructure/pull/3423 is finalised. Repeating what I mentioned here, we need a system in place to keep this file updated, ie an automated github action which runs the script, if it detects differences between the output and the existing DockerInventory.json file, it makes a pr to update it

Docs pr is in https://github.com/adoptium/infrastructure/pull/3537


Pricing Scenarios

Given the multitude of potential backend providers, it is desirable to develop a system for comparing operating costs across providers.
Expected Behaviour
Ideally there would be cost estimates for network, compute and storage bound functions.
A simple experiment on each provider should be able to provide function costs.
Assuming operating costs could be parametrized, a pricing calculator could be constructed to assist users in selecting the backend provider best suited for their use case.
Current Behaviour
Currently, it is difficult to understand what the operating cost will be due to differences in the infrastructure as a service costs across providers. Since one of the main benefits of cloud functions is their ability to avoid continuous operation costs and capture true utilization costs, it would be excellent if consumers could know the cost of function execution before they purchase the associated service.
Possible Solution
Design a sample service for testing costs for network, compute and storage bound functions.
Perform a test of each service on a given provider and measure the cost.
Report the results, so a model or price calculator can be constructed.
Ideally this could be done without actually costing the tester money, or a given provider could assist by providing compensation for the test.
Context
I'm designing a secure multiparty computation product where consumers of the service purchase executions of a function at the operating cost plus a markup chosen by the service provider. I want consumers to be confident that the markup is not egregious, and have confidence in the transparency of the pricing of the service.
Your Environment

Are you using Docker Swarm or Kubernetes (FaaS-netes)?
Kubernetes

Operating System and version (e.g. Linux, Windows, MacOS):
Linux

Derek add label: proposal

Derek add label: revisit

This would require a significant amount of consulting and analysis - and I suspect it would be a moving target also. Also worth nothing that the spot instance market with AWS, Packet and others could easily provide a much cheaper back-end for some of the stack - particularly functions.

Derek add label: priority/low

While this would likely take a pile of effort, as @alexellis mentioned, it's something that would be pretty cool IMO.
I wonder if this information could be 'derived' through some base pricing, and some more detailed metrics collected around the function execution.
For example, it might not be possible/practical to say the exact cost to run on EC2/similar given spot pricing/changes/etc. But it might be a nice feature (or at the very least a research project/blog post) to compare what running a function on OpenFaaS is like in comparison to AWS lambda/similar providers.
I know of at least one other project (https://github.com/apex/apex) that has done similar, though it was AWS Lambda specific. The following isn't a great example since i've barely run the function.. but..
⇒  apex metrics
  gobuster
    total cost: $0.00
    invocations: 0 ($0.00)
    duration: 0s ($0.00)
    throttles: 0
    errors: 0
    memory: 128

Related:

http://apex.run/#viewing-metrics
https://github.com/apex/apex/tree/master/metrics
https://github.com/apex/apex/blob/master/cost/cost.go

Closing and marking #revisit for follow-up at a later date.


returned field methods to public

Pull request type

Please check the type of change your PR introduces:

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Issue Number: N/A
What is the new behavior?

Other information

Codecov Report
Attention: Patch coverage is 0% with 20 lines in your changes missing coverage. Please review.

Project coverage is 40.88%. Comparing base (25a0667) to head (c5fb45e).

Files with missing lines
Patch %
Lines

log/field.go
0.00%
20 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (25a0667) and HEAD (c5fb45e). Click for more details.
HEAD has 61 uploads less than BASE

Flag
BASE (25a0667)
HEAD (c5fb45e)

unit
9
2

go-1.21.x
6
1

ubuntu
13
0

go-1.22.x
6
1

go-1.23.x
7
0

macOS
3
2

windows
3
0

integration
9
0

ydb-23.3
3
0

ydb-24.1
3
0

ydb-24.2
3
0

ydb-nightly
1
0

experiment
1
0

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #1531       +/-   ##
===========================================
- Coverage   70.93%   40.88%   -30.06%
===========================================
  Files         356      353        -3
  Lines       37490    37272      -218
===========================================
- Hits        26594    15238    -11356
- Misses       9787    21526    +11739
+ Partials     1109      508      -601

Flag
Coverage Δ

experiment
?

go-1.21.x
42.33% <0.00%> (-28.03%)
:arrow_down:

go-1.22.x
40.88% <0.00%> (-27.54%)
:arrow_down:

go-1.23.x
?

integration
?

macOS
40.88% <0.00%> (-0.02%)
:arrow_down:

ubuntu
?

unit
40.88% <0.00%> (-0.06%)
:arrow_down:

windows
?

ydb-23.3
?

ydb-24.1
?

ydb-24.2
?

ydb-nightly
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Would it be possible to integrate libsql?

Hello,
I like the idea of this tool a lot,
I'm recently using Turso(libsql), and would like to use it from this tool,
Thanks.

Use the sqlite connection string for turso local development.


Revert "Merge pull request #1753 from 1e1/locale_fr"

This reverts commit 5c2783971ac101f65f6b3deaa8ec6592bd9e8393, reversing changes made to 3dcc978c9c865985e4b365ab14e162e2216bda5f.

I had to revert the PR because honnestly, that was really bad...


List should show timestamp for recent files instead of current year

First of all, when I implement list, I read, that there is no standard for list format. So there is nothing good and nothing bad with this pull. So, it will be accepted, since human readability. :ok_hand:

What is the reason to change datetime.datetime.fromtimestamp to time.time?
Why half of year branching?
More obvious way to find delta is timedelta.days, I think.

it's a copy-paste from asyncssh (https://github.com/ronf/asyncssh/blob/master/asyncssh/sftp.py#L3372), i didn't bother to improve the code - it's simple enough
regarding 6 month it's how *n*x ls -l works, see http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/ls.c#n4015


Copilot-based TS refactors

This PR adds copilot suggestions to various Typescript refactors and quickfixes. It also adds a separate quickfix for implicit any that lets Copilot infer types from the body of a function, which is less context than the TS codefix. However, it creates new interfaces with reasonable names and uses them where appropriate, which the TS codefix cannot.
The existing refactor/quickfixes are followed by Copilot's inline chat now:

Extract to X now asks copilot to provide a name for the new constant/function/type instead of showing the rename UI.
Implement X now asks copilot to provide reasonable implementations for inherited interface or abstract methods. This replaces the TS-generated throw new Error("Not implemented").

I think the the UX and performance are both good enough for insider use at this point, so I think it's worth merging behind the typescript.experimental.aiQuickFix flag.
Some future work:

Could the codefix could be skipped if there are lots of types around already?
Could the codefix prompt by modified to omit "Please generate new interfaces" if there lots of interfaces already declared?
Fix code formatting!

Extends #190690
Question and work from that PR that are now resolved:

Infer from types now works in JS files.
When given enough context, inline chat works well, especially since it will edit code if the AI produces a code sample. I tried showing multiple name suggestions in the chat panel, but I felt like it didn't add enough utility -- and the suggestions were usually all good or all bad, so taking the first suggestion works well enough.

@mjbvz and @jrieken can you take a look at this?

@sandersn There are some eslint failures. You can see them when running yarn eslint (actually the precommit hooks should have reported them...)
I wonder if we should grow the typescript.experimental.aiQuickFix setting, e.g this isn't just quick fixes anymore and also more nuanced. Maybe a new setting like typescript.exprimental.aiCodeActions which value is a complex object would be best, e.g each key would correspond to a feature

Ah, that's right. I completely messed up the setup of vscode's formatter, so I skipped the precommit hook. I'll go back and figure that out so the machine can fix the formatting for me.

OK, I added a more complex options object. If you just create the top-level object as {}, all the code actions are enabled. Then you can set individual ones to false.
I don't know if this is the usual way to do it, so please let me know if there's a standard I should follow.

I realised that the config still effectively defaulted to true, and that the altered descriptions weren't working.
I fixed that and also created custom descriptions, so please take a look at those for wording.

Merging caused a build failure down the line (when doing the product build...) A revert PR is up (#193810) and this needs to be reopened and looked at again


Minor housekeeping

Removes the old script for rtti-compatible emsdk
Fixes linter warning in build_wasm.sh
Updates wasm/README to point to the right scripts

hmm maybe there's something in here

::::: building @rive-app/canvas_advanced_single

Cleaning project ...
Error: invalid option 'single_file '
Error: Process completed with exit code 1.

test seems to be failing to build ::::: building @rive-app/canvas_advanced_single

I'll try to repro locally, let's see.


Various SERVER connection types

@ivan-stankov-golang it looks good. We got some merge conflicts after merging some of the other PR. Once resolved then we are good to merge.


{bp-15258} gcc/gcov: fix problems with fork

Summary
gcov: Fix gcov fork() issue:

After code coverage is enabled, fork will be replaced by __gcov_fork
gcov: Prevent pile insertion recursion
gcov: Disable stack checking
 When enable CONFIG_STACK_CANARIES, in general, the stack check in the __gcov_fork function is:

" return fork();
18: e59f3020 ldr r3, [pc, #32] @ 40 <__gcov_fork+0x40>
1c: e5932000 ldr r2, [r3]
20: e59d3004 ldr r3, [sp, #4]
24: e0332002 eors r2, r3, r2
28: e3a03000 mov r3, #0
2c: 1a000002 bne 3c <__gcov_fork+0x3c>"
r3 is obtained by taking the value of sp offset. But after opening thumb, the second comparison value in
"8c6: 4a06 ldr r2, [pc, #24] @ (8e0 <__gcov_fork+0x30>)
8c8: 6811 ldr r1, [r2, #0]
8ca: 687a ldr r2, [r7, #4]
8cc: 4051 eors r1, r2"
is obtained through r7. Since r7 stores the stack address at this time, which stores the address of the parent process, the stack out of bounds will occur in the child process

Impact
RELEASE
Testing
CI

[Experimental Bot, please feedback here]
Yes, this PR appears to meet the NuttX requirements, although some sections could be more detailed.
Here's a breakdown of why and suggestions for improvement:
Strengths:

Clear Summary: The summary explains the "why," "what," and "how" of the changes. The mention of __gcov_fork and stack checking provides technical context.
Impact Section:  Marking it as a RELEASE impact highlights its significance.
Testing Section:  Mentioning CI implies that Continuous Integration tests were run and passed.  This is good, but could be strengthened (see below).

Weaknesses & Suggestions:

Missing Issue References:  Even if there isn't a directly related issue, it's good practice to briefly state that (e.g., "No related NuttX issue").

Impact Section Too Brief: While RELEASE indicates a significant impact, the other fields should still be explicitly addressed.  Even if the answer is "NO," state it explicitly.  For example:

"Impact on user: NO"
"Impact on build: NO (unless CONFIG_GCOV is enabled)"  (This is an assumption, but illustrates the level of detail desired)
"Impact on hardware: NO"
"Impact on documentation: YES (Documentation should be updated to reflect the changes to fork() behavior when gcov is enabled)"
"Impact on security: YES (The fix for the stack canary issue addresses a potential security vulnerability)"
"Impact on compatibility:  NO (Expected to be backward compatible)"

Testing Section Lacks Detail: "CI" is a start, but insufficient. Specify:

Build Host(s):  The operating system, compiler, and architecture of the CI environment. (e.g., "GitHub Actions Ubuntu Latest, GCC 11.x, x86_64")
Target(s): The architectures and boards tested on CI. (e.g., "sim:qemu-x86_64, stm32f4discovery:nsh")
Missing Logs: The "Testing logs before change" and "Testing logs after change" sections are empty.  While full logs might be excessive, include snippets that demonstrate the issue being fixed and the successful resolution.  Even a simple "before: segmentation fault," "after: program completes successfully" would be better than nothing.  Ideally, show how the code coverage data is now correctly collected after the fix.

Example of improved Testing Section:
Testing

I confirm that changes are verified on local setup and works as intended:
* Build Host(s): GitHub Actions: Ubuntu Latest, GCC 11.x, x86_64
* Target(s): sim:qemu-x86_64, stm32f4discovery:nsh

Testing logs before change:

qemu-system-x86_64: terminating on signal SIGSEGV

Testing logs after change:

nsh> gcov_example # Example command demonstrating the application works after the change
... output showing program completion and potential code coverage statistics ...

By adding this level of detail, the PR becomes much stronger and easier for reviewers to assess and approve confidently.


A new tagging build UI has been added.

It allows engineers to mark a build as Bad, then all the subsequential builds will also be marked as bad until the engineer makred one build as good. Also a warning message is added when engineers try to deploy a bad build. Detailed screenshots can be found in https://docs.google.com/document/d/1hlZwmlXvHxqp51LOPQc4OBRv44qwQLeqQoZaoelJfm0/edit
The details of changes include:

Add UI to show the tag in variant pages  of builds, environment.
Service API got two changes:
a. Use Jackson instead of Gson so that the metainfo deserialization respects jsonproperty on beans
b. API changes to search search by commit.

@sbaogang @yujunglo

Look good to me r+


Fix some compiler warnings pointed by clang. v2

Changes since v1.

On  36dbea2 - Now uses sol_str_table_fallback()
On 88e8af1 - Removed r = -ENOMEM from sol-bus.c, since there's a patch fixing the problem.

:+1:

merged


refactor: update method name

refactor: change method name

:tada: This PR is included in version 0.28.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Online/Streaming mode

Question
Is there any official API/doc/example to use flashlight decoder in streaming mode?

@naymaraq — if you're using the CTC decoder, you can pass data into decodeStep incrementally, and the decoder will behave in a streaming fashion. Streaming isn't currently supported for the autoregressive decoder, although that may change down the line.
There isn't documentation for streaming as it stands, but I hope to add some soon. As always, feel free to contribute back.


Question: How to comfortably parse tokens with metadata

To allow me to show the exact position of issues I added start and end fields as indexes to the source into my tokens:
#[derive(Debug, Clone, Hash, PartialEq, Eq)]
pub enum TokenValue<'a> {
    Assign,
    BitwiseAnd,
    ...
}

#[derive(Debug, Clone, Hash, PartialEq, Eq)]
pub struct Token<'a> {
    pub value: TokenValue<'a>,
    pub start: usize,
    pub end: usize,
}

When I do that i seemingly have to use either select! or filter for every single token when parsing those into an AST:
let assign = select! {
    Token {value: TokenValue::Identifier(ident), start, ..} => (ident, start)
}
.then(filter(Token::assign()))
.then(Expression::parser()).then(filter(Token::newline()).or(filter(Token::semicolon())))
.map(|((((ident, start), _), expr), Token { end, .. })| ASTNode {
    start,
    end: expr.end,
    value: Statement::Expression(ASTNode {
        start,
        end,
        value: Expression::Assign(ident, Box::new(expr)),
    }),
});

Is it possible to do this in a better way or is my whole approach flawed? If so, can you show me a better approach?

The term you're looking for is 'span', and chumsky is designed from the start to have first-class support for them. Check out one of the many examples that demonstrate the use of spanned tokens.

i see what you mean now, tho the specific usage feels quite hard to decipher, as the examples integrate it with a lot of stuff at once and the documentation on this topic is currently quite sparse

You can find more information in the guide and the various doc examples.


Please help me what went wrong here..

Aww shucks.
Something completely unexpected happened. This is a bug in cf.
Please file this bug : https://github.com/cloudfoundry/cli/issues
Tell us that you ran this command:

	/usr/bin/cf-orig/cf push InsuranceDigest

using this version of the CLI:

	6.7.0-IDS

and that this error occurred:

	runtime error: invalid memory address or nil pointer dereference

and this stack trace:

	goroutine 16 [running]:
main.generateBacktrace(0x0, 0x0)
	/root/gocode/src/github.com/cloudfoundry/cli/main/main.go:147 +0xcf
main.handlePanics(0x7f42d379e990, 0xc208043900)
	/root/gocode/src/github.com/cloudfoundry/cli/main/main.go:131 +0x15e
runtime.panic(0x9f9d00, 0x19249b3)
	/usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/cloudfoundry/cli/cf/net.Gateway.performRequestForResponseBytes(0x7f42d37a39e8, 0xc20804f830, 0x167d190, 0x1, 0x12a05f200, 0x0, 0x0, 0x0, 0x7f42d37a37c8, 0xc20807ae40, ...)
	/root/gocode/src/github.com/cloudfoundry/cli/tmp/cli_gopath/src/github.com/cloudfoundry/cli/cf/net/gateway.go:248 +0x2e4
github.com/cloudfoundry/cli/cf/net.Gateway.PerformPollingRequestForJSONResponse(0x7f42d37a39e8, 0xc20804f830, 0x167d190, 0x1, 0x12a05f200, 0x0, 0x0, 0x0, 0x7f42d37a37c8, 0xc20807ae40, ...)
	/root/gocode/src/github.com/cloudfoundry/cli/tmp/cli_gopath/src/github.com/cloudfoundry/cli/cf/net/gateway.go:287 +0x2a9
github.com/cloudfoundry/cli/cf/net.Gateway.createUpdateOrDeleteResource(0x7f42d

Build step 'Execute shell' marked build as failure
Finished: FAILURE

@paranigit - please try updating your version of the CLI and verify whether the error reproduces.  6.7.0 is a pretty old.  Newest versions are here:  https://github.com/cloudfoundry/cli/releases

@paranigit It appears that the CLI upgrade resolved the issue.  If not, please reopen the issue and we will take another look.


Dedupe identical joins

Consider you have two optional where statements you need add for a query, both requiring the same join, but you don't want to add the join if it's not needed. It's doable like this
let qb = ctx.db.selectFrom('person')
let filterA: string | undefined = 'a'
let filterB: string | undefined = 'b'

if (filterA) {
  qb = qb
    .innerJoin('pet', 'person.id', 'pet.owner_id')
    .where('pet.name', '=', filterA)
}

if (filterB) {
  qb = qb
    .innerJoin('pet', 'person.id', 'pet.owner_id')
    .where('pet.name', '=', filterB)
}

const res = await qb.selectAll('person').execute()

But it produces two identical joins if both filters are applied and causes an error in the db. You could do this
let qb = ctx.db.selectFrom('person')
let filterA: string | undefined = 'a'
let filterB: string | undefined = 'b'

if (filterA || filterB) {
  let qb2 = qb.innerJoin('pet', 'person.id', 'pet.owner_id')

  if (filterA) {
    qb2 = qb2.where('pet.name', '=', filterA)
  }

  if (filterB) {
    qb2 = qb2.where('pet.name', '=', filterB)
  }

  qb = qb2
}

const res = await qb.selectAll('person').execute()

but is that enough?
Suggested solution
The first example would work if kysely ignored identical joins (but the types would still change of course).

Implemented as a plugin. See this recipe


Reconsider in-app dark/light mode preference

Spot has a preference that lets you choose between dark mode / light mode / follow system. GNOME's app styling guidelines say this:

Most apps should use the standard light UI style by default. However, apps can alternatively choose to use the dark style by default instead. This is primarily recommended for apps which display rich visual content like images or video.
[...]
Per-app style preferences are primarily useful for text editing apps, or apps which users use for long periods of time. In these cases, the style preference can be useful as a way of controlling visual contrast and brightness.

I think we should reconsider if Spot needs this preference. It doesn't do any harm, but every preference has a cost, and consequently every preference should have a purpose, a clear reason to exist. I don't see many reasons why a user would want Spot's color scheme to deviate from the rest of their system.
One possible reason a user would want this preference, is because the album art stands more out in dark mode. Even if we consider this enough to warrant a preference, we don't really need an "Always Light" preference. I suggest that we at least simplify this to a toggle for "Always Dark":

I think maybe spot could adapt something like this.

Yes, but what purpose would it serve?

Personal opinion on this: what we have at the moment is... fine?
Judging by the large amount of theme-related issues in the past, it's an option people care about, it's easy enough to support the three options, and there are more complex options in the settings anyway. I'd leave it as it is and maybe just change the default to follow the system theme so it's not too jarring the first time you use the app.

I'll close this one for now unless there's a good reason to change this if that's fine with you all!


Output path to javascript files via craft.vite.entry

Question
I have a need to output the build versions of select CSS and Javascript files from a site. I have the CSS contents output with craft.vite.inline(craft.vite.entry("file.css")), but attempting to do the same for a corresponding Javascript file nets a return of just the development site base URL when craft.vite.entry("file.js") is called. The docs suggest that craft.vite.entry supports resolving a path to a built Javascript file, unless I'm misinterpreting.
Additional context
Snippet of my vite.config.js:
build: {
            emptyOutDir: false,
            manifest: true,
            outDir: './web/',
            rollupOptions: {
                input: {
                    about: '/_src/vite/about/index.ts',
                }
            },
        },

Example contents of one of the Typescript files that goes into the rollup config:
import '../scss/about.scss';

import '../js/about.js';

Thanks!

I'm curious about the use-case here? Is there a reason you can't use craft.vite.script() which will include the JS as well as the CSS?

Will re-open if more information is provided.


Add geometry to chattanooga.json to complete coverage.

Should complete coverage info for chattanooga.json.

Thanks!


Accessing non-SSL via IP yields INCORROUT

When the non-secure site is accessed via IP, a typical situation in initial setup when DNS/names are not available, the page does not render:

No topline header or menu
Garbage in the "GO!" button

In the nonssl.conf, the definition of the non-secure site is selected using
<VirtualHost {{ guest_install_hostname }}.{{ cluster_domain_name }}:8080>

So it will only ever apply if the site is accessed using the exact host name.  If not, the default definitions apply and the SSIs do not get processed.
Changing to
<VirtualHost *:8080>

will allow for the site to be viewed properly no matter what address/name is used to access.

Reopening.  Turns out that accessing the RHOCP install-time content is affected slightly by the config in nonssl.conf.  Specifically, the FilesMatch that blocks PHP and certain HTML files overrides the Options Indexes specification, which means that the bootfiles and install sections cannot be indexed.

Added <Directory> stanzas that enable the required Indexes and FollowSymlinks options for bootfiles and install (and also disabling the SymlinksIfOwnerMatch option for install).  To fully enable indexing however it was necessary to add a <FilesMatch> for index.php, even though there is no such file.


Implement Victory Containers

Implement the Victory Containers, like zoom and voronoi, in victory-native.

Here are my notes on using react-native-svg events (definitely the most straightforward approach) for this feature:

There is very poor multi-touch support in react-native-svg. There is only handling for the beginning and ending of presses, as they use RN’s Touchable elements. There is therefore nothing analogous to onWheel that we heavily depend on.
The onPressOut event fires on a timeout, before the touch is finished (if it’s over a second or so). This makes it very hard to handle long interactions like zooms or drags.
Svg elements cannot handle any events. Therefore the “background” of the chart cannot be interacted with as is. To get around this, we could add a G and a Rect inside of it, along with the children. All events would be added to the G, which would pass them on to the Rect and other children. The Rect would be made transparent and would simply catch events.

Therefore I think we should not use react-native-svg events for containers. This means we’ll have to use RN’s PanResponder or Gesture Responder System. I’ll dive into those next.

That sounds right. You should be able to reference the native version of VictoryZoom. Jani used PanResponder there. The part I'm really worried about is react-native support for https://github.com/FormidableLabs/victory-core/blob/master/src/victory-util/selection.js#L16. If it doesn't exist we'll have to figure out the best way to replace those methods for converting touch position to svg coordinate space.

@boygirl I completely understand. From my searching I think we'll have to create our own touch-to-svg mapping functions.

Shoot, I found an extant issue with VictoryZoom on master; the VictoryZoom component seems to screw up the x-position of the child chart.
Chart with and without VictoryZoom:

        <VictoryZoom>
          <VictoryChart>
            <VictoryBar/>
          </VictoryChart>
        </VictoryZoom>

        <VictoryChart>
          <VictoryBar/>
        </VictoryChart>

Re: transforms... We'll be able to use the RN NativeMethodsMixin's measure functions https://facebook.github.io/react-native/docs/nativemethodsmixin.html#measure on the SVG element itself.

We'll also need to accommodate for this (common) situation: using a VictoryContainer somewhere inside of a ScrollView. Because the ScrollView will "steal" control from the VictoryContainer when swiping up and down, it's prop scrollEnabled needs to be set to false while the interacting with the Container (e.g. during panning or selection).
I think we should add two events to VictoryContainer: onInteractionStart and onInteractionEnd (names are negotiable). Inside of these two events, the user can put handling to respectively disable and enable the ScrollView. Does that sound like a decent solution, @jevakallio?
<VictoryChart
  containerComponent={<VictoryContainer onInteractionStart={...} onInteractionEnd={...}/>}
>
  <VictoryBar/>
</VictoryChart>

@chrisbolin that sounds good. The common pattern in these cases (at least in my own codebases :)) is something like following:
<ScrollView scrollEnabled={!this.state.scrollLocked}>
  <ChildComponent
    onInteractionStart={() => this.setState.bind({scrollLocked: true})}
    onInteractionEnd={() => this.setState.bind({scrollLocked: false})}
  />
</ScrollView>

So this pattern would fit well.
Having a child GestureResponder steal the touch handling from the ScrollView is something that's been planned for a long time, but in the meantime this has been necessary.

Hi everyone, sorry for interrupting, but i was looking for information about voronoi, because i need to use it in a victoryLine in my app and i found this thread...
So, voronoi isn't available for native yet?
Sorry for interrupting again, hope you're having a good day, and thanks for the amazing work!
Greetings!

hey @mauriguz! the quick answer: there are Voronoi tooltips in victory-native that you can use right now.
The longer answer: Victory-native uses VictoryVoronoiTooltip. When this issue is completed, there will be a new and preferred method: VictoryVoronoiContainer. But if you want to use them now, I would go with VictoryVoronoiTooltip. Check out the demo.js for an example using it (it's near the bottom, right under the Tooltips section).

merged. will release after #81, https://github.com/FormidableLabs/victory-docs/issues/276, and https://github.com/FormidableLabs/victory-docs/issues/277 are resolved


Download only work with svg

Hello!
I have an issue with the download feature. It works only if I chose svg as downloadUrlFileFormat. Anything else will give me a "data:," when I console.log the event.detail.url on downloadUrlGenerated.

Did I miss something?
Thank you for this amazing plugin btw!

Hey, I tried recreating your issue but failed. Are you sure you're using the latest library version?
I'm using Svelte 5.

Ok I think I found the issue. I used the "isResponsive" property.  I'll resize the qr code without this one!

Wow that's a bit weird, because I was able to have it worked with that property on.

Hmmm the test you mentioned download a svg. If I add downloadUrlFileFormat="png" in the exact line you mentionned, it doesn't work anymore

This only works because on the same page you already have another QRCode that will generate the PNG.
If you create a svelte component with only your test and add downloadUrlFileFormat="png", it will not work...
I just tested it on my side and your component work until I delete this :
`With download url generated for PNG

 handleDownloadUrlGeneratedPng(event.detail.url)} />

{#if downloadUrlPng}
Download QR Code in png format
{/if}`

Take your time, now that I understand the issue, the workaround is create another QRCode with the png format and hide it 😁

I published a test project on https://github.com/etheonor/test-qrcode
in src/routes/test you have a simple page with two qr code, one with a isResponsive tag and the other without.
If you load the page with both QR code, the download link will work for both of them, but if you delete the second one, the first QR code (with isResponsive) will not have any data to dl.