Failover taking too long

Discussed in https://github.com/cloudnative-pg/cloudnative-pg/discussions/6146

Originally posted by albionb96 November 21, 2024
Hi guys,
we were using another solution for our database until now, and lately we wanted to have a look at the coudnativePG solution.
The first thing that we tested was the failover time.
We killed the Kubernetes node where the cluster primary was living in order to simulate a node failure, and we observed from a client for how long this client cannot send inserts into db.
With our old solution we had a failover time between 25-35 seconds (during this time the application was not able to insert data  into db).
When executing the same failover tests with cloudnativePG we got a failover time somewhere between 40-80seconds.
(If the controller manager pod would be living in the same node like the primary this time could be much longer.)
What we noticed is that the WAL receivers are timed out exactly after 5seconds but it takes some 30-37 additional seconds  (which sums up to 35-42 seconds after the node failure, if we calculate the 5s timeout we had for wal receivers) for the cluster to even just start the failover process in the first place. At least we do not see anything in the logs related to the failover during this time. And although we tried to modify some of the config parameters we didn't achieve better results.
I thought to myself that maybe this time might be impacted from something hardcoded, and after looking a little bit at the source code I doubt that this time is coming from the cluster status timeout defined here, but I am not a golang dev or expert, so I don't know:

https://github.com/cloudnative-pg/cloudnative-pg/blob/72e2f5c490fd17346f1f9ff539589a8d9f01690c/internal/cmd/manager/instance/status/cmd.go#L121
Somewhere I read that in case of node failure the cloudnativepg looks at the node status if it is not-ready or unreachable, and then based on tolerations decides to kill the primary pod and start the failover process. If that is true I now that by default the node-monitor-grace-period is 40s that means that it takes 40s for the kube controller manager to mark node as unhealthy, and that config cannot be adapted. If that is true why is the cloudnativePG relying on that and not triggering the failover process based on its cluster status results/timeouts?
My questions is: Is it possible to reduce somehow the failover time when the node of primary is killed or fails? Can we achieve the old range we had 25-35 seconds? Any idea?
Some more details:
Below we see the logs from the client which was not able to write to db for around 51 seconds during the failover:
Connection is Valid: 2024-11-19T15:02:17.141189046+01:00Sql: INSERT INTO time (time) VALUES (to_timestamp('2024-11-19T15:02:17.141189046+01:00', 'YYYY-MM-DDThh24:mi:ss')::timestamp with time zone at time zone 'Etc/UTC'); Connection is Valid: **2024-11-19T15:02:18.141194779+01:00**Sql: INSERT INTO time (time) VALUES (to_timestamp('2024-11-19T15:02:18.141194779+01:00', 'YYYY-MM-DDThh24:mi:ss')::timestamp with time zone at time zone 'Etc/UTC'); 2024-11-19T14:02:29.150Z  WARN 1 --- [   scheduling-1] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@3fc0c66f (This connection has been closed.). Possibly consider using a shorter maxLifetime value. Problem: HikariPool-1 - Connection is not available, request timed out after 10009ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:19.141219498+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:29.155141629+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:31.156196969+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:33.156514975+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:35.156847851+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:37.157299363+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:39.157611065+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:41.157907340+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:43.158215925+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:45.158519959+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:47.158822915+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:49.159112575+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:51.159612646+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:53.159916412+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:55.160769376+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:57.161084043+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:02:59.161413739+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:03:01.161729031+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:03:03.162039029+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:03:05.162364989+01:00 Problem: HikariPool-1 - Connection is not available, request timed out after 2000ms (total=0, active=0, idle=0, waiting=0), Failure: 2024-11-19T15:03:07.162657327+01:00 Connection is Valid: **2024-11-19T15:03:09.162953329+01:00**Sql: INSERT INTO time (time) VALUES (to_timestamp('2024-11-19T15:03:09.162953329+01:00', 'YYYY-MM-DDThh24:mi:ss')::timestamp with time zone at time zone 'Etc/UTC'); Outage duration: 51 Connection is Valid: 2024-11-19T15:03:09.331016425+01:00Sql: INSERT INTO time (time) VALUES (to_timestamp('2024-11-19T15:03:09.331016425+01:00', 'YYYY-MM-DDThh24:mi:ss')::timestamp with time zone at time zone 'Etc/UTC');
Below we see the logs of the cluster replica which will be elected as primary after the failover.
We see that the wal receivers started to terminate 5seconds after the failure but it took around 37 additional seconds too see some further logs regarding the failover process itself:
{"level":"info","ts":"2024-11-19T14:02:23.147275668Z","logger":"postgres","msg":"record","logging_pod":"cluster-sample-1","record":{"log_time":"2024-11-19 14:02:23.147 UTC","process_id":"130","session_id":"673c9568.82","session_line_num":"2","session_start_time":"2024-11-19 13:40:56 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"08006","message":**"terminating walreceiver** due to timeout","backend_type":"walreceiver","query_id":"0"}} {"level":"debug","ts":"2024-11-19T14:02:23.172584863Z","msg":"Cached object request received","caller":"pkg/mod/github.com/cloudnative-pg/machinery@v0.0.0-20241014090714-c27747f9974b/pkg/log/log.go:178","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:02:23.303569229Z","msg":"Cached object request received","caller":"pkg/mod/github.com/cloudnative-pg/machinery@v0.0.0-20241014090714-c27747f9974b/pkg/log/log.go:178","logging_pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:02:23.408597375Z","logger":"postgres","msg":"record","logging_pod":"cluster-sample-1","record":{"log_time":"2024-11-19 14:02:23.408 UTC","process_id":"43","session_id":"673c9567.2b","session_line_num":"8","session_start_time":"2024-11-19 13:40:55 UTC","virtual_transaction_id":"148/0","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"invalid record length at 0/E0083D8: expected at least 24, got 0","backend_type":"startup","query_id":"0"}} {"level":"info","ts":"2024-11-19T14:02:58.375493965Z","logger":"Replicator","msg":"synchronizing replication slots","logger":"instance-manager","logging_pod":"cluster-sample-1","err":"getting replication slot status from primary: failed to connect to user=streaming_replica database=postgres: <IP_ADDRESS>:5432 (cluster-sample-rw): dial error: dial tcp <IP_ADDRESS>:5432: connect: connection refused"} {"level":"info","ts":"2024-11-19T14:02:58.376720004Z","logger":"Replicator","msg":"synchronizing replication slots","logger":"instance-manager","logging_pod":"cluster-sample-1","err":"getting replication slot status from primary: failed to connect to user=streaming_replica database=postgres: <IP_ADDRESS>:5432 (cluster-sample-rw): dial error: dial tcp <IP_ADDRESS>:5432: connect: connection refused"} {"level":"debug","ts":"**2024-11-19T14:03:00.430552602Z**","logger":"external_servers_reconciler","msg":"starting up the external servers reconciler","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-external-server","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"575b16c0-d8c4-48a1-b3af-c29fff36cf1c","caller":"internal/management/controller/externalservers/reconciler.go:56","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.430567114Z","msg":"Reconciling Cluster","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"5b2e572b-ae50-4134-9c2f-0844f9f2ceda","caller":"internal/management/controller/instance_controller.go:115","logging_pod":"cluster-sample-1","cluster":{"apiVersion":"postgresql.cnpg.io/v1","kind":"Cluster","namespace":"openshift-operators","name":"cluster-sample"}} {"level":"debug","ts":"2024-11-19T14:03:00.430617394Z","msg":"Reconciling custom monitoring queries","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"5b2e572b-ae50-4134-9c2f-0844f9f2ceda","caller":"internal/management/controller/instance_controller.go:807","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.430674265Z","logger":"tbs_reconciler","msg":"skipping the tablespace reconciler in replicas","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-tablespaces","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"bda1796f-7c85-407d-bf7d-3b9f7a6cf9b5","caller":"internal/management/controller/tablespaces/reconciler.go:52","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.440898957Z","logger":"tbs_reconciler","msg":"skipping the tablespace reconciler in replicas","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-tablespaces","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4f42d5ab-b60c-4025-8837-f959b277137d","caller":"internal/management/controller/tablespaces/reconciler.go:52","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.440916345Z","logger":"external_servers_reconciler","msg":"starting up the external servers reconciler","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-external-server","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"94d16774-2ee8-4d11-97a6-1b6c42da40a5","caller":"internal/management/controller/externalservers/reconciler.go:56","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.455005845Z","logger":"external_servers_reconciler","msg":"starting up the external servers reconciler","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-external-server","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"9f1e441b-8e61-4364-94a4-2c0b5d24bc5c","caller":"internal/management/controller/externalservers/reconciler.go:56","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.455086917Z","logger":"tbs_reconciler","msg":"skipping the tablespace reconciler in replicas","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-tablespaces","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"0000e57f-6943-453b-8239-bb9dd88ba92b","caller":"internal/management/controller/tablespaces/reconciler.go:52","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.464739371Z","logger":"tbs_reconciler","msg":"skipping the tablespace reconciler in replicas","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-tablespaces","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"e1bdbf8f-4c97-41eb-b7f1-98348b7eaa8b","caller":"internal/management/controller/tablespaces/reconciler.go:52","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.46486145Z","logger":"external_servers_reconciler","msg":"starting up the external servers reconciler","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-external-server","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"e830b45c-fcdf-4214-890a-da3429522b03","caller":"internal/management/controller/externalservers/reconciler.go:56","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.476379816Z","logger":"tbs_reconciler","msg":"skipping the tablespace reconciler in replicas","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-tablespaces","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4ca5d71d-f9b4-4a3a-aa65-9e75d819ba6f","caller":"internal/management/controller/tablespaces/reconciler.go:52","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.476433933Z","logger":"external_servers_reconciler","msg":"starting up the external servers reconciler","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-external-server","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"3e0bf7f3-9a00-4043-8b79-590f86b500d2","caller":"internal/management/controller/externalservers/reconciler.go:56","logging_pod":"cluster-sample-1"} {"level":"debug","ts":"2024-11-19T14:03:00.495962846Z","msg":"Reconciling Cluster","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","caller":"internal/management/controller/instance_controller.go:115","logging_pod":"cluster-sample-1","cluster":{"apiVersion":"postgresql.cnpg.io/v1","kind":"Cluster","namespace":"openshift-operators","name":"cluster-sample"}} {"level":"debug","ts":"2024-11-19T14:03:00.496001228Z","msg":"Reconciling custom monitoring queries","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","caller":"internal/management/controller/instance_controller.go:807","logging_pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.497249192Z","logger":"Replicator","msg":"synchronizing replication slots","logger":"instance-manager","logging_pod":"cluster-sample-1","err":"getting replication slot status from primary: failed to connect to user=streaming_replica database=postgres: <IP_ADDRESS>:5432 (cluster-sample-rw): dial error: dial tcp <IP_ADDRESS>:5432: connect: connection refused"} {"level":"info","ts":"2024-11-19T14:03:00.551899561Z","msg":"Setting myself as primary","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","logging_pod":"cluster-sample-1","phase":"Failing over","currentTimestamp":"2024-11-19T14:03:00.551882Z","targetPrimaryTimestamp":"2024-11-19T14:03:00.463507Z","currentPrimaryTimestamp":"2024-11-19T13:35:09.903398Z","msPassedSinceTargetPrimaryTimestamp":88,"msPassedSinceCurrentPrimaryTimestamp":1670648,"msDifferenceBetweenCurrentAndTargetPrimary":-1670560} {"level":"info","ts":"2024-11-19T14:03:00.551924762Z","msg":"I'm the target primary, wait for the wal_receiver to be terminated","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","logging_pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.554814361Z","msg":"I'm the target primary, applying WALs and promoting my instance","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","logging_pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.554833716Z","msg":"Extracting pg_controldata information","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","logging_pod":"cluster-sample-1","reason":"promote"} {"level":"info","ts":"2024-11-19T14:03:00.555859512Z","logger":"pg_controldata","msg":"pg_control version number:            1700\nCatalog version number:               202406281\nDatabase system identifier:<PHONE_NUMBER>660221468\nDatabase cluster state:               in archive recovery\npg_control last modified:             Tue 19 Nov 2024 02:00:56 PM UTC\nLatest checkpoint location:           0/E002458\nLatest checkpoint's REDO location:    0/D0150D8\nLatest checkpoint's REDO WAL file:<PHONE_NUMBER>0000000000000D\nLatest checkpoint's TimeLineID:       2\nLatest checkpoint's PrevTimeLineID:   2\nLatest checkpoint's full_page_writes: on\nLatest checkpoint's NextXID:          0:2380\nLatest checkpoint's NextOID:          24578\nLatest checkpoint's NextMultiXactId:  1\nLatest checkpoint's NextMultiOffset:  0\nLatest checkpoint's oldestXID:        730\nLatest checkpoint's oldestXID's DB:   1\nLatest checkpoint's oldestActiveXID:  2380\nLatest checkpoint's oldestMultiXid:   1\nLatest checkpoint's oldestMulti's DB: 1\nLatest checkpoint's oldestCommitTsXid:0\nLatest checkpoint's newestCommitTsXid:0\nTime of latest checkpoint:            Tue 19 Nov 2024 02:00:09 PM UTC\nFake LSN counter for unlogged rels:   0/3E8\nMinimum recovery ending location:     0/E004618\nMin recovery ending loc's timeline:   2\nBackup start location:                0/0\nBackup end location:                  0/0\nEnd-of-backup record required:        no\nwal_level setting:                    logical\nwal_log_hints setting:                on\nmax_connections setting:              100\nmax_worker_processes setting:         32\nmax_wal_senders setting:              10\nmax_prepared_xacts setting:           0\nmax_locks_per_xact setting:           64\ntrack_commit_timestamp setting:       off\nMaximum data alignment:               8\nDatabase block size:                  8192\nBlocks per segment of large relation: 131072\nWAL block size:                       8192\nBytes per WAL segment:                16777216\nMaximum length of identifiers:        64\nMaximum columns in an index:          32\nMaximum size of a TOAST chunk:        1996\nSize of a large-object chunk:         2048\nDate/time type storage:               64-bit integers\nFloat8 argument passing:              by value\nData page checksum version:           0\nMock authentication nonce:            394ee91a692b830e68552a0f1ced84562d6115a83a3d2179e7ca9d5f34dffb03\n","pipe":"stdout","logging_pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.555881163Z","msg":"Promoting instance","logger":"instance-manager","logging_pod":"cluster-sample-1","controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"4a2d265e-5a49-458b-bae6-f1dd8cc957ff","logging_pod":"cluster-sample-1","pgctl_options":["-D","/var/lib/postgresql/data/pgdata","-w","promote","-t 40000000"]}
Below we see the logs from the controller manager, which pod doesn't log anything at all during the first 42 seconds, after the node failure. It starts logging something about the failover process at the same time as the logs in the replica that we have seen.
{"level":"info","ts":"2024-11-19T13:40:57.085473677Z","msg":"Cluster is healthy","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"cc46950c-dbd8-4ff7-9473-87c616ad1206"} {"level":"info","ts":"**2024-11-19T14:03:00.419148934Z**","msg":"Cannot extract Pod status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-2","error":"Get \"https://<IP_ADDRESS>:8000/pg/status\": dial tcp <IP_ADDRESS>:8000: i/o timeout"} {"level":"info","ts":"2024-11-19T14:03:00.429902684Z","msg":"Current primary isn't healthy, initiating a failover","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d"} {"level":"info","ts":"2024-11-19T14:03:00.42992148Z","msg":"pod status (1 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-1","currentLsn":"","receivedLsn":"0/E0083D8","replayLsn":"0/E0083D8","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":null} {"level":"info","ts":"2024-11-19T14:03:00.429937552Z","msg":"pod status (2 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-3","currentLsn":"","receivedLsn":"0/E0083D8","replayLsn":"0/E0083D8","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":null} {"level":"info","ts":"2024-11-19T14:03:00.429944256Z","msg":"pod status (3 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-2","currentLsn":"","receivedLsn":"","replayLsn":"","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":"Get \"https://<IP_ADDRESS>:8000/pg/status\": dial tcp <IP_ADDRESS>:8000: i/o timeout"} {"level":"info","ts":"2024-11-19T14:03:00.430038243Z","msg":"Cluster is not healthy","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d"} {"level":"info","ts":"2024-11-19T14:03:00.45246805Z","msg":"Failing over","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","newPrimary":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.452497233Z","msg":"pod status (1 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-1","currentLsn":"","receivedLsn":"0/E0083D8","replayLsn":"0/E0083D8","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":null} {"level":"info","ts":"2024-11-19T14:03:00.452513933Z","msg":"pod status (2 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-3","currentLsn":"","receivedLsn":"0/E0083D8","replayLsn":"0/E0083D8","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":null} {"level":"info","ts":"2024-11-19T14:03:00.452520661Z","msg":"pod status (3 of 3)","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","name":"cluster-sample-2","currentLsn":"","receivedLsn":"","replayLsn":"","isPrimary":false,"isPodReady":true,"pendingRestart":false,"pendingRestartForDecrease":false,"statusCollectionError":"Get \"https://<IP_ADDRESS>:8000/pg/status\": dial tcp <IP_ADDRESS>:8000: i/o timeout"} {"level":"info","ts":"2024-11-19T14:03:00.475254228Z","msg":"Waiting for the new primary to notice the promotion request","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a773349e-854e-4e7a-959d-c2c667e4ca1d","newPrimary":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.537044226Z","msg":"There is a switchover or a failover in progress, waiting for the operation to complete","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"470d2f40-97a4-4262-90ea-d75074d539d2","currentPrimary":"cluster-sample-2","targetPrimary":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:00.592989434Z","msg":"There is a switchover or a failover in progress, waiting for the operation to complete","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"78b742d6-3da3-4ed1-9fb2-1d270e23bdb2","currentPrimary":"cluster-sample-2","targetPrimary":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:03.575714289Z","msg":"Cannot extract Pod status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"a92cad56-4eec-4a17-9f55-9b5f8ebf59cc","name":"cluster-sample-2","error":"Get \"https://<IP_ADDRESS>:8000/pg/status\": dial tcp <IP_ADDRESS>:8000: i/o timeout"} {"level":"info","ts":"2024-11-19T14:03:05.708673906Z","msg":"Cannot extract Pod status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","name":"cluster-sample-2","error":"Get \"https://<IP_ADDRESS>:8000/pg/status\": dial tcp <IP_ADDRESS>:8000: i/o timeout"} {"level":"info","ts":"2024-11-19T14:03:05.718818785Z","msg":"Updating pvc metadata","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pvc":"cluster-sample-1","reconciler":"instance-role"} {"level":"info","ts":"2024-11-19T14:03:05.727137941Z","msg":"Updating pvc metadata","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pvc":"cluster-sample-1-wal","reconciler":"instance-role"} {"level":"info","ts":"2024-11-19T14:03:05.735681781Z","msg":"Updating pvc metadata","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pvc":"cluster-sample-2","reconciler":"instance-role"} {"level":"info","ts":"2024-11-19T14:03:05.747854475Z","msg":"Updating pvc metadata","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pvc":"cluster-sample-2-wal","reconciler":"instance-role"} {"level":"info","ts":"2024-11-19T14:03:05.753609429Z","msg":"Setting primary label","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pod":"cluster-sample-1"} {"level":"info","ts":"2024-11-19T14:03:05.76791717Z","msg":"Setting replica label","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-sample","namespace":"openshift-operators"},"namespace":"openshift-operators","name":"cluster-sample","reconcileID":"065e7936-51e9-4979-a4a9-4eff0cc315bc","pod":"cluster-sample-2"}

Thank you for opening this ticket and for your time spent investigating. We will look into this and check all timeouts.
FWIW, we highly discourage running the controller on the same nodes where Postgres runs. Postgres should be isolated from the rest, and proper anti-affinity rules should be in place: https://cloudnative-pg.io/documentation/current/architecture/#reserving-nodes-for-postgresql-workloads

@gbartolini Thank you for your reply. If you need anything else from me just let me know.
Actually putting the controller in the same node as a postgres workloads was only an early experiment, all the later failovertests were executed, when the controller pod was deployed in another node.
Bur are you saying that not only the controller pod, but also all workloads other than the postgres workloads should not be deployed to the three Kubernetes nodes where the postgres workloads live?
That means that it is also recommended to reseize these Kubernetes worker nodes differently from the others, and also have only one postgres cluster for multiple (or all) apps inside a Kubernetes cluster?

@gbartolini Thank you for your reply. If you need anything else from me just let me know. I believe you saw also the other ticket with a feature request #6168 , because from my lasts tests I believe that the reason for the failover taking so long is because the failover on case of node failure is dependent on the node-monitor-grace-period parameter which is by default in Kubernetes 40s and cannot be changed in a supported way.

Well ... we are talking about a Kubernetes node failure. And our priority is to ensure that there is a consistent view in Kubernetes of not just the cluster, but everything around it, starting from services (through which apps connect). You don't want Kubernetes and the failover manager to have different views on the cluster (this is why we don't have a failover manager in CNPG).
Can you provide more information on the environment you have used for these tests and the actual steps to reproduce the issue exactly?

Actually putting the controller in the same node as a postgres workloads was only an early experiment, all the later failovertests were executed, when the controller pod was deployed in another node.

Ok. Thanks.

Bur are you saying that not only the controller pod, but also all workloads other than the postgres workloads should not be deployed to the three Kubernetes nodes where the postgres workloads live? That means that it is also recommended to reseize these Kubernetes worker nodes differently from the others, and also have only one postgres cluster for multiple (or all) apps inside a Kubernetes cluster?

Yes. That is in line with our principle of "cattle vs pets vs elephant herds". PostgreSQL must be treated in a kind of special way, if you want to have best results. Dedicating nodes for Postgres workloads shouldn't be a problem today, thanks to logical (affinity) and physical (taints and labels) separation of workloads that makes them more similar to cattle than pets. This opens up for bare metal Postgres nodes with directly attached storage.
I covered this in my last blog article: https://www.cncf.io/blog/2024/11/20/cloud-neutral-postgres-databases-with-kubernetes-and-cloudnativepg/
Thanks,
Gabriele

@gbartolini I am running openshift 4.15 on prem on top of vSphere cluster (vmware), I have multiple worker nodes and 3 master nodes all of them are in the end effect Virtual Machines on vSphere.

I installed the latest cloudnativepg operator from the openshift operator market, and deployed a postgres cluster without changing much from the default configuration., except that I set the tolerationSeconds parameter to 0, since I don't want to wait another additional 300sec before the failover starts, that's documented here: https://cloudnative-pg.io/documentation/1.20/failure_modes/#:~:text=will happen after-,tolerationSeconds,-.

In the end effect I have a postgres cluster with three replicas each of them deployed in different worker nodes (on those worker nodes there are still other workloads which are not related to postgres), and the controller manager pod of cloudnativepg is running on a fourth node.

After everything is up and running I start my simple client pod which enters data (e.g. timestamp info) to database each second.

After sometime I look in which worker node the primary replica of postgres cluster is living and I go on my vSphere cluster and shut-down exactly that virtualmachine which represents the worker node which hosts the primary replica.

It takes something between 40-60seconds until my client is again able to write into it.

As explained in this ticket and other linked tickets, I was assuming that this time is coming from the node-monitor-grace-period parameter which is by default in Openshift 40s and cannot be changed in a supported way. That's why I made a test and changed this parameter with an unsupportedConfigOverride

$ oc edit kubecontrollermanagers cluster -n openshift-config spec: unsupportedConfigOverrides: extendedArguments: node-monitor-grace-period: - 20s
After this change the failover time was ca. 20s smaller.
Kubernetes may have its own reasons why it waits by default 40s+5mins until the pods are notified that their node is gone. But if we want to have an application which is high available then those default values can never be true. We are using for example NATS as message broker they are also very cloud native from very beginning and they have a very fast failover time, based on their own timeouts. They have also a kind of controller and as soon as the controller notices that a replica is not healthy it will initiate some changes. They are not building the failover mechanism based on what some Kubernetes standards try to introduce as a silver bullet. The custom timeout would be very similar to readinessProbe or livnessProbe which are faster than the node failure mechanism for now.
You said the cloudnativepg is special and should be treated special, why should it be different this time, why should we use only the default solution that Kubernetes offers us in this particular case?


New release?

I'm trying to add ZArchive to vcpkg (microsoft/vcpkg#26694), and a new release would be very helpful, as the current v0.1.1 fails to build.

I bumped the version and published a new release.


[Work in progress] Concurrent kafka instances

THIS IS A WORK IN PROGRESS

Functionally the changes are made. I would like to make some refactorings first and also look into the query raised by @shashidesai.
I'm raising this PR now in the hopes that you can take a look at my approach. If you have any major concerns it would be best to find out sooner!
Closes #101
TODO:

[ ] @shashidesai's query around runStreams
[ ] type class for test to go from Seq[EmbeddedServer] => Seq[EmbeddedZ] or Seq[EmbeddedK], replacing the boiler place isInstanceOf + asInstanceOf
[ ] clean up of new tests for conciseness
[x] javadoc for EmbeddedServer.scala

Problem
Running tests in parallel which use EmbeddedKafka can run into an issue with the log files as EmbeddedKafka#stop clears all temporary log directories, which another instance of kafka may still be relying on in another test.
To see this error I changed logback.xml line 1 to: log4j.rootLogger=error,stdout:
FATAL [Replica Manager on Broker 0]: Error writing to highwatermark file ... java.io.FileNotFoundException ... kafka-logsXXXXXXXXX.tmp/replication-offset-checkpoint.tmp (No such file or directory)
Testing
I would like to improve these as there is quite a lot of repetition

start and stop a specific Kafka
start and stop multiple Kafka instances on specified ports

Solution
I have created a model of the running servers, both Kafka and Zookeeper. Starting one of these appends an EmbeddedServer to managed list of servers.
The following functions have had their signatures modified. Given that they all used to return Unit, this results in a non-breaking change to users. On upgrading the returned objects will be ignored and the existing functionality will be preserved.
EmbeddedKafka#start now returns an EmbeddedServer.
EmbeddedKafka#startZooKeeper now returns an EmbeddedZ.
EmbeddedKafka#startKafka now returns an EmbeddedK.
I have added an overload to EmbeddedKafka#stop which will stop the specific server and remove it from the list of managed servers.

Thanks @tjheslin1 - the approach looks good to me. Let me know when you've finished and I'll carry on with the review.

Ready to be reviewed.

All looks good - could you please address the 2 minor things that are still pending?

Ready now @manub. @shashidesai will be raising another issue.

Thanks. Will merge now and release in the next days.


Notification CC: automatic idle reset after 5 minutes not happening

https://github.com/home-assistant/core/issues/46982

@AlCalzone. I'm not 100% sure this addresses all the instances of auto idle.
See below 113-0-Home Security-unknown does not auto-idle, where 113-0-Home Security-Sensor status does.
2021-03-30 22:10:13.953 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.233 => 706.234
2021-03-30 22:10:14.083 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.076 => 5182.089
2021-03-30 22:10:14.164 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.501 => 2882.503
2021-03-30 22:10:23.877 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:24.161 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:26.912 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:28.012 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:29.401 INFO ZWAVE: Node 24: value updated: 48-0-Any true => false
2021-03-30 22:10:29.420 INFO ZWAVE: Node 24: value added: 113-0-Home Security-Sensor status => 2
2021-03-30 22:10:29.421 INFO ZWAVE: Node 24: value added 24-113-0-Home Security-Sensor status => 2
2021-03-30 22:10:33.774 INFO ZWAVE: Node 24: value updated: 48-0-Any false => true
2021-03-30 22:10:33.798 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 2
2021-03-30 22:10:37.869 INFO ZWAVE: Node 24: value updated: 48-0-Any true => false
2021-03-30 22:10:37.892 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 2
2021-03-30 22:10:39.257 INFO ZWAVE: Node 24: value updated: 48-0-Any false => true
2021-03-30 22:10:39.284 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 2
2021-03-30 22:10:43.094 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:44.205 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-unknown 254 => 254
2021-03-30 22:10:46.404 INFO ZWAVE: Node 24: value updated: 48-0-Any true => false
2021-03-30 22:10:46.411 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 2
2021-03-30 22:10:47.784 INFO ZWAVE: Node 24: value updated: 48-0-Any false => true
2021-03-30 22:10:47.806 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 2
2021-03-30 22:11:06.584 INFO ZWAVE: Node 4: value updated: 128-0-level 100 => 100
2021-03-30 22:11:06.586 INFO ZWAVE: Node 4: value updated: 128-0-isLow false => false
2021-03-30 22:11:06.792 INFO ZWAVE: Node 4: metadata updated: 49-0-Illuminance
2021-03-30 22:11:06.794 INFO ZWAVE: Node 4: value updated: 49-0-Illuminance 0 => 0
2021-03-30 22:11:08.047 INFO ZWAVE: Node 4: metadata updated: 49-0-Air temperature
2021-03-30 22:11:08.049 INFO ZWAVE: Node 4: value updated: 49-0-Air temperature 19.5 => 19.5
2021-03-30 22:11:08.289 INFO ZWAVE: Node 4: metadata updated: 49-0-Humidity
2021-03-30 22:11:08.291 INFO ZWAVE: Node 4: value updated: 49-0-Humidity 29 => 30
2021-03-30 22:11:13.497 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1060.955 => 1064.084
2021-03-30 22:11:13.573 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.826 => 8770.842
2021-03-30 22:11:13.679 INFO ZWAVE: Node 19: value updated: 50-1-value-66049 78.013 => 78.579
2021-03-30 22:11:13.775 INFO ZWAVE: Node 19: value updated: 50-2-value-66049 814.862 => 817.226
2021-03-30 22:11:13.862 INFO ZWAVE: Node 19: value updated: 50-3-value-66049 168.08 => 168.279
2021-03-30 22:11:13.971 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.234 => 706.235
2021-03-30 22:11:14.106 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.089 => 5182.102
2021-03-30 22:11:14.158 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.503 => 2882.505
2021-03-30 22:12:13.511 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1064.084 => 1074.906
2021-03-30 22:12:13.574 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.842 => 8770.858
2021-03-30 22:12:13.672 INFO ZWAVE: Node 19: value updated: 50-1-value-66049 78.579 => 78.146
2021-03-30 22:12:13.780 INFO ZWAVE: Node 19: value updated: 50-2-value-66049 817.226 => 828.88
2021-03-30 22:12:13.881 INFO ZWAVE: Node 19: value updated: 50-3-value-66049 168.279 => 167.88
2021-03-30 22:12:13.966 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.235 => 706.236
2021-03-30 22:12:14.076 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.102 => 5182.115
2021-03-30 22:12:14.192 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.505 => 2882.507
2021-03-30 22:12:21.079 INFO ZWAVE: Node 13: value updated: 48-0-Any true => false
2021-03-30 22:13:13.510 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1074.906 => 1085.327
2021-03-30 22:13:13.581 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.858 => 8770.874
2021-03-30 22:13:13.681 INFO ZWAVE: Node 19: value updated: 50-1-value-66049 78.146 => 78.246
2021-03-30 22:13:13.784 INFO ZWAVE: Node 19: value updated: 50-2-value-66049 828.88 => 839.135
2021-03-30 22:13:13.888 INFO ZWAVE: Node 19: value updated: 50-3-value-66049 167.88 => 167.946
2021-03-30 22:13:14.009 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.236 => 706.237
2021-03-30 22:13:14.315 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.115 => 5182.128
2021-03-30 22:13:14.381 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.507 => 2882.509
2021-03-30 22:13:16.322 INFO ZWAVE: Node 13: value updated: 48-0-Any false => true
2021-03-30 22:14:13.510 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1085.327 => 1205.228
2021-03-30 22:14:13.635 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.874 => 8770.891
2021-03-30 22:14:13.704 INFO ZWAVE: Node 19: value updated: 50-1-value-66049 78.246 => 192.286
2021-03-30 22:14:13.897 INFO ZWAVE: Node 19: value updated: 50-2-value-66049 839.135 => 845.162
2021-03-30 22:14:14.128 INFO ZWAVE: Node 19: value updated: 50-3-value-66049 167.946 => 167.78
2021-03-30 22:14:14.188 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.237 => 706.239
2021-03-30 22:14:14.336 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.128 => 5182.141
2021-03-30 22:14:14.391 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.509 => 2882.511
2021-03-30 22:15:06.588 INFO ZWAVE: Node 4: value updated: 128-0-level 100 => 100
2021-03-30 22:15:06.589 INFO ZWAVE: Node 4: value updated: 128-0-isLow false => false
2021-03-30 22:15:06.788 INFO ZWAVE: Node 4: metadata updated: 49-0-Illuminance
2021-03-30 22:15:06.792 INFO ZWAVE: Node 4: value updated: 49-0-Illuminance 0 => 0
2021-03-30 22:15:08.044 INFO ZWAVE: Node 4: metadata updated: 49-0-Air temperature
2021-03-30 22:15:08.047 INFO ZWAVE: Node 4: value updated: 49-0-Air temperature 19.5 => 19.5
2021-03-30 22:15:08.289 INFO ZWAVE: Node 4: metadata updated: 49-0-Humidity
2021-03-30 22:15:08.290 INFO ZWAVE: Node 4: value updated: 49-0-Humidity 30 => 30
2021-03-30 22:15:13.521 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1205.228 => 1174.961
2021-03-30 22:15:13.599 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.891 => 8770.909
2021-03-30 22:15:13.688 INFO ZWAVE: Node 19: value updated: 50-1-value-66049 192.286 => 183.662
2021-03-30 22:15:13.781 INFO ZWAVE: Node 19: value updated: 50-2-value-66049 845.162 => 823.353
2021-03-30 22:15:13.925 INFO ZWAVE: Node 19: value updated: 50-3-value-66049 167.78 => 167.946
2021-03-30 22:15:13.991 INFO ZWAVE: Node 19: value updated: 50-1-value-65537 706.239 => 706.242
2021-03-30 22:15:14.084 INFO ZWAVE: Node 19: value updated: 50-2-value-65537 5182.141 => 5182.154
2021-03-30 22:15:14.192 INFO ZWAVE: Node 19: value updated: 50-3-value-65537 2882.511 => 2882.513
2021-03-30 22:15:47.811 INFO ZWAVE: Node 24: value updated: 113-0-Home Security-Sensor status 2 => 0
2021-03-30 22:16:13.536 INFO ZWAVE: Node 19: value updated: 50-0-value-66049 1174.961 => 1178.591
2021-03-30 22:16:13.612 INFO ZWAVE: Node 19: value updated: 50-0-value-65537 8770.909 => 8770.927

113-0-Home Security-unknown does not auto-idle

Yes, because that is not a known notification. Probably shouldn't even exist. To understand why it keeps popping up, I'll need to see a zwave-js log with loglevel debug.

In OZW this was supported. When you say not known notification, is that a device definition parameter?
With the most recent version 7.1.1 neither value is idling after 5 minutes.
I apologize if these logs do not have the necessary info. I'm having some issues with zwavejs2mqqt exporting logs.
If that is the case, I'll post updated logs once that is addressed.
zwavejs2mqtt-store (2).zip
zwavejs2mqtt-store (3).zip

When you say not known notification, is that a device definition parameter?

No, the Z-Wave Alliance defines which combination of notification type and notification event means what: https://github.com/zwave-js/specs/blob/master/SDS13713 Notification Command Class%2C list of assigned Notifications.xlsx
Notification event 254 is always labeled as an unknown notification. That means it isn't clear whether this is an event or a variable, and whether it may be idled if it is actually a variable. So zwave-js just tells you that it was received but does not make an assumption about idling.

In OZW this was supported

That's a bad argument :) OZW supported many things but also did many things wrong.

With the most recent version 7.1.1 neither value is idling after 5 minutes.

Yes, see https://github.com/zwave-js/node-zwave-js/blob/master/CHANGELOG.md#710-2021-04-05
This behavior is now opt-in on a per-device basis, because it broke motion sensors that don't continuously refresh their status.
I took a look at your logs. Neither of them seem to be from the newest version, so I can't confirm. Anyways, we can add the flag to the device files if you tell me which devices are affected. Node IDs plus the <homeid>.values.jsonl file from the store directory should suffice.


The Big 2.0 PR

Reopening because I decided not to move yarn-bound functionality into this yet, though I do think it probably belongs in the same repo (also I was trying to merge from master which was annoying and that can't be changed on github). I can move the comments over if desired
@hylyh @blurymind @alforno
I have here a fork of bondage.js that attempts full Yarn 2.0 compliance (minus the wait command)
https://github.com/YarnSpinnerTool/YarnSpinner/blob/9275277f50a6acbe8438b29596acc8527cf5581a/Documentation/Yarn-Spec.md
The changes are sweeping enough that I see this as basically an overwrite (hence the strategy=ours merge strategy).
Problems that necessitated large structural changes included:

lines after a jump or stop were not ignored properly
jumping added a layer of recursion that never cleared. So, large dialogues caused stack overflow

Could be a problem in a kiosk scenario

File size was 300+kb. It is now <50.

Additional features like markup (and abstracting away ugly generators) are currently in a separate project, YarnBound. This is an additional can easily be incorporated into the repo if desired, but the vision is that it is a separate layer that consumes this core language API.
If this is accepted I would also request maintainer status.

Closing due to lack of interest

I have created a 2.0 branch - to become master once tested
https://github.com/hylyh/bondage.js/tree/2.0
Can we merge this pr into it and continue there please? There is interest, there just arent maintainers. We're all busy with life it looks like

Target has changed

merged now :) thank you for the hard work :+1: :tada:

@mnbroatch I had a look at YarnBound.
Would it be hard for you to make a pr that incorporates it into this repository? I would very much like these new features to be available straight out of the box for yarn 2 users.
My idea now is to make yarn 2 the official branch - once yarn bound becomes a part of yarn 2

Hey, haven't had much time for this but I did open a draft PR. It needs READMEs updated, and some things renamed. Not sure how you want to handle branding and whatnot. But the tests are passing and it's building. I'm just using the yarn-bound build system so if you want continuity in that regard it'll be up to you


Benchmark max concurrent players

It would be nice to have a snapshot of performance on the website or in the book so that everyone knows what can be achieved with Ambient.
I only have that very rough smoke test I did months ago and it didn't even test a real game - it was just to double check if we can handle ~100 concurrent connections. We could start with:

a baseline -  an empty project (one created with ambient new).
something that at least has input messages (not only ECS diffs) and maybe some processing on the server (a physics simulation running) - based on third party controller or some other example so that people know what to expect in a closer to real game scenario.

For all cases we want to answer the question how many concurrent connection a single Ambient server can sustain while still running the simulation reliably at 60 FPS.

I know @droqen has been working on these benchmarks, specifically for the up and coming wasm runtime improvements

After a conversation w @pierd I have an idea for how to tackle this
Basic requirement for a benchmark that would be useful for benchmarking max concurrent players

Simulate clientside inputs, so that any # can be run
My idea
Boids where each boid is an individual automated client

A related thing would be to state Ambient's optimal cloud VM type: General purpose vs Memory optimised vs Compute optimised. Including some recommended specs.

That's highly dependent on the game but I agree that having ballpark numbers from the benchmark would be useful.


Improve website styling ❤✨🔥😊🐦✅🎃⭐😂

This will greatly improve the design quality of the website.

Please check this out Mr. @StevenKGER

thank you my friend


Introduce a cluster UUID persisted to a ConfigMap

Antrea Controller persists a randomly-generated cluster UUID tp the
antrea-cluster-uuid ConfigMap during initialization. The cluster UUID
stays the same across all Antrea Controller restarts and Antrea
upgrades. The UUID can be consummed by other Antrea components when
needed. One current example is the Flow Aggregator, which can use the
cluster UUID to generate a deterministic IPFIX Observation Domain ID to
uniquely identify the exporting process to the external flow collector.
An error during the generation of the cluster UUID will not cause
Controller initialization to fail. Instead the Controller will keep
retrying until it can successfully 1) establish that a cluster UUID
already exists and is correct, or 2) generate and persist a new cluster
UUID. If a component strictly depends on the cluster UUID, it should
fail its own initialization process until it becomes available.

I'm planning to add more tests (including an e2e test), but I'd like to get some early feedback first

I'm planning to add more tests (including an e2e test), but I'd like to get some early feedback first

Questions:

We must use UUID, or can also use a unique string to identify a cluster?
I assume we can support customer specified UUID as well?
Have you considered using the antrea ConfigMap?

Right now, I'm using a UUID, but we can change that to arbitrary string (especially if we want to support user-provided ids, as per your second question). Of course, in that case the user is in charge of guaranteeing uniqueness. When I designed this, I mostly had telemetry in mind, in which case user-provided UUID was not the best solution.
This is not something I specifically designed for, but yes the user can provide a UUID by editing the ConfigMap definition in the YAML: the Antrea Controller will use the existing user-provided UUID and will not generate a new one.
IMO, letting Antrea mutate the main "configuration" ConfigMap is not right. If you think about some operator design, the operator should own the ConfigMap, and be the only one making changes. So unless the only case we want to support is "user-provided" UUID, then it's not ideal. And I think we want to support auto-generating an UUID in the Controller to guarantee the existence of the UUID in all cases. What we could do is accept an ID as part of the main antrea ConfigMap. If present the ID will be used directly, as-is, in the antrea-cluster-uuid ConfigMap. If no ID is provided (default), then the Controller generates a UUID and stores it in antrea-cluster-uuid. All consumers should get the ID from antrea-cluster-uuid, but for the user, there is still a single ConfigMap for all configuration (antrea).

If we ever want to use the UUID for telemetry / usage reporting, and if we support user-provided values (arbitrary strings), we should probably hash the UUID in case the user includes some identifiable information in the ID string.

I assume when used by distros, we might populate some distro ID to identify a cluster.

If present the ID will be used directly, as-is, in the antrea-cluster-uuid ConfigMap. If no ID is provided (default), then the Controller generates a UUID and stores it in antrea-cluster-uuid.
The only thing then it introduces redundancy and potential inconsistency when an ID is configured via the antrea ConfigMap. Might not be a big deal though.

If we ever want to use the UUID for telemetry / usage reporting, and if we support user-provided values (arbitrary strings), we should probably hash the UUID in case the user includes some identifiable information in the ID string.

Sounds good to me.

I assume when used by distros, we might populate some distro ID to identify a cluster.

If present the ID will be used directly, as-is, in the antrea-cluster-uuid ConfigMap. If no ID is provided (default), then the Controller generates a UUID and stores it in antrea-cluster-uuid.

The only thing then it introduces redundancy and potential inconsistency when an ID is configured via the antrea ConfigMap. Might not be a big deal though.

If we ever want to use the UUID for telemetry / usage reporting, and if we support user-provided values (arbitrary strings), we should probably hash the UUID in case the user includes some identifiable information in the ID string.

Sounds good to me.

Setting in configMap directly could also mean updating the map with new cluster ID, i guess we won't support such a scenario right? Where the clusters'ID can be updated after deployment

I have two questions on flow exporter and flow aggregator side:

why we would want users to input the custom observation domain id? We can use the Service name as before if UUID is unavailable.
will controller generate a new UUID each time when Antrea restarts? I'm asking this because if users want to config flow exporter (e.g. poll interval), they will need to restart the Antrea.

I have two questions on flow exporter and flow aggregator side:

why we would want users to input the custom observation domain id? We can use the Service name as before if UUID is unavailable.
will controller generate a new UUID each time when Antrea restarts? I'm asking this because if users want to config flow exporter (e.g. poll interval), they will need to restart the Antrea.

Using the Service name may yield the same observation domain ID for all aggregators running in all clusters, which may share the same external flow collector. I think that users should have the opportunity to specify their own observation domain ID if they want to be able to map the flow records received by the flow collector to a specific aggregator, i.e. a specific cluster. If they don't, we use the cluster UUID to make sure we have a persistent observation domain ID. If by any chance the cluster UUID is not available, we just generate a random ID (not persistent). That last case is unlikely to occur and we could use the Service name there. It depends on what is more important 1) unique domain IDs across aggregators connected to the same collector, or 2) persistent domain ID. If you think it's 2), then I can use the Service name instead of a random number for the fallback case.
No, the whole point of using a ConfigMap is that the UUID is persistent and stays across all restarts and upgrades

I have two questions on flow exporter and flow aggregator side:

why we would want users to input the custom observation domain id? We can use the Service name as before if UUID is unavailable.
will controller generate a new UUID each time when Antrea restarts? I'm asking this because if users want to config flow exporter (e.g. poll interval), they will need to restart the Antrea.

Using the Service name may yield the same observation domain ID for all aggregators running in all clusters, which may share the same external flow collector. I think that users should have the opportunity to specify their own observation domain ID if they want to be able to map the flow records received by the flow collector to a specific aggregator, i.e. a specific cluster. If they don't, we use the cluster UUID to make sure we have a persistent observation domain ID. If by any chance the cluster UUID is not available, we just generate a random ID (not persistent). That last case is unlikely to occur and we could use the Service name there. It depends on what is more important 1) unique domain IDs across aggregators connected to the same collector, or 2) persistent domain ID. If you think it's 2), then I can use the Service name instead of a random number for the fallback case.

Thanks for elaboration. Much clearer to me. I prefer choice 1) as it is now.

No, the whole point of using a ConfigMap is that the UUID is persistent and stays across all restarts and upgrades

Got it. Thanks!

I updated my approach based on review comments and pushed a new commit: https://github.com/vmware-tanzu/antrea/pull/1805/commits/c3714253d4c5e3dd2f3038721fcfa005c470e20f

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@9a8939a). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1805   +/-   ##
=======================================
  Coverage        ?   49.14%
=======================================
  Files           ?      195
  Lines           ?    16645
  Branches        ?        0
=======================================
  Hits            ?     8181
  Misses          ?     7384
  Partials        ?     1080

Flag
Coverage Δ

kind-e2e-tests
49.14% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Questions:
I do not see clusterName is used anywhere. What is its usage?

It's not used right now. I added it so we have a user-friendly id for applications that may require it, like cross-cluster features. This PR doesn't really try to introduce use cases for clusterName / clusterUUID; I did modify the FlowAggregator code to use the cluster UUID to generate an observation domain ID, because that's one use case I had in mind and it's a good proof-of-concept.

For clusterID, is that any use case requires a UUID (but not a string ID)?

The idea is to have a globally unique cluster id for scenarios where cluster names may be selected by different entities. At this time, the one use case I have in mind is for downstream projects built for Antrea which may want to enable telemetry / usage reporting. Of course it's always possible for a user to override the UUID with a fixed value, but that's not the intent.
Typically I would expect users to set the clusterName and let the clusterUUID be automatically generated.

For clusterID, is that any use case requires a UUID (but not a string ID)?

The idea is to have a globally unique cluster id for scenarios where cluster names may be selected by different entities. At this time, the one use case I have in mind is for downstream projects built for Antrea which may want to enable telemetry / usage reporting. Of course it's always possible for a user to override the UUID with a fixed value, but that's not the intent.

But does the unique ID must be UUID or it can be any string? Or you like follow K8s resources which have both a name and a UID, but in K8s both names and IDs must be unique?
I am asking because I know some systems we might integrate with do not require UUID but need a unique string ID. But I do not know if any system must require a UUID (but you can still generate UUID from the string ID in that case?).

For clusterID, is that any use case requires a UUID (but not a string ID)?

The idea is to have a globally unique cluster id for scenarios where cluster names may be selected by different entities. At this time, the one use case I have in mind is for downstream projects built for Antrea which may want to enable telemetry / usage reporting. Of course it's always possible for a user to override the UUID with a fixed value, but that's not the intent.

But does the unique ID must be UUID or it can be any string? Or you like follow K8s resources which have both a name and a UID, but in K8s both names and IDs must be unique?
I am asking because I know some systems we might integrate with do not require UUID but need a unique string ID. But I do not know if any system must require a UUID (but you can still generate UUID from the string ID in that case?).

@jianjuns My concern is with the definition of uniqueness. My intent was to have a high probability of global uniqueness (unique among all clusters from all Antrea users). If we start accepting an arbitrary string id, I'm afraid that we may have increased collisions as users may start providing ids which are unique to them. Of course the risk also exists when a user provides a UUID,
but is lesser IMO based on how UUIDs are typically generated. Generating a UUID from the arbitrary string ID does not solve that problem, at least not if we want to generate it deterministically.
But then maybe my goal to have a "globally" unique ID is not practical...
I'd like to understand more about the systems you mention. So far this PR seeks to provide:

a cluster name, which is meant to be unique for a given user; it can be user-provided and human-readable
a standard UUID, which is meant to be globally unique and auto-generated by Antrea

For the systems you mention, I imagine using the UUID generated by Antrea is not an option and they want to use Antrea to store / consume their own ID within the scope of a cluster. And is it the case that the "cluster name" cannot be used to store the id in that case, i.e. there is a need for both cluster name and cluster id (as an arbitrary string)?
Of course, I can always add a 3rd entry in the cluster identity ConfigMap (ConfigMap used to persist the name / id values) to have:

cluster name
cluster id
cluster UUID
And to keep the antrea ConfigMap simple, I can remove the UUID from it and replace it with the arbitrary id. This may be better actually because now it makes it more unlikely for the UUID not to be globally unique (admin would have to overwrite the identify ConfigMap directly). If the user provides a name, but no id, the generated UUID will be used as the id by default. What do you think?

Updated the PR based on recent offline discussions with @jianjuns and @edwardbadboy. The PR is much closer now to what it was originally:

a cluster UUID is persisted to the antrea-cluster-identity ConfigMap by the Controller
users cannot provide their own UUID, except by editing the antrea-cluster-identity ConfigMap before starting the Contoller

/test-all


Harvest Finance Mainnet QA

Review Info

Description
Info

Subgraph Reviewed
https://subgraphs.messari.io/subgraph?endpoint=https://api.thegraph.com/subgraphs/id/QmcBPHEPEB9VtgXFw9hefqh7jafZMr7LfLvn5knorGz8SH

Schema Version
1.3.0

Subgraph Version
0.1.0

Methodology Version
1.0.0

Evidence Spreadsheet
https://docs.google.com/spreadsheets/d/13Yc46gVMeKG_hsnUNgq2BdWyfIq0PwVu/edit?usp=share_link&ouid=104027705798814439445&rtpof=true&sd=true

Metric to Review
TVL

The subgraph's TVL data shows a 20.19% five-day average discrepancy with data from Defi Llama which is significant. In addition, the discrepancy between the recent subgraph's and the official website's data also shows a high discrepancy of 14.53%.
This is probably due to the reason that some vaults are missing in the subgraph data. For instance, the vault named 'Uniswap v3: ORC-ETH' has a TVL of $1.1 million and it is missing in the subgraph data.
See 'Protocol' and 'Support-TVL-and-Revenue' sheets for more information.

Subgraph TVL 5-day average
Defi Llama 5-day average
Absolute Difference
% Difference

11,266,843
14,116,663
-2,849,820
-20.19%

Recent Subgraph TVL
Implied TVL
Absolute Difference
% Difference

11,266,843
13,139,050
-1,908,765.37
-14.53%

Revenue

Daily revenue data must not be zero. Our computations for the implied revenue show that the total revenue for the recent data must be around $2,006.46 (including both Base APY an $FARM reward token APY).
Refer to 'Protocol' and 'Support-TVL-and-Revenue' sheets

Subgraph Daily Supply Side Revenue
Implied Revenue
Absolute Difference
% Difference

0
1,410.10
-1,410.10
-100%

Subgraph Daily Protocol Side Revenue
Implied Revenue
Absolute Difference
% Difference

0
604.33
-604.33
-100%

Subgraph Total Daily Revenue
Implied Revenue
Absolute Difference
% Difference

0
2,006.46
-2,006.46
-100%

Daily Transaction Count

The recent daily transaction count data (as well as the data from the 22nd and 23rd of December) is not in line with the sum of the daily deposit and withdraw transactions.
Refer to the 'Protocol' sheet

Pool Overview

Only 9 out of 50 vaults in the subgraph data are shown on the official website.
Out of these 9 vaults, three vaults' TVL data have more than a 10% discrepancy with the official website TVL data. The vaults' names are iFARM, FARM_LOOKS, and FARM_cvxCRV.
The top 7, 9-17, and 20-50 pools are inactive vaults and are not shown on the official website
11 vaults/pools on the official website are not shown in the subgraph data. See the 'Support-TVL-and-Revenue' sheet for more info. The missing vaults' names are Uniswap v3: ORC-ETH, Uniswap V3: CNG-ETH, Uniswap V3: ETH-sETH2, FARM/ETH, YEL-ETH, MUSE-ETH, Uniswap V3: WBTC-ETH, nDAI, CRV:TriCrypto2, FARMStead USDC-24, and FARM/GRAIN vaults.
There must be $FARM Reward Token data for each active pool
Refer to the 'Pool_Overview' sheet

Please refer to the attached worksheet for more details.

@llopez QA comments on harvest here!

thanks for reviewing the data. We will take a look and back to you.

We found the problem with dailyTransactionCount. We are tracking more than the withdrawals and deposits. We are also tracking the mint and burn. We will make the required changes to fix this issue

Regarding TVL our numbers are closer to the protocol's website than defi llama.

@llopez do you have any notes on the difficulties you faced trying to address this issue?


Update CLI extensions available doc

Update CLI extensions available doc.
Triggered by Azure/azure-cli-extensions - TRAVIS_BUILD_ID=617407603
https://github.com/Azure/azure-cli-extensions/commit/862348442236829acd3efeb54c0e8d3379c9b4a6

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: azuresdkci sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Docs Build status updates of commit f6a53b4:
:clock10: Full build: the average full build time is 7 min(s) 9 sec(s), this is based on the last 287 build(s) for this repository.

Docs Build status updates of commit f6a53b4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/azure-cli-extensions-list.md
:white_check_mark:Succeeded
View (azure-cli-latest) View (azure-cli-2017-03-09-profile) View (azure-cli-2018-03-01-hybrid)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


EAGLE-298 Oozie auditlog integration for Oozie security monitoring

Add oozie auditlog parse code and related test code.
Add job.html and controll.js code for new view type job.
Add oozie auditlog entity ,oozie metatdata access config code and related test code.

https://issues.apache.org/jira/browse/EAGLE-298

LGTM.

@yonzhang have you reviewed the PR detail? If you still have problem about manual PR patch applying, please try the PR tools under eagle-dev? If you are ok with the code change, please help merge the PR.

@haoch What I need to do next?

@r7raul1984 no need, thanks very much for the contribution 👍 and welcome to join eagle community!

The patch has been merged into https://github.com/apache/incubator-eagle/commit/fd39e6e14ed5cdca1e7b4314b2b4e5e1d1d97425, please close the pull request.

Thank you !


doc: Update links to markdown format, add a minor UART connection fix

Summary

Minor fixes to links format and STLINK UART connection.
Closes #36
Closes #19
Screenshots

Not needed.
Changes

Adjust links from Hugo to MD
Fix UART STLINK connection

Notes for Reviewers

Minor fix.
Checklist

[x] Scope: The referenced issue is addressed, there are no unrelated changes.
[x] Run Locally: Verified that the docs build using make server, posted screenshots, verified external links.
[x] New Features Marked: Documentation for new features is marked using the new-in-version shortcode, according to the guidelines in CONTRIBUTING.
[x] Style Guidelines: Documentation obeys style guidelines in CONTRIBUTING.
[x] Commits: Commit messages follow guidelines in CONTRIBUTING, there are no fixup commits left.

@benolayinka would be great if you can check this out.


[stable/suitecrm] fix(ingress): fix chart label

What this PR does / why we need it:
Which issue this PR fixes

fixes ingress chart label

Special notes for your reviewer:
template suitecrm.chart is undefined
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

/assign @juan131

Thanks for the contribution, I would use the template approach as it is a common practice in other charts and also in other labels:

Create a function in _helpers.tpl:

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "suitecrm.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Replace

chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

by
chart: "{{ template "suitecrm.chart" . }}"

in the following places:
templates/deployment.yaml
8:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
21:        chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

templates/externaldb-secrets.yaml
8:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

templates/ingress.yaml
8:    chart: "{{ template "suitecrm.chart" . }}"

templates/suitecrm-pvc.yaml
8:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

templates/secrets.yaml
7:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

templates/svc.yaml
7:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
34:    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

Modified last commit with @carrodher comments

/ok-to-test
/lgtm


Compatibility issue

It is not compatible with reactive form version 9.1.0. Any update regarding that ?

We are currently moving to monorepo
Check individual packages
https://github.com/artflutter/reactive_forms_widgets/issues/6#issuecomment-791905298

@maanhaider any feedback?

I didn't check it yet because of my busy schedule . Will try it soon and then give you feedback :)

Closing this issue for now. Feel free to open new one if there will be any issues


False positive warning in markdown: links inside math

VS Code Version:  1.84.0
OS Version: macOS 13.2

Square brackets inside katex fragments are mistaken as markdown links.

Please always share text along with screenshots so we can easily reproduce the issue

$$
gain^{(s)}
  = \Delta - \Delta ^{(s)}
  = \frac{1}{2}\left[
    gain_{j_L}^{(s)} + gain_{j_R}^{(s)} - gain_j \right]
$$

Duplicate of https://github.com/microsoft/vscode-markdown-languageservice/issues/14


Ex3/Task4 - Need to reconnect to Exchange Online after Windows CL1 reboot in Task3

After Task3 reboot of CL1 Windows VM the student needs to

Open Power Shell as a regular user Joni.
Reconnect to Exchange Online

Before proceeding with Task 4 Step 2.
Please provide more detailed and proper instructions.

@arnoldvilleneuve, Than you for bringing this to my attention. To ensure I address your concern accurately, can you please clarify which lab requires clearer instructions in exercise 3, task 4?

Lab1/Ex4/Task4
Task 4 – Create custom branding template
Protected messages sent by your organizations finance department require special branding, including customized introduction and body texts and a Disclaimer link in the footer. The finance messages shall also expire after seven days. In this task, you will create a new custom OME configuration and create a transport rule to apply the OME configuration to all mail sent from the finance department.
Log into Client 1 VM (LON-CL1) as the lon-cl1\admin account and there should still be an open PowerShell window with Exchange Online connected.
Run the following cmdlet to create a new configuration:
powershell
New-OMEConfiguration -Identity "Finance Depa...............


Desired tag & extendable topic

desired tag and ext content in topic
Signed-off-by: 元守<EMAIL_ADDRESS>

@absolute8511
NewMessageWithExt() constructor added and leave NewMessage() in testcase alone in current commit.
for your review.

@absolute8511 for your review
mainly update:

Remove unnecessary log
RemoveTagClientMsgChan into RemoveClient
Keep default message loop in channel
Client keep tag message channel
benchmark

@absolute8511  for your review
mainly modification after review 2017/07/06:

merge tag client cnt into tag msg channel map
ext param in internalWriteTo
testcase refine
remove lock in Tag setter/getter in client_v2
set tag message channel for client when subEvent received

@absolute8511 for your review
mainly modification after review 2017/07/06:

add test case for normal&ext msg pub & sub
change label in nsqadmin to Ext
Remove struct unused in ext.go
change Tag to TagExt, apply default val for ExtVer
use topic.IsExt() in message encode&decode
delete unnecessary log
remove unused channel in client_v2, avoid using GetTag() in acquiring
msg channel


About page background updated

Fixes Issue🛠

Closes #
Description👨💻

Type of change📄

[] Bug fix (non-breaking change which fixes an issue)
[] New feature (non-breaking change which adds functionality)

How this has been tested✅

Checklist✅

[] My code follows the style guidelines of this project
[] I have performed a self-review of my own code
[] I have commented my code, particularly in hard-to-understand areas
[] I have added demonstration in the form of GIF/video file
[] I am an Open Source Contributor

Screenshots/GIF📷

Mention the issue number for this PR.
Also, attach the screenshots of your work @syannam-12

issue number #766

Edit and fill all the details in the PR description above properly @syannam-12

I have filled the details properly. I hope you let me know in detail if
there's still any problem.
On Fri, Jun 14, 2024, 00:01 Rakesh Roshan @.***> wrote:

Edit and fill all the details in the PR description above properly
@syannam-12 https://github.com/syannam-12
—
Reply to this email directly, view it on GitHub
https://github.com/Rakesh9100/Beautiify/pull/959#issuecomment-2166517541,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZZM2HFSQETPL3HMR7IGZNLZHHQQ7AVCNFSM6AAAAABJA57FAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWGUYTONJUGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

@Rakesh9100 It's been a long time since I provided my PR. Please review it and let me know if there's still any modifications to be done. I hope you do it as soon as possible


[FEATURE REQUEST] ScheduleToStartLatency Metrics

Hi guys, I can't activity_task_schedule_to_start_latency metrics on php-sdk. Is it not implemented yet ?
I found the metrics exist on go-sdk here. Where can I grab this metrics on php-sdk ? I have enabled roadrunner metrics but there is nothing related to the temporal metrics.

Hey @shineirvin , we don't implement these metrics yet. Planned for the near future. Sorry for the inconvenience.

Hi @rustatian thanks for the answer, when will this be implemented?

Hey @shineirvin. We need to finish updating Jobs and gRPC plugins to RR v2. Metrics you found are located in the internal package, so, they can not be imported from our package. These metrics are available by default within the workflowTaskHandler. Have you tried to install Prometheus and use these metrics? According to https://github.com/temporalio/roadrunner-temporal/issues/73#issuecomment-868696461, there are a lot of metrics available. Also, have you tried to configure the temporal server for the metrics?

That is the temporal server metrics, not the worker metrics.
I have to use the temporal worker metrics for HorizontalAutoScalling the desired worker.

Ok, got u. I just wanted to ask, if these temporal server metrics are suitable for you.

When will this be implemented ? Have you schedule it yet ?

@shineirvin These metrics are scheduled for the current sprint (17 Aug - 17 Sept).

In the beta now: https://github.com/temporalio/roadrunner-temporal/pull/82

Thanks a lot @rustatian ! Nice work!

Hi @rustatian, could you please guide me how to expose the metrics ? I can't find any of requrested metrics. Here's what I found.
I put these on .rr.yaml
  # Prometheus client address (path /metrics added automatically).
  #
  # Default: "<IP_ADDRESS>:2112"
  address: "<IP_ADDRESS>:2112

and then I found this.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.7835e-05
go_gc_duration_seconds{quantile="0.25"} 3.7835e-05
go_gc_duration_seconds{quantile="0.5"} 7.2264e-05
go_gc_duration_seconds{quantile="0.75"} 0.005480553
go_gc_duration_seconds{quantile="1"} 0.005480553
go_gc_duration_seconds_sum 0.005590652
go_gc_duration_seconds_count 3
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 50
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.17"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 8.250336e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.3733296e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.453218e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 58881
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0011424553949754845
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.289568e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 8.250336e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.12992e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.764864e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 32636
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.097152e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.1894784e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6311051373002143e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 91517
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 1200
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 123352
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.0687536e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 532742
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 688128
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 688128
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.0005896e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 10
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.26
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 23
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.2944128e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.63110511147e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.53258496e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19```

Hey @shineirvin . These metrics are included in the tests. Please, have a look at the test here and configuration used: https://github.com/temporalio/roadrunner-temporal/blob/master/tests/metrics_test.go

Make sure, that you are made at least 1 call to the activity/workflow.


The described method in the article how to debug is not working you get an authentication failure

[Enter feedback here]
If you follow this article, you will get a AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application: '00000015-0000-0000-c000-000000000000'
Most probably the reply address needs to be modified in AAD

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cb08fb04-94d5-ef25-8162-a0d79f5ee70a
Version Independent ID: 6d43b4b7-cc2e-49bc-564a-5eb5eacefa84
Content: Debug a copy of the production database - Finance & Operations | Dynamics 365
Content Source: articles/fin-ops-core/dev-itpro/database/dbmovement-scenario-debugdiag.md
Service: dynamics-365-cross-app
Product: ****
Technology: ****
GitHub Login: @laneswenka
Microsoft Alias: laswenka

@illese, thank you for contacting us regarding this help topic. I have passed your comment on to the author for consideration.

Closed as won't fix.


Tags?

I spent quite some time on Hugo themes and this one seems to fit to all my needs as nice clean and tight theme for blog about programming.
But I found there is no tags support in it. Are there any plans to do that?

+1 it will be great when you support this!

Added tags in a fork of mine (link). Thanks!


App crashed for GIF images

Check List
Thanks for considering to open an issue. Before you submit your issue, please confirm these boxes are checked.

[x] I have read the wiki page and cheat sheet, but there is no information I need.
[x] I have searched in existing issues, but did not find a same one.
[x] I want to report a problem instead of asking a question. It'd better to use kingfisher tag in Stack Overflow to ask a question.

Issue Description
App Crashes while trying to show carousel of gif images on real device. Working perfectly on simulator.
Below are the list of urls
https://blogmedia.testbook.com/blog/wp-content/uploads/2024/01/exam-notifications-banners-31fa430c.gif
https://blogmedia.testbook.com/blog/wp-content/uploads/2024/01/exam-notifications-banners-24-2cfd5cbb.gif
https://blogmedia.testbook.com/blog/wp-content/uploads/2024/01/exam-notifications-banners-29-304d3b48.gif
https://blogmedia.testbook.com/blog/wp-content/uploads/2024/01/exam-notifications-banners-20-29fddb6c.gif
https://blogmedia.testbook.com/blog/wp-content/uploads/2024/01/exam-notifications-banners-14-4eea21f0.gif
What
Below is the reported error on Crashlytics
GIFBufferInfo::GIFBufferInfo(unsigned char*, bool, unsigned int, unsigned int, unsigned int) + 116

我也遇到了, 来自firebase的crash收集
Crashed: com.apple.main-thread
0  libsystem_platform.dylib       0x91c _platform_memmove + 76
1  ImageIO                        0x1e9634 GIFBufferInfo::GIFBufferInfo(unsigned char*, bool, unsigned int, unsigned int, unsigned int) + 108
2  ImageIO                        0x1fb464 std::__1::__shared_ptr_emplace<GIFBufferInfo, std::__1::allocator<GIFBufferInfo> >::__shared_ptr_emplace<unsigned char*&, bool, unsigned int&, unsigned int&, unsigned int>(std::__1::allocator<GIFBufferInfo>, unsigned char*&, bool&&, unsigned int&, unsigned int&, unsigned int&&) + 76
3  ImageIO                        0x1fb3e4 std::__1::shared_ptr<GIFBufferInfo> std::__1::allocate_shared<GIFBufferInfo, std::__1::allocator<GIFBufferInfo>, unsigned char*&, bool, unsigned int&, unsigned int&, unsigned int, void>(std::__1::allocator<GIFBufferInfo> const&, unsigned char*&, bool&&, unsigned int&, unsigned int&, unsigned int&&) + 88
4  ImageIO                        0x1facd4 GIFReadPlugin::copyImageBlockSet(InfoRec*, CGImageProvider*, CGRect, CGSize, __CFDictionary const*) + 800
5  ImageIO                        0x1a8a0 IIO_Reader::CopyImageBlockSetProc(void*, CGImageProvider*, CGRect, CGSize, __CFDictionary const*) + 280
6  ImageIO                        0x16370 IIOImageProviderInfo::copyImageBlockSetWithOptions(CGImageProvider*, CGRect, CGSize, __CFDictionary const*) + 744
7  ImageIO                        0x1e480 IIOImageProviderInfo::CopyImageBlockSetWithOptions(void*, CGImageProvider*, CGRect, CGSize, __CFDictionary const*) + 836
8  QuartzCore                     0xa4200 CA::Render::copy_image(CGImage*, CGColorSpace*, unsigned int, double, double) + 2944
9  QuartzCore                     0x1430c -[CAKeyframeAnimation _setCARenderAnimation:layer:] + 468
10 QuartzCore                     0x140cc -[CAKeyframeAnimation _copyRenderAnimationForLayer:] + 172
11 QuartzCore                     0x13908 CA::Layer::commit_animations(CA::Transaction*, double (*)(CA::Layer*, double, void*), void (*)(CA::Layer*, CA::Render::Animation*, void*), void (*)(CA::Layer*, __CFString const*, void*), CA::Render::TimingList* (*)(CA::Layer*, void*), void*) + 780
12 QuartzCore                     0x2620 invocation function for block in CA::Context::commit_transaction(CA::Transaction*, double, double*) + 220
13 QuartzCore                     0x50388 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 368
14 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
15 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
16 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
17 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
18 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
19 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
20 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
21 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
22 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
23 QuartzCore                     0x50314 CA::Layer::commit_if_needed(CA::Transaction*, void (CA::Layer*, unsigned int, unsigned int) block_pointer) + 252
24 QuartzCore                     0x2f928 CA::Context::commit_transaction(CA::Transaction*, double, double*) + 6320
25 QuartzCore                     0x63630 CA::Transaction::commit() + 652
26 QuartzCore                     0x4c8b4 CA::Transaction::flush_as_runloop_observer(bool) + 88
27 CoreFoundation                 0x91d30 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 36
28 CoreFoundation                 0x1a280 __CFRunLoopDoObservers + 532
29 CoreFoundation                 0x7bc50 __CFRunLoopRun + 1048
30 CoreFoundation                 0x80eb0 CFRunLoopRunSpecific + 612
31 GraphicsServices               0x1368 GSEventRunModal + 164
32 UIKitCore                      0x3a1668 -[UIApplication _run] + 888
33 UIKitCore                      0x3a12cc UIApplicationMain + 340
34 AmongChat                      0x9944 main + 12 (main.swift:12)
35 ???                            0x1c37d0960 (Missing)

SDWebImage 似乎也有类似问题https://github.com/SDWebImage/SDWebImage/issues/3515


Support for beta feature Concurrency

GitHub workflows have now a feature in beta, concurrency. One can read more in here: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#concurrency
The feature enables to stop an already running workflow when a new one is enqueued. Enabling this would save lots of time on CI servers for jobs that are already obsolete.
To configure it, this should be added to the continuous integration yml file:
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

@djspiewak Would you consider adding this feature even though is still in Beta on GitHub side? I would be willing to contribute it.

@jlprat Finally got around to this! This looks really awesome honestly, and feels like something that should definitely be well integrated into this plugin. Would love to have it as a feature!
In particular, I think we should ensure that it's configurable, but the default should be to group based on Pull Request specifier (I believe that is what github.ref covers) or upstream branch if not on a PR.

@mdedetrich this is useful. sbt-typelevel's equivalent plugin to this supports it.

There are plans to upstream this