unix  3      [ ]         STREAM     CONNECTED     23159    /run/dbus/system_bus_socket
unix  3      [ ]         DGRAM      CONNECTED     3220
unix  3      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/s/9db198f2f76fd354ed64de3888d049421b347b390aa1810d76cc161adafa281b
unix  3      [ ]         STREAM     CONNECTED     4587
unix  3      [ ]         STREAM     CONNECTED     1313
unix  3      [ ]         DGRAM      CONNECTED     1227
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     18868
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     23160
unix  3      [ ]         STREAM     CONNECTED     21862
unix  3      [ ]         DGRAM      CONNECTED     3219
unix  2      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     2566
unix  3      [ ]         STREAM     CONNECTED     19896
unix  3      [ ]         STREAM     CONNECTED     1208
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/containerd.sock.ttrpc
unix  3      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     3352
unix  3      [ ]         STREAM     CONNECTED     152289265 /run/containerd/s/1ed0aa4a2fa3da2ff4790f6ab76414bb99c0dff3869e17f2e9ee0684f57a692d
unix  3      [ ]         STREAM     CONNECTED     18874
unix  3      [ ]         STREAM     CONNECTED     18841
unix  3      [ ]         DGRAM      CONNECTED     23701
unix  3      [ ]         STREAM     CONNECTED     18906    /run/containerd/containerd.sock
unix  2      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         DGRAM      CONNECTED     2376
unix  3      [ ]         STREAM     CONNECTED     152289979 /run/containerd/s/1ed0aa4a2fa3da2ff4790f6ab76414bb99c0dff3869e17f2e9ee0684f57a692d
unix  2      [ ]         DGRAM      CONNECTED     1331
unix  3      [ ]         STREAM     CONNECTED     22062
unix  3      [ ]         STREAM     CONNECTED     23199    /run/containerd/s/bc145d3ada58815c9d0d0d295cb94a012af019002fc10ec313e70a4048e7eda4
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     23945
unix  3      [ ]         STREAM     CONNECTED     18861    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     152293644
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     2574
unix  3      [ ]         STREAM     CONNECTED     19871    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     18863    /run/systemd/journal/stdout
unix  3      [ ]         DGRAM      CONNECTED     3218
unix  3      [ ]         STREAM     CONNECTED     762
unix  3      [ ]         STREAM     CONNECTED     24611    /run/containerd/containerd.sock.ttrpc
unix  3      [ ]         STREAM     CONNECTED     22042    /run/systemd/journal/stdout
unix  3      [ ]         DGRAM      CONNECTED     23700
unix  2      [ ]         DGRAM      CONNECTED     2565
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/s/87d624b28c6ba2a53863d80b6d245e4efd47a6a99ccf744304612fa196d52017
unix  2      [ ]         DGRAM      CONNECTED     1131
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     23207
unix  3      [ ]         STREAM     CONNECTED     22656
unix  3      [ ]         STREAM     CONNECTED     23739
unix  2      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     22059
unix  3      [ ]         STREAM     CONNECTED     22720    /run/dbus/system_bus_socket
unix  3      [ ]         DGRAM      CONNECTED     3217
unix  3      [ ]         STREAM     CONNECTED     19892    /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED     21859    /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM      CONNECTED     756
unix  3      [ ]         STREAM     CONNECTED     19895
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/containerd.sock.ttrpc
unix  3      [ ]         STREAM     CONNECTED     23709
unix  3      [ ]         STREAM     CONNECTED     3244     /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     5074     /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/s/eb83d8eb5e692f378a8db54cb184e9aaad4b8e9f8f1f626b88c326cfbaf625eb
unix  3      [ ]         STREAM     CONNECTED     152289264
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         DGRAM      CONNECTED     2377
unix  3      [ ]         DGRAM      CONNECTED     1228
unix  3      [ ]         STREAM     CONNECTED     2479     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     1431     /run/containerd/s/0f2775cd4200e32cdd861974a362ca03cb4e899f7391e299da416b4448261b56
unix  3      [ ]         STREAM     CONNECTED     3262
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER>
unix  3      [ ]         DGRAM      CONNECTED     18778
unix  3      [ ]         STREAM     CONNECTED     2515
unix  2      [ ]         DGRAM      CONNECTED<PHONE_NUMBER>
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     23743    /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     4604     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     23741    /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     3239     /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     716      /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     715      /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     22985
unix  3      [ ]         STREAM     CONNECTED<PHONE_NUMBER> /run/containerd/s/9db198f2f76fd354ed64de3888d049421b347b390aa1810d76cc161adafa281b
root@k8s-eu-1-worker-3:~#

Do you see any critical element, or any resources that are running out?

@dothebart is this the problem: the memory used ?:
root@k8s-eu-1-worker-5:~#  sar -r 1 5
Linux 5.15.0-67-generic (k8s-eu-1-worker-5) 	09/26/23 	x86_64	(4 CPU)
17:36:29    kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
17:36:30      2113844   2579724   5191216     63.77     22008    730632   2185104     26.84    257840   5596016        72
17:36:31      2113604   2579484   5191448     63.77     22008    730636   2185104     26.84    257836   5596088        72
17:36:32      2113356   2579236   5191696     63.78     22008    730636   2185104     26.84    257836   5596088        72
17:36:33      2113104   2578984   5191948     63.78     22008    730636   2185104     26.84    257836   5596332        72
17:36:34      2112628   2578508   5192424     63.79     22008    730636   2185104     26.84    257836   5596512        72
Average:      2113307   2579187   5191746     63.78     22008    730635   2185104     26.84    257837   5596207        72
root@k8s-eu-1-worker-5:~#

root@k8s-eu-1-worker-4:~# sar -r 1 5
Linux 5.15.0-67-generic (k8s-eu-1-worker-4) 	09/26/23 	_x86_64_	(4 CPU)

17:38:01    kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
17:38:02      2476696   3025084   4744680     58.29     23356    811472   2220392     27.28    299184   5192480        80
17:38:03      2476696   3025092   4744672     58.29     23356    811480   2220392     27.28    299192   5192528        88
17:38:04      2473672   3022068   4747696     58.32     23356    811480   2466024     30.29    299196   5201340        80
17:38:05      2477024   3025416   4744268     58.28     23356    811480   2224288     27.32    299196   5194204        76
17:38:06      2476776   3025168   4744552     58.28     23356    811480   2220580     27.28    299192   5192204         8
Average:      2476173   3024566   4745174     58.29     23356    811478   2270335     27.89    299192   5194551        66

And this is the complete description of the arangodeployment/cluster :
root@k8s-eu-1-master:~# kubectl describe  arangodeployment/cluster
Name:         cluster
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  database.arangodb.com/v1
Kind:         ArangoDeployment
Metadata:
  Creation Timestamp:  2023-04-07T08:58:57Z
  Finalizers:
    database.arangodb.com/remove-child-finalizers
  Generation:  1637803
  Managed Fields:
    API Version:  database.arangodb.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:agents:
          .:
          f:volumeClaimTemplate:
            .:
            f:spec:
              .:
              f:accessModes:
              f:resources:
                .:
                f:requests:
                  .:
                  f:storage:
              f:storageClassName:
              f:volumeMode:
        f:dbservers:
          .:
          f:volumeClaimTemplate:
            .:
            f:spec:
              .:
              f:accessModes:
              f:resources:
                .:
                f:requests:
                  .:
                  f:storage:
              f:storageClassName:
              f:volumeMode:
        f:mode:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-04-07T08:58:57Z
    API Version:  database.arangodb.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"database.arangodb.com/remove-child-finalizers":
      f:spec:
        f:agents:
          f:count:
          f:resources:
          f:volumeClaimTemplate:
            f:metadata:
              .:
              f:creationTimestamp:
            f:status:
        f:auth:
          .:
          f:jwtSecretName:
        f:bootstrap:
          .:
          f:passwordSecretNames:
            .:
            f:root:
        f:chaos:
          .:
          f:interval:
          f:kill-pod-probability:
        f:coordinators:
          .:
          f:count:
          f:resources:
        f:dbservers:
          f:count:
          f:resources:
          f:volumeClaimTemplate:
            f:metadata:
              .:
              f:creationTimestamp:
            f:status:
        f:environment:
        f:externalAccess:
        f:image:
        f:imagePullPolicy:
        f:license:
        f:lifecycle:
          .:
          f:resources:
        f:metrics:
          .:
          f:authentication:
            .:
            f:jwtTokenSecretName:
          f:enabled:
          f:resources:
        f:rocksdb:
          .:
          f:encryption:
        f:single:
          .:
          f:resources:
          f:volumeClaimTemplate:
            .:
            f:metadata:
              .:
              f:creationTimestamp:
            f:spec:
              .:
              f:accessModes:
              f:resources:
                .:
                f:requests:
                  .:
                  f:storage:
              f:volumeMode:
            f:status:
        f:storageEngine:
        f:sync:
          .:
          f:auth:
            .:
            f:clientCASecretName:
            f:jwtSecretName:
          f:externalAccess:
          f:image:
          f:monitoring:
            .:
            f:tokenSecretName:
          f:tls:
            .:
            f:caSecretName:
            f:ttl:
        f:syncmasters:
          .:
          f:resources:
        f:syncworkers:
          .:
          f:resources:
        f:tls:
          .:
          f:caSecretName:
          f:ttl:
      f:status:
        .:
        f:accepted-spec:
          .:
          f:agents:
            .:
            f:count:
            f:resources:
            f:volumeClaimTemplate:
              .:
              f:metadata:
                .:
                f:creationTimestamp:
              f:spec:
                .:
                f:accessModes:
                f:resources:
                  .:
                  f:requests:
                    .:
                    f:storage:
                f:storageClassName:
                f:volumeMode:
              f:status:
          f:auth:
            .:
            f:jwtSecretName:
          f:bootstrap:
            .:
            f:passwordSecretNames:
              .:
              f:root:
          f:chaos:
            .:
            f:interval:
            f:kill-pod-probability:
          f:coordinators:
            .:
            f:count:
            f:resources:
          f:dbservers:
            .:
            f:count:
            f:resources:
            f:volumeClaimTemplate:
              .:
              f:metadata:
                .:
                f:creationTimestamp:
              f:spec:
                .:
                f:accessModes:
                f:resources:
                  .:
                  f:requests:
                    .:
                    f:storage:
                f:storageClassName:
                f:volumeMode:
              f:status:
          f:environment:
          f:externalAccess:
          f:image:
          f:imagePullPolicy:
          f:license:
          f:lifecycle:
            .:
            f:resources:
          f:metrics:
            .:
            f:authentication:
              .:
              f:jwtTokenSecretName:
            f:enabled:
            f:resources:
          f:mode:
          f:rocksdb:
            .:
            f:encryption:
          f:single:
            .:
            f:resources:
            f:volumeClaimTemplate:
              .:
              f:metadata:
                .:
                f:creationTimestamp:
              f:spec:
                .:
                f:accessModes:
                f:resources:
                  .:
                  f:requests:
                    .:
                    f:storage:
                f:volumeMode:
              f:status:
          f:storageEngine:
          f:sync:
            .:
            f:auth:
              .:
              f:clientCASecretName:
              f:jwtSecretName:
            f:externalAccess:
            f:image:
            f:monitoring:
              .:
              f:tokenSecretName:
            f:tls:
              .:
              f:caSecretName:
              f:ttl:
          f:syncmasters:
            .:
            f:resources:
          f:syncworkers:
            .:
            f:resources:
          f:tls:
            .:
            f:caSecretName:
            f:ttl:
        f:acceptedSpecVersion:
        f:agency:
          .:
          f:ids:
          f:size:
        f:agents:
        f:appliedVersion:
        f:arangodb-images:
        f:backoff:
          .:
          f:check:
          f:license:
          f:timezone:
        f:conditions:
        f:coordinators:
        f:current-image:
          .:
          f:arangodb-version:
          f:image:
          f:image-id:
        f:dbservers:
        f:hashes:
          .:
          f:jwt:
            .:
            f:active:
          f:rocksDBEncryption:
          f:tls:
            .:
            f:ca:
            f:propagated:
            f:truststore:
        f:members:
          .:
          f:agents:
          f:coordinators:
          f:dbservers:
        f:phase:
        f:secret-hashes:
          .:
          f:auth-jwt:
        f:serviceName:
        f:version:
    Manager:         arangodb_operator
    Operation:       Update
    Time:            2023-09-27T11:26:46Z
  Resource Version:  56603074
  UID:               5c2358f0-45cf-47a6-9038-8594bc374e37
Spec:
  Agents:
    Count:  3
    Resources:
    Volume Claim Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         1Gi
        Storage Class Name:  my-local-ssd
        Volume Mode:         Filesystem
      Status:
  Auth:
    Jwt Secret Name:  cluster-jwt
  Bootstrap:
    Password Secret Names:
      Root:  None
  Chaos:
    Interval:<PHONE_NUMBER>0
    Kill - Pod - Probability:  50
  Coordinators:
    Count:  3
    Resources:
  Dbservers:
    Count:  3
    Resources:
    Volume Claim Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         1Gi
        Storage Class Name:  my-local-ssd
        Volume Mode:         Filesystem
      Status:
  Environment:  Development
  External Access:
  Image:              arangodb/arangodb:latest
  Image Pull Policy:  IfNotPresent
  License:
  Lifecycle:
    Resources:
  Metrics:
    Authentication:
      Jwt Token Secret Name:  cluster-exporter-jwt-token
    Enabled:                  false
    Resources:
  Mode:  Cluster
  Rocksdb:
    Encryption:
  Single:
    Resources:
    Volume Claim Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  8Gi
        Volume Mode:  Filesystem
      Status:
  Storage Engine:  RocksDB
  Sync:
    Auth:
      Client CA Secret Name:  cluster-sync-client-auth-ca
      Jwt Secret Name:        cluster-sync-jwt
    External Access:
    Image:  <nil>
    Monitoring:
      Token Secret Name:  cluster-sync-mt
    Tls:
      Ca Secret Name:  cluster-sync-ca
      Ttl:             2610h
  Syncmasters:
    Resources:
  Syncworkers:
    Resources:
  Tls:
    Ca Secret Name:  cluster-ca
    Ttl:             2610h
Status:
  Accepted - Spec:
    Agents:
      Count:  3
      Resources:
      Volume Claim Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:         1Gi
          Storage Class Name:  my-local-ssd
          Volume Mode:         Filesystem
        Status:
    Auth:
      Jwt Secret Name:  cluster-jwt
    Bootstrap:
      Password Secret Names:
        Root:  None
    Chaos:
      Interval:<PHONE_NUMBER>0
      Kill - Pod - Probability:  50
    Coordinators:
      Count:  3
      Resources:
    Dbservers:
      Count:  3
      Resources:
      Volume Claim Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:         1Gi
          Storage Class Name:  my-local-ssd
          Volume Mode:         Filesystem
        Status:
    Environment:  Development
    External Access:
    Image:              arangodb/arangodb:latest
    Image Pull Policy:  IfNotPresent
    License:
    Lifecycle:
      Resources:
    Metrics:
      Authentication:
        Jwt Token Secret Name:  cluster-exporter-jwt-token
      Enabled:                  false
      Resources:
    Mode:  Cluster
    Rocksdb:
      Encryption:
    Single:
      Resources:
      Volume Claim Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec:
          Access Modes:
            ReadWriteOnce
          Resources:
            Requests:
              Storage:  8Gi
          Volume Mode:  Filesystem
        Status:
    Storage Engine:  RocksDB
    Sync:
      Auth:
        Client CA Secret Name:  cluster-sync-client-auth-ca
        Jwt Secret Name:        cluster-sync-jwt
      External Access:
      Image:  <nil>
      Monitoring:
        Token Secret Name:  cluster-sync-mt
      Tls:
        Ca Secret Name:  cluster-sync-ca
        Ttl:             2610h
    Syncmasters:
      Resources:
    Syncworkers:
      Resources:
    Tls:
      Ca Secret Name:     cluster-ca
      Ttl:                2610h
  Accepted Spec Version:  141781fecea250fea200b468ab647676bd44d23a5d16b94bcff1448ade837b61
  Agency:
    Ids:
      AGNT-apt6zd1n
      AGNT-mwchqsli
      AGNT-uhbsq34m
    Size:  3
  Agents:
  Applied Version:  141781fecea250fea200b468ab647676bd44d23a5d16b94bcff1448ade837b61
  Arangodb - Images:
    Arangodb - Version:  3.10.5
    Image:               arangodb/arangodb:latest
    Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
  Backoff:
    Check:     2023-09-27T11:27:13Z
    License:   2023-09-27T11:27:16Z
    Timezone:  2023-09-27T11:27:13Z
  Conditions:
    Hash:                  141781fecea250fea200b468ab647676bd44d23a5d16b94bcff1448ade837b61
    Last Transition Time:  2023-04-07T08:58:58Z
    Last Update Time:      2023-04-07T08:58:58Z
    Message:               Spec Object accepted
    Reason:                Spec Accepted
    Status:                True
    Type:                  SpecAccepted
    Last Transition Time:  2023-09-20T12:41:08Z
    Last Update Time:      2023-09-20T12:41:08Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-09-27T11:04:31Z
    Last Update Time:      2023-09-27T11:04:31Z
    Reason:                ArangoDB is reachable
    Status:                True
    Type:                  Reachable
    Last Transition Time:  2023-04-07T09:01:36Z
    Last Update Time:      2023-04-07T09:01:36Z
    Message:               The bootstrap process has been completed successfully
    Reason:                Bootstrap successful
    Status:                True
    Type:                  BootstrapCompleted
    Last Transition Time:  2023-04-07T09:01:36Z
    Last Update Time:      2023-04-07T09:01:36Z
    Message:               The bootstrap process has been completed successfully
    Reason:                Bootstrap successful
    Status:                True
    Type:                  BootstrapSucceded
    Hash:                  141781fecea250fea200b468ab647676bd44d23a5d16b94bcff1448ade837b61
    Last Transition Time:  2023-09-27T11:23:17Z
    Last Update Time:      2023-09-27T11:23:17Z
    Message:               Spec is Up To Date
    Reason:                Spec is Up To Date
    Status:                True
    Type:                  UpToDate
    Last Transition Time:  2023-04-07T09:01:52Z
    Last Update Time:      2023-04-07T09:01:52Z
    Message:               Spec is Propagated
    Reason:                Spec is Propagated
    Status:                True
    Type:                  SpecPropagated
  Coordinators:
  Current - Image:
    Arangodb - Version:  3.10.5
    Image:               arangodb/arangodb:latest
    Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
  Dbservers:
  Hashes:
    Jwt:
      Active:  sha256:f65ced043d51903980b16f97ff3751d3de21af3b944174323081e5228fda98be
    Rocks DB Encryption:
    Tls:
      Ca:          sha256:526d6fd5f02b46f902c05e0de4f6f41d9ab44b4ad2fe8f67f00ccb20a30b79b0
      Propagated:  true
      Truststore:
        sha256:526d6fd5f02b46f902c05e0de4f6f41d9ab44b4ad2fe8f67f00ccb20a30b79b0
  Members:
    Agents:
      Arango - Version:  3.10.5
      Architecture:      amd64
      Cid:               5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-09-27T11:22:43Z
        Last Update Time:      2023-09-27T11:22:43Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T03:01:16Z
        Last Update Time:      2023-07-18T03:01:16Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T03:01:16Z
        Last Update Time:      2023-07-18T03:01:16Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T03:00:59Z
        Last Update Time:      2023-07-18T03:00:59Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T03:01:16Z
        Last Update Time:      2023-07-18T03:01:16Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T03:01:16Z
        Last Update Time:      2023-07-18T03:01:16Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:14Z
      Id:                      AGNT-apt6zd1n
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-agent-apt6zd1n
      Persistent Volume Claim Name:  cluster-agent-apt6zd1n
      Phase:                         Created
      Pod:
        Name:                 cluster-agnt-apt6zd1n-75b1c6
        Spec Version:         43e8397927d8d6a50f2bd060b1344ad24b8a7bcd3228a258eaec56132bf2ec21
        UID:                  4bc89af5-dd53-46fa-ba25-e0229fa6e496
      Pod Name:               cluster-agnt-apt6zd1n-75b1c6
      Pod Spec Version:       43e8397927d8d6a50f2bd060b1344ad24b8a7bcd3228a258eaec56132bf2ec21
      Pod UID:                4bc89af5-dd53-46fa-ba25-e0229fa6e496
      Recent - Terminations:  <nil>
      Rid:                    e5ca0bd2-9097-4ad4-9290-adf5004a7077
      UID:                    24568e17-e730-49d4-bb59-9c1eb9b38378
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-09-27T11:24:29Z
        Last Update Time:      2023-09-27T11:24:29Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T03:03:00Z
        Last Update Time:      2023-07-18T03:03:00Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T03:03:17Z
        Last Update Time:      2023-07-18T03:03:17Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T03:03:17Z
        Last Update Time:      2023-07-18T03:03:17Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:14Z
      Id:                      AGNT-mwchqsli
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-agent-mwchqsli
      Persistent Volume Claim Name:  cluster-agent-mwchqsli
      Phase:                         Created
      Pod:
        Name:                 cluster-agnt-mwchqsli-75b1c6
        Spec Version:         d927411947f6bb1492ee95f3dc9aa1986ecbb47a8c54f838d06e7e1b8bb7f2c0
        UID:                  555f1327-dfa3-4a12-981d-b1b78abd4f27
      Pod Name:               cluster-agnt-mwchqsli-75b1c6
      Pod Spec Version:       d927411947f6bb1492ee95f3dc9aa1986ecbb47a8c54f838d06e7e1b8bb7f2c0
      Pod UID:                555f1327-dfa3-4a12-981d-b1b78abd4f27
      Recent - Terminations:  <nil>
      Rid:                    5ec3e3d7-e534-4a3f-bc22-7edc9c1a02fc
      UID:                    07997836-7702-42ac-9d11-7fefd4bbd576
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-09-27T11:20:50Z
        Last Update Time:      2023-09-27T11:20:50Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T03:05:06Z
        Last Update Time:      2023-07-18T03:05:06Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T03:05:23Z
        Last Update Time:      2023-07-18T03:05:23Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T03:05:23Z
        Last Update Time:      2023-07-18T03:05:23Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:14Z
      Id:                      AGNT-uhbsq34m
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-agent-uhbsq34m
      Persistent Volume Claim Name:  cluster-agent-uhbsq34m
      Phase:                         Created
      Pod:
        Name:                 cluster-agnt-uhbsq34m-75b1c6
        Spec Version:         a02dcd9006d8573b785d306a54b2662bc210c49ad83cd55c60a822d226cede80
        UID:                  906b7f47-313d-4c2f-807c-0431792a4eb1
      Pod Name:               cluster-agnt-uhbsq34m-75b1c6
      Pod Spec Version:       a02dcd9006d8573b785d306a54b2662bc210c49ad83cd55c60a822d226cede80
      Pod UID:                906b7f47-313d-4c2f-807c-0431792a4eb1
      Recent - Terminations:  <nil>
      Rid:                    404a70ff-aaec-4fac-bc86-45f968c77561
      UID:                    1c58c0e3-e7ce-49b1-971c-1c70428f6a29
    Coordinators:
      Arango - Version:  3.10.5
      Architecture:      amd64
      Cid:               5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:22:43Z
        Last Update Time:      2023-09-27T11:22:43Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T11:19:36Z
        Last Update Time:      2023-07-18T11:19:36Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T11:19:36Z
        Last Update Time:      2023-07-18T11:19:36Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T07:28:01Z
        Last Update Time:      2023-07-18T07:28:01Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T07:28:06Z
        Last Update Time:      2023-07-18T07:28:06Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T07:28:11Z
        Last Update Time:      2023-07-18T07:28:11Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:16Z
      Id:                      CRDN-2avzgbi1
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Phase:                 Created
      Pod:
        Name:                 cluster-crdn-2avzgbi1-75b1c6
        Spec Version:         28ae45b80375463d87d50f208376c8188fbc81e412513436d0ef4413b8164054
        UID:                  b6a47586-7baa-4d61-ac30-122ab299c788
      Pod Name:               cluster-crdn-2avzgbi1-75b1c6
      Pod Spec Version:       28ae45b80375463d87d50f208376c8188fbc81e412513436d0ef4413b8164054
      Pod UID:                b6a47586-7baa-4d61-ac30-122ab299c788
      Recent - Terminations:  <nil>
      Rid:                    a833e91a-088d-4550-9bdd-28c77e659e6e
      UID:                    be93f987-a2c9-4de7-bbeb-9bd060486322
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:23:56Z
        Last Update Time:      2023-09-27T11:23:56Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T11:20:09Z
        Last Update Time:      2023-07-18T11:20:09Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T11:20:09Z
        Last Update Time:      2023-07-18T11:20:09Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T11:19:59Z
        Last Update Time:      2023-07-18T11:19:59Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T11:20:04Z
        Last Update Time:      2023-07-18T11:20:04Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T11:20:09Z
        Last Update Time:      2023-07-18T11:20:09Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:16Z
      Id:                      CRDN-ets3nigq
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Phase:                 Created
      Pod:
        Name:                 cluster-crdn-ets3nigq-75b1c6
        Spec Version:         646cec6d1ce8eccb2ec64d2cdfe6f818c6c1a2030bf976d47145ca8862f00bee
        UID:                  91f187da-da5d-4ed2-b0dc-448b8570665a
      Pod Name:               cluster-crdn-ets3nigq-75b1c6
      Pod Spec Version:       646cec6d1ce8eccb2ec64d2cdfe6f818c6c1a2030bf976d47145ca8862f00bee
      Pod UID:                91f187da-da5d-4ed2-b0dc-448b8570665a
      Recent - Terminations:  <nil>
      Rid:                    5ad93f4a-8218-40b8-b5ea-a7403b61bc6e
      UID:                    4563c57f-8371-4149-88ba-9ffa74ca63de
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:20:50Z
        Last Update Time:      2023-09-27T11:20:50Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-09-20T12:41:08Z
        Last Update Time:      2023-09-20T12:41:08Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-09-20T12:41:08Z
        Last Update Time:      2023-09-20T12:41:08Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-09-20T12:40:57Z
        Last Update Time:      2023-09-20T12:40:57Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-09-20T12:41:05Z
        Last Update Time:      2023-09-20T12:41:05Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-09-20T12:41:08Z
        Last Update Time:      2023-09-20T12:41:08Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:16Z
      Id:                      CRDN-n3r0x9by
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Phase:                 Created
      Pod:
        Name:                 cluster-crdn-n3r0x9by-75b1c6
        Spec Version:         f40a176f78840a8d934947d2072f89816fd3f761466f7308549b2fa7387228c1
        UID:                  e984982a-86a7-474f-8bca-a185df4b089b
      Pod Name:               cluster-crdn-n3r0x9by-75b1c6
      Pod Spec Version:       f40a176f78840a8d934947d2072f89816fd3f761466f7308549b2fa7387228c1
      Pod UID:                e984982a-86a7-474f-8bca-a185df4b089b
      Recent - Terminations:  <nil>
      Rid:                    b1c8ed25-ac5a-47aa-bda3-6fc97dd3729d
      UID:                    8b80af3d-b262-4fb5-ac59-be93880d7b05
    Dbservers:
      Arango - Version:  3.10.5
      Architecture:      amd64
      Cid:               5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:24:29Z
        Last Update Time:      2023-09-27T11:24:29Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-07-18T11:19:33Z
        Last Update Time:      2023-07-18T11:19:33Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-07-18T03:06:06Z
        Last Update Time:      2023-07-18T03:06:06Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-07-18T03:06:27Z
        Last Update Time:      2023-07-18T03:06:27Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-07-18T03:06:27Z
        Last Update Time:      2023-07-18T03:06:27Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:15Z
      Id:                      PRMR-07wqascu
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-dbserver-07wqascu
      Persistent Volume Claim Name:  cluster-dbserver-07wqascu
      Phase:                         Created
      Pod:
        Name:                 cluster-prmr-07wqascu-75b1c6
        Spec Version:         59984172a8e556331e54eac53d93573bf853b563b75f9e907b0d9130cb2ae937
        UID:                  02420fdb-17d9-41dd-94d9-e25889ef28b7
      Pod Name:               cluster-prmr-07wqascu-75b1c6
      Pod Spec Version:       59984172a8e556331e54eac53d93573bf853b563b75f9e907b0d9130cb2ae937
      Pod UID:                02420fdb-17d9-41dd-94d9-e25889ef28b7
      Recent - Terminations:  <nil>
      Rid:                    95e11d65-f151-4dcc-8c4c-4043c325356f
      UID:                    e3efbb2f-60f2-43c5-b950-d22ee58de227
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:22:43Z
        Last Update Time:      2023-09-27T11:22:43Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-09-03T05:02:43Z
        Last Update Time:      2023-09-03T05:02:43Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-09-03T05:02:43Z
        Last Update Time:      2023-09-03T05:02:43Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-09-03T05:02:31Z
        Last Update Time:      2023-09-03T05:02:31Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-09-03T05:02:39Z
        Last Update Time:      2023-09-03T05:02:39Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-09-03T05:02:43Z
        Last Update Time:      2023-09-03T05:02:43Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:15Z
      Id:                      PRMR-kenmxcns
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-dbserver-kenmxcns
      Persistent Volume Claim Name:  cluster-dbserver-kenmxcns
      Phase:                         Created
      Pod:
        Name:                 cluster-prmr-kenmxcns-75b1c6
        Spec Version:         c94260e75d3fa94a37a13aded2418dccea7176d1d756eef6cf89c56378fdd49f
        UID:                  c2b6d931-8e6f-493c-912f-c75f91156a8d
      Pod Name:               cluster-prmr-kenmxcns-75b1c6
      Pod Spec Version:       c94260e75d3fa94a37a13aded2418dccea7176d1d756eef6cf89c56378fdd49f
      Pod UID:                c2b6d931-8e6f-493c-912f-c75f91156a8d
      Recent - Terminations:  <nil>
      Rid:                    ec50295a-1813-417e-98b6-de50abd84449
      UID:                    fbe1ad5f-8fcd-4d35-ac53-490a15c6f43e
      Arango - Version:       3.10.5
      Architecture:           amd64
      Cid:                    5c2358f0-45cf-47a6-9038-8594bc374e37
      Conditions:
        Last Transition Time:  2023-04-07T09:01:05Z
        Last Update Time:      2023-04-07T09:01:05Z
        Status:                True
        Type:                  MemberOfCluster
        Last Transition Time:  2023-09-27T11:22:43Z
        Last Update Time:      2023-09-27T11:22:43Z
        Reason:                ArangoDB is reachable
        Status:                True
        Type:                  Reachable
        Last Transition Time:  2023-09-03T05:03:31Z
        Last Update Time:      2023-09-03T05:03:31Z
        Reason:                Pod Ready
        Status:                True
        Type:                  Ready
        Last Transition Time:  2023-09-03T05:03:31Z
        Last Update Time:      2023-09-03T05:03:31Z
        Reason:                Pod Serving
        Status:                True
        Type:                  Serving
        Last Transition Time:  2023-09-03T05:03:19Z
        Last Update Time:      2023-09-03T05:03:19Z
        Reason:                Pod is scheduled
        Status:                True
        Type:                  Scheduled
        Last Transition Time:  2023-09-03T05:03:29Z
        Last Update Time:      2023-09-03T05:03:29Z
        Reason:                Core Pod Container started
        Status:                True
        Type:                  Active
        Last Transition Time:  2023-09-03T05:03:31Z
        Last Update Time:      2023-09-03T05:03:31Z
        Reason:                Pod Started
        Status:                True
        Type:                  Started
      Created - At:            2023-04-07T08:59:15Z
      Id:                      PRMR-nsgmx9lg
      Image:
        Arangodb - Version:  3.10.5
        Image:               arangodb/arangodb:latest
        Image - Id:          docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Image - Id:            docker.io/arangodb/arangodb@sha256:a8f4da685bb4d98c8e912feaa5d2ad812ec4f378c1d984d7bc4bb10f05586d90
      Initialized:           true
      Persistent Volume Claim:
        Name:                        cluster-dbserver-nsgmx9lg
      Persistent Volume Claim Name:  cluster-dbserver-nsgmx9lg
      Phase:                         Created
      Pod:
        Name:                 cluster-prmr-nsgmx9lg-75b1c6
        Spec Version:         8cf7916d69bc2b07168435e1d3165f65730fe21d372b8993343a7a399a411723
        UID:                  2b020fcf-393d-4095-94e0-4af2532a6ae8
      Pod Name:               cluster-prmr-nsgmx9lg-75b1c6
      Pod Spec Version:       8cf7916d69bc2b07168435e1d3165f65730fe21d372b8993343a7a399a411723
      Pod UID:                2b020fcf-393d-4095-94e0-4af2532a6ae8
      Recent - Terminations:  <nil>
      Rid:                    2b2585dd-14aa-4086-bfcd-204cd4a8e2eb
      UID:                    ae37a3ac-f20c-409d-aa45-d7e9fae9560a
  Phase:                      Running
  Secret - Hashes:
    Auth - Jwt:  65f3fb856a2b218c2db108daa7d30a9eba6ee99b0c67ae141e3aed70f239237c
  Service Name:  cluster
  Version:       <IP_ADDRESS>
Events:
  Type    Reason             Age                    From                                            Message
  ----    ------             ----                   ----                                            -------
  Normal  Plan Action added  3m35s (x217 over 20h)  arango-kube-arangodb-operator-844d65cdd5-88shw  A plan item of type Idle has been added

Any ideas about how to solve the issue?

Hi,
You need to make sure your system is properly sized and I/O overload doesn't happen.
Possible measures include using bulk operations in the client code, or changing the hardware.
So you should identify the parts of your usage patterns that run the system over its limits, and then work on improving them.

@dothebart did you have a look at what I in depth reported
here: https://github.com/arangodb/arangodb/issues/19823#issuecomment-1735452759
and here : https://github.com/arangodb/arangodb/issues/19823#issuecomment-1737214702 ?
Can you please point me the specific, detailed critical points you found here: https://github.com/arangodb/arangodb/issues/19823#issuecomment-1735452759 and here https://github.com/arangodb/arangodb/issues/19823#issuecomment-1737214702 ?
Without spotting the precise, specific critical points to improve, I cannot make them improve

Sorry, that doesn't give useful information to me.
Its you who run the application and know the queries you run, the code you use; you need to work on these; you wrote nothing about them.
Please take a look at https://docs.arangodb.com/3.11/operations/administration/reduce-memory-footprint/ it explains how ram usage etc. plays together with load etc.
If you need more in depth consulting about how to optimize your application or size your hardware, a chat via arangodb.com may better bring you to the results you expect.

Hello @raphael10-collab
I see that you are using operator version 1.2.24 which seems quite outdated. Can you please try next things?

Update to the latest operator (1.2.33). There were many changes related to local storage handling.
Try reproducing the issue without local-ssd provisioned: e.g. try spin-up the simple-cluster-no-tls.yaml from https://github.com/arangodb/kube-arangodb/tree/master/examples

If this does not help, please provide the next info:

List pods running for your deployment
What is the network delay between your two nodes?

Please ping me here for quick reply 👌🏻
Best,
Nikita

Hello!
As it might be a network issue, can you use ip or headless-dns communication method to skip the DNS step in the cluster?
Doc: https://github.com/arangodb/kube-arangodb/blob/master/docs/api/ArangoDeployment.V1.md#speccommunicationmethod-string
Note: It will restart all pods in the cluster
Best,
Adam.

@nikita-vanyasin @ajanikow
I specified headless-dns as communication method, I updated to the latest operator  and  I skipped (commenting in the configuration file) the local-ssd:
root@k8s-eu-1-master:~# cat arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
        communicationMethod: headless-dns
  #dbservers:
    #volumeClaimTemplate:
      #spec:
        #strategy:
          #type: Recreate
        #image: 1.2.33
        #storageClassName: my-local-ssd
        #accessModes:
        #- ReadWriteOnce
        #resources:
          #requests:
            #storage: 1Gi
        #volumeMode: Filesystem
        #communicationMethod: headless-dns

I then restarted the whole cluster :
root@k8s-eu-1-master:~# kubectl apply -f arangoDeployment.yaml
arangodeployment.database.arangodb.com/cluster configured

But the issue persists : Sending a heartbeat took longer than 2 seconds, this might be causing trouble with health checks. as you can see from the output of kubectl cluster-info dump command : https://drive.google.com/file/d/1Q1aLepbkyij_C0cQgjL9IozfIP7nvmn3/view?usp=drive_link
List of pods running:
root@k8s-eu-1-master:~# kubectl get pods
NAME                                             READY   STATUS    RESTARTS        AGE
arango-kube-arangodb-operator-844d65cdd5-88shw   1/1     Running   204 (40h ago)   85d
cluster-agnt-apt6zd1n-75b1c6                     1/1     Running   0               85d
cluster-agnt-mwchqsli-75b1c6                     1/1     Running   0               85d
cluster-agnt-uhbsq34m-75b1c6                     1/1     Running   0               85d
cluster-crdn-2avzgbi1-75b1c6                     1/1     Running   0               85d
cluster-crdn-ets3nigq-75b1c6                     1/1     Running   0               85d
cluster-crdn-n3r0x9by-75b1c6                     1/1     Running   0               21d
cluster-prmr-07wqascu-75b1c6                     1/1     Running   0               85d
cluster-prmr-kenmxcns-75b1c6                     1/1     Running   0               38d
cluster-prmr-nsgmx9lg-75b1c6                     1/1     Running   0               38d
local-storage-4hkdx                              1/1     Running   0               187d
local-storage-5xhmh                              1/1     Running   0               187d
local-storage-b654d                              1/1     Running   0               187d
local-storage-mvndq                              1/1     Running   0               187d
local-storage-rk85s                              1/1     Running   0               187d
root@k8s-eu-1-master:~#
Cluster Nodes:
root@k8s-eu-1-master:~# kubectl get nodes
NAME                STATUS   ROLES           AGE    VERSION
k8s-eu-1-master     Ready    control-plane   212d   v1.26.2
k8s-eu-1-worker-1   Ready    <none>          212d   v1.26.2
k8s-eu-1-worker-2   Ready    <none>          212d   v1.26.2
k8s-eu-1-worker-3   Ready    <none>          212d   v1.26.2
k8s-eu-1-worker-4   Ready    <none>          212d   v1.26.2
k8s-eu-1-worker-5   Ready    <none>          210d   v1.26.2

About the network delay between two nodes : how to actually detect and measure it?
I searched for how to do it, and I've found this: https://devops.stackexchange.com/questions/8260/kubernetes-metric-or-command-to-show-liveness-delay-time-app-startup-time but I'm not that sure it is what you meant
Anyway, I tried, and this is the output:
root@k8s-eu-1-master:~# kubectl get pod arango-kube-arangodb-operator-844d65cdd5-88shw -o 'go-template={{range .status.conditions}}
  {{if and (or (eq .type "Initialized") (eq .type "Ready")) (eq .status "True")}}
    {{printf "%s ==> %s\n" .type .lastTransitionTime }}
  {{end}}
{{end}}'

    Initialized ==> 2023-07-18T07:26:55Z

    Ready ==> 2023-10-10T00:07:06Z

Hello!
CommunicationMethod field is under spec, not agent PVC template.
spec:
  communicationMethod: ip

Best Regards,
Adam.

Hi @ajanikow !
modified communicationMethod :
root@k8s-eu-1-master:~# cat arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  #dbservers:
    #volumeClaimTemplate:
      #spec:
        #communicationMethod: ip
        #strategy:
          #type: Recreate
        #image: 1.2.33
        #storageClassName: my-local-ssd
        #accessModes:
        #- ReadWriteOnce
        #resources:
          #requests:
            #storage: 1Gi
        #volumeMode: Filesystem
        #communicationMethod: headless-dns
root@k8s-eu-1-master:~#

root@k8s-eu-1-master:~# kubectl apply -f arangoDeployment.yaml
arangodeployment.database.arangodb.com/cluster configured

But same error : ATTENTION: Sending a heartbeat took longer than 2 seconds, this might be causing trouble with health checks. Please contact ArangoDB Support :
https://drive.google.com/file/d/1mrSA8zPg0cUnUAEeyDwIiU8DfIqAJO16/view?usp=drive_link

@dothebart @nikita-vanyasin @ajanikow
I wiped out the kubernetes cluster and created a new one from scratch.
Now, without Arangodb, and, without any pods, everything seems working fine, as you can see from the output of kubectl cluster-info dump :  https://drive.google.com/file/d/1GMRgvT5rn99Qodm9VfRMCG1IfNRfZLTj/view?usp=sharing
I then proceeded to deploy ArangoDB cluster:  https://github.com/arangodb/kube-arangodb#community-edition
root@k8s-eu-1-master:~# kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/1.2.34/manifests/arango-crd.yaml
customresourcedefinition.apiextensions.k8s.io/arangojobs.apps.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangobackups.backup.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangobackuppolicies.backup.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangoclustersynchronizations.database.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangodeployments.database.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangomembers.database.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangotasks.database.arangodb.com created
customresourcedefinition.apiextensions.k8s.io/arangodeploymentreplications.replication.database.arangodb.com created

root@k8s-eu-1-master:~# kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/1.2.34/manifests/arango-deployment.yaml
serviceaccount/arango-deployment-operator created
clusterrole.rbac.authorization.k8s.io/arango-deployment-operator-rbac-crd created
clusterrole.rbac.authorization.k8s.io/arango-deployment-operator-rbac-deployment created
clusterrolebinding.rbac.authorization.k8s.io/arango-deployment-operator-rbac-crd created
clusterrolebinding.rbac.authorization.k8s.io/arango-deployment-operator-rbac-deployment created
role.rbac.authorization.k8s.io/arango-deployment-operator-rbac-default created
role.rbac.authorization.k8s.io/arango-deployment-operator-rbac-deployment created
rolebinding.rbac.authorization.k8s.io/arango-deployment-operator-rbac-default created
rolebinding.rbac.authorization.k8s.io/arango-deployment-operator-rbac-deployment created
service/arango-deployment-operator created
deployment.apps/arango-deployment-operator created

root@k8s-eu-1-master:~# kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/1.2.34/manifests/arango-storage.yaml
serviceaccount/arango-storage-operator created
customresourcedefinition.apiextensions.k8s.io/arangolocalstorages.storage.arangodb.com created
clusterrole.rbac.authorization.k8s.io/arango-storage-operator-rbac-crd created
clusterrole.rbac.authorization.k8s.io/arango-storage-operator-rbac-storage created
clusterrolebinding.rbac.authorization.k8s.io/arango-storage-operator-rbac-crd created
clusterrolebinding.rbac.authorization.k8s.io/arango-storage-operator-rbac-storage created
role.rbac.authorization.k8s.io/arango-storage-operator-rbac-storage created
rolebinding.rbac.authorization.k8s.io/arango-storage-operator-rbac-storage created
service/arango-storage-operator created
deployment.apps/arango-storage-operator created

root@k8s-eu-1-master:~# kubectl apply -f https://raw.githubusercontent.com/arangodb/kube-arangodb/1.2.34/manifests/arango-deployment-replication.yaml
serviceaccount/arango-deployment-replication-operator created
clusterrole.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-crd created
clusterrole.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-deployment-replication created
clusterrolebinding.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-crd created
clusterrolebinding.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-deployment-replication created
role.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-deployment-replication created
rolebinding.rbac.authorization.k8s.io/arango-deployment-replication-operator-rbac-deployment-replication created
service/arango-deployment-replication-operator created
deployment.apps/arango-deployment-replication-operator created

As starting point for ArangoDB Cluster Deployment, I used this simple arangoDeployment.yaml operator :
root@k8s-eu-1-master:~/ArangoDBDeployment# cat arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  #dbservers:
    #volumeClaimTemplate:
      #spec:
        #communicationMethod: ip
        #strategy:
          #type: Recreate
        #image: 1.2.33
        #storageClassName: my-local-ssd
        #accessModes:
        #- ReadWriteOnce
        #resources:
          #requests:
            #storage: 1Gi
        #volumeMode: Filesystem
        #communicationMethod: headless-dns

root@k8s-eu-1-master:~# kubectl apply -f ./ArangoDBDeployment/arangoDeployment.yaml
arangodeployment.database.arangodb.com/cluster created

root@k8s-eu-1-master:~# kubectl get pods
NAME                                                      READY   STATUS              RESTARTS   AGE
arango-deployment-operator-dd965c4d4-whsvt                0/1     ContainerCreating   0          112s
arango-deployment-replication-operator-864666fcbb-hf5hc   0/1     ContainerCreating   0          87s
arango-storage-operator-599f547648-hzbsd                  0/1     ContainerCreating   0          98s

root@k8s-eu-1-master:~# kubectl get services
NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
arango-deployment-operator               ClusterIP   <IP_ADDRESS>    <none>        8528/TCP,8628/TCP,8728/TCP   2m15s
arango-deployment-replication-operator   ClusterIP   <IP_ADDRESS>   <none>        8528/TCP,8628/TCP,8728/TCP   109s
arango-storage-operator                  ClusterIP   <IP_ADDRESS>   <none>        8528/TCP,8628/TCP,8728/TCP   2m
kubernetes                               ClusterIP   <IP_ADDRESS>       <none>        443/TCP                      156m
root@k8s-eu-1-master:~#

root@k8s-eu-1-master:~# kubectl get namespaces
NAME              STATUS   AGE
default           Active   157m
kube-flannel      Active   58m
kube-node-lease   Active   157m
kube-public       Active   157m
kube-system       Active   157m

root@k8s-eu-1-master:~# kubectl get persistentvolumes
No resources found

This Simple Arango Deployment (without DB Servers) seems working fine: https://drive.google.com/file/d/1mTVR7fkacgHifb5QBecXzLq_C7XZAlvf/view?usp=sharing
I then added DB Servers :
root@k8s-eu-1-master:~/ArangoDBDeployment# cat arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        strategy:
          type: Recreate
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
        communicationMethod: headless-dns

root@k8s-eu-1-master:~# kubectl apply -f ./ArangoDBDeployment/arangoDeployment.yaml
arangodeployment.database.arangodb.com/cluster configured

And this DB Servers, it seems working fine as well : https://drive.google.com/file/d/1dANK6OZ8iLa7eqUQSoU7vqYVqTvISM80/view?usp=sharing
Now I would like to add ArangoLocalStorage : https://docs.arangodb.com/stable/deploy/deployment/kubernetes/storage-resource/
And about this further step, I would like to ask you few questions :
Question 1 :  It is not clear to me if
Option A: I can add :
kind: "ArangoLocalStorage"
metadata:
  name: "example-arangodb-storage"
spec:
  storageClass:
    name: my-local-ssd
  localPath:
  - /mnt/big-ssd-disk

to the already present  'ArangoDeployment` operator :
root@k8s-eu-1-master:~/ArangoDBDeployment# ls -lah
total 12K
drwxr-xr-x 2 root root 4.0K Oct 17 16:31 .
drwx------ 7 root root 4.0K Oct 17 16:36 ..
-rw-r--r-- 1 root root  793 Oct 17 16:30 arangoDeployment.yaml

root@k8s-eu-1-master:~/ArangoDBDeployment# cat arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        communicationMethod: ip
        strategy:
          type: Recreate
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
        communicationMethod: headless-dns

or,
Option B : if I have to apply a new operator ArangoLocalStorage.yaml :
root@k8s-eu-1-master:~/ArangoDBDeployment# ls -lah
total 16K
drwxr-xr-x 2 root root 4.0K Oct 17 16:52 .
drwx------ 7 root root 4.0K Oct 17 16:36 ..
-rw-r--r-- 1 root root  793 Oct 17 16:30 arangoDeployment.yaml
-rw-r--r-- 1 root root  195 Oct 17 16:52 arangoLocalStorage.yaml

root@k8s-eu-1-master:~/ArangoDBDeployment# cat arangoLocalStorage.yaml
apiVersion: "storage.arangodb.com/v1alpha"
kind: "ArangoLocalStorage"
metadata:
  name: "example-arangodb-storage"
spec:
  storageClass:
    name: my-local-ssd
  localPath:
  - /mnt/big-ssd-disk

Question 2 :  is localPath compulsory ? And it has to be the same for all nodes of the cluster ?

@raphael10-collab hello!
I see that you already have my-local-ssd storage class specified for volumeClaimTemplate.
Just create ArangoLocalStorage with same storage class name - it will be linked to deployment automatically through storage class.
localPath is compulsory. One ArangoLocalStorage CR should be enough for a whole deployment.
Please refer to https://docs.arangodb.com/3.11/deploy/deployment/kubernetes/storage/#local-storage for more details.
Also, please fix your CR definition: communicationMethod is the field under spec, not under volumeClaimTemplate. In future versions of operator we will enable schema validation so such error swill be printed explicitly.
Best,
Nikita

Thanks @nikita-vanyasin
Moving the communicationMethod field one level, under the "general"  spec :
root@k8s-eu-1-master:~# cat ./ArangoDBDeployment/arangoDeployment.yaml
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  strategy:
    type: Recreate
  mode: Cluster
  communicationMethod: ip
  agents:
    volumeClaimTemplate:
      spec:
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        strategy:
          type: Recreate
        image: 1.2.33
        storageClassName: my-local-ssd
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
        communicationMethod: headless-dns

produced a bigger flow of information from kubectl cluster-info dump  : https://drive.google.com/file/d/1It-runcm8kbgL4RPnMR41i5z44fOyQji/view?usp=sharing
And I do not understand this output :  ( https://drive.google.com/file/d/1It-runcm8kbgL4RPnMR41i5z44fOyQji/view?usp=sharing )
==== START logs for container coredns of pod kube-system/coredns-5dd5756b68-g2bkc ====
Request log error: the server rejected our request for an unknown reason (get pods coredns-5dd5756b68-g2bkc)
==== END logs for container coredns of pod kube-system/coredns-5dd5756b68-g2bkc ====
==== START logs for container coredns of pod kube-system/coredns-5dd5756b68-gt7xt ====
Request log error: the server rejected our request for an unknown reason (get pods coredns-5dd5756b68-gt7xt)
==== END logs for container coredns of pod kube-system/coredns-5dd5756b68-gt7xt ====
==== START logs for container etcd of pod kube-system/etcd-k8s-eu-1-master ====
{"level":"warn","ts":"2023-10-17T13:02:21.177178Z","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}

@raphael10-collab the link is not working for me.
To debug coredns or other k8s issues please refer to https://kubernetes.io/docs/tasks/debug/debug-application/ and https://kubernetes.io/docs/tasks/debug/debug-cluster/

Thank you  @nikita-vanyasin
Debugging references come right at help, because suddenly the Arango Pods that were till now working fine :
root@k8s-eu-1-master:~# kubectl get pods
NAME                                         READY   STATUS    RESTARTS      AGE
arango-deployment-operator-dd965c4d4-xmgcx   0/1     Running   3 (42s ago)   3m53s
arango-storage-operator-599f547648-x4kzv     0/1     Running   3 (42s ago)   3m43s

now they have both CrashLoopBackOff status, which is strange for me, because I didn't do anything in the meanwhile :
root@k8s-eu-1-master:~# kubectl get pods
NAME                                         READY   STATUS             RESTARTS        AGE
arango-deployment-operator-dd965c4d4-xmgcx   0/1     CrashLoopBackOff   23 (102s ago)   70m
arango-storage-operator-599f547648-x4kzv     0/1     CrashLoopBackOff   23 (82s ago)    70m

root@k8s-eu-1-master:~# kubectl describe pod arango-deployment-operator-dd965c4d4-xmgcx
Name:             arango-deployment-operator-dd965c4d4-xmgcx
Namespace:        default
Priority:         0
Service Account:  arango-deployment-operator
Node:             k8s-eu-1-worker-2/xx.xxx.xxx.xxx
Start Time:       Wed, 18 Oct 2023 16:56:14 +0200
Labels:           app.kubernetes.io/instance=deployment
                  app.kubernetes.io/managed-by=Tiller
                  app.kubernetes.io/name=kube-arangodb
                  helm.sh/chart=kube-arangodb-1.2.34
                  pod-template-hash=dd965c4d4
                  release=deployment
Annotations:      <none>
Status:           Running
IP:               <IP_ADDRESS>
IPs:
  IP:           <IP_ADDRESS>
Controlled By:  ReplicaSet/arango-deployment-operator-dd965c4d4
Containers:
  operator:
    Container ID:  containerd://fc9f570a28e101ca55f387b22d9664644eeda85cc956fe5776880bb1d3040630
    Image:         arangodb/kube-arangodb:1.2.34
    Image ID:      docker.io/arangodb/kube-arangodb@sha256:a25d031e87ba5b0f3038ce9f346553b69760a3a065fe608727cde188602b59e8
    Port:          8528/TCP
    Host Port:     0/TCP
    Args:
      --scope=legacy
      --operator.deployment
      --mode.single
      --chaos.allowed=false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 18 Oct 2023 18:04:16 +0200
      Finished:     Wed, 18 Oct 2023 18:05:15 +0200
    Ready:          False
    Restart Count:  23
    Liveness:       http-get https://:8528/health delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get https://:8528/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
      MY_POD_NAME:       arango-deployment-operator-dd965c4d4-xmgcx (v1:metadata.name)
      MY_POD_IP:          (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xp5h9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-xp5h9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 5s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 5s
Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Warning  Unhealthy  31m (x110 over 71m)   kubelet  Readiness probe failed: Get "https://<IP_ADDRESS>:8528/ready": dial tcp <IP_ADDRESS>:8528: connect: connection refused
  Warning  Unhealthy  11m (x62 over 71m)    kubelet  Liveness probe failed: Get "https://<IP_ADDRESS>:8528/health": dial tcp <IP_ADDRESS>:8528: connect: connection refused
  Warning  BackOff    113s (x223 over 65m)  kubelet  Back-off restarting failed container operator in pod arango-deployment-operator-dd965c4d4-xmgcx_default(d31bc38f-7b69-4cee-a4d8-26e0f2ab3aee)
root@k8s-eu-1-master:~#

root@k8s-eu-1-master:~# kubectl describe pod arango-storage-operator-599f547648-x4kzv
Name:             arango-storage-operator-599f547648-x4kzv
Namespace:        default
Priority:         0
Service Account:  arango-storage-operator
Node:             k8s-eu-1-worker-2/xx.xxx.xxx.xxx
Start Time:       Wed, 18 Oct 2023 16:56:25 +0200
Labels:           app.kubernetes.io/instance=storage
                  app.kubernetes.io/managed-by=Tiller
                  app.kubernetes.io/name=kube-arangodb
                  helm.sh/chart=kube-arangodb-1.2.34
                  pod-template-hash=599f547648
                  release=storage
Annotations:      <none>
Status:           Running
IP:               <IP_ADDRESS>
IPs:
  IP:           <IP_ADDRESS>
Controlled By:  ReplicaSet/arango-storage-operator-599f547648
Containers:
  operator:
    Container ID:  containerd://08946928494bbe24991938f08cf3b8a119caeba9dfa9c93b7f7ed580d92a9903
    Image:         arangodb/kube-arangodb:1.2.34
    Image ID:      docker.io/arangodb/kube-arangodb@sha256:a25d031e87ba5b0f3038ce9f346553b69760a3a065fe608727cde188602b59e8
    Port:          8528/TCP
    Host Port:     0/TCP
    Args:
      --scope=legacy
      --operator.storage
      --mode.single
      --chaos.allowed=false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 18 Oct 2023 18:04:36 +0200
      Finished:     Wed, 18 Oct 2023 18:05:35 +0200
    Ready:          False
    Restart Count:  23
    Liveness:       http-get https://:8528/health delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get https://:8528/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
      MY_POD_NAME:       arango-storage-operator-599f547648-x4kzv (v1:metadata.name)
      MY_POD_IP:          (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-96jjt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-96jjt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 5s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 5s
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  33m (x107 over 73m)    kubelet  Readiness probe failed: Get "https://<IP_ADDRESS>:8528/ready": dial tcp <IP_ADDRESS>:8528: connect: connection refused
  Normal   Pulling    13m (x21 over 73m)     kubelet  Pulling image "arangodb/kube-arangodb:1.2.34"
  Warning  BackOff    3m30s (x224 over 67m)  kubelet  Back-off restarting failed container operator in pod arango-storage-operator-599f547648-x4kzv_default(135cdc0f-c38b-4e33-a4d4-5bbbcdb8bb98)
root@k8s-eu-1-master:~#

Both Arango Pods are on the same Kubernetes Node :
root@k8s-eu-1-master:~# kubectl describe node k8s-eu-1-worker-2
Name:               k8s-eu-1-worker-2
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-eu-1-worker-2
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 17 Oct 2023 13:52:08 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k8s-eu-1-worker-2
  AcquireTime:     <unset>
  RenewTime:       Wed, 18 Oct 2023 18:12:05 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 18 Oct 2023 18:08:22 +0200   Tue, 17 Oct 2023 15:05:14 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 18 Oct 2023 18:08:22 +0200   Tue, 17 Oct 2023 15:05:14 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 18 Oct 2023 18:08:22 +0200   Tue, 17 Oct 2023 15:05:14 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 18 Oct 2023 18:08:22 +0200   Tue, 17 Oct 2023 15:15:07 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  xx.xxx.xxx.xxx
  Hostname:    k8s-eu-1-worker-2
Capacity:
  cpu:                10
  ephemeral-storage:<PHONE_NUMBER>Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             61714452Ki
  pods:               110
Allocatable:
  cpu:                10
  ephemeral-storage:<PHONE_NUMBER>566
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             61612052Ki
  pods:               110
System Info:
  Machine ID:                 2cab19cb27f3c80f005066a8652d0eff
  System UUID:                29cbe26c-1bd1-4544-a085-0475c3f5c3b0
  Boot ID:                    d5d7015e-3243-45ac-847b-31683f032469
  Kernel Version:             5.15.0-86-generic
  OS Image:                   Ubuntu 22.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.24
  Kubelet Version:            v1.28.2
  Kube-Proxy Version:         v1.28.2
Non-terminated Pods:          (3 in total)
  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---
  default                     arango-deployment-operator-dd965c4d4-xmgcx    0 (0%)        0 (0%)      0 (0%)           0 (0%)         75m
  default                     arango-storage-operator-599f547648-x4kzv      0 (0%)        0 (0%)      0 (0%)           0 (0%)         75m
  kube-system                 kube-proxy-fq6vz                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         28h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests  Limits
  --------           --------  ------
  cpu                0 (0%)    0 (0%)
  memory             0 (0%)    0 (0%)
  ephemeral-storage  0 (0%)    0 (0%)
  hugepages-1Gi      0 (0%)    0 (0%)
  hugepages-2Mi      0 (0%)    0 (0%)
Events:              <none>

And, based on the node's description, it does not seem to have any problems
So, I need to understand :

why suddenly both Arango Pods changed their state from running to CrashLoopBackOff
how to make them work again
how to prevent this occuring "out of the blue"

I'm going to dive into debugging phase

I had a look at the syslog of the node, just after the pod in this node went from Running to CrashLoopBackOff State
These are the last lines of syslog :
Oct 20 15:44:10 k8s-eu-1-worker-1 kubelet[599]: I1020 15:44:10.594513     599 scope.go:117] "RemoveContainer" containerID="3e618ac247c1392fd6a6d67fad93d187c0dfae4d2cfe77c6a8b244c831dd0852"
    Oct 20 15:44:10 k8s-eu-1-worker-1 kubelet[599]: E1020 15:44:10.594988     599 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=operator pod=arango-deployment-operator-5f4d66bd86-4pxkn_default(397bd5c4-2bfc-4ca3-bc7d-bd149932e4b8)\"" pod="default/arango-deployment-operator-5f4d66bd86-4pxkn" podUID="397bd5c4-2bfc-4ca3-bc7d-bd149932e4b8"
    Oct 20 15:44:21 k8s-eu-1-worker-1 kubelet[599]: I1020 15:44:21.594619     599 scope.go:117] "RemoveContainer" containerID="3e618ac247c1392fd6a6d67fad93d187c0dfae4d2cfe77c6a8b244c831dd0852"
    Oct 20 15:44:21 k8s-eu-1-worker-1 kubelet[599]: E1020 15:44:21.595036     599 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=operator pod=arango-deployment-operator-5f4d66bd86-4pxkn_default(397bd5c4-2bfc-4ca3-bc7d-bd149932e4b8)\"" pod="default/arango-deployment-operator-5f4d66bd86-4pxkn" podUID="397bd5c4-2bfc-4ca3-bc7d-bd149932e4b8"

@nikita-vanyasin Why it says "restarting failed container" ?

@nikita-vanyasin I described the pods issue in:
`StackOverFlow: https://stackoverflow.com/questions/77325355/pods-go-back-and-forth-between-state-running-and-state-crashloopbackoff ,
ServerFault: https://serverfault.com/questions/1146252/kubernetes-pods-go-back-and-forth-from-running-state-to-crashloopbackoff-state
ArangoDBCommunity-Slack : https://arangodb-community.slack.com/archives/C0XSAHQRJ/p1697705579786639

but, unfortunately, I didn't solve it.
So I opened a new issue in ArangoDB github repo: https://github.com/arangodb/arangodb/issues/20010


Problem with grid sorting

First thanks for this sortable component. It was just what I had needed.
But somehow I am having trouble getting the grid sorting to work properly. I am not sure what I am doing wrong, but I can't seem to get the stability you show in your example.
When I move an item to a new spot, it seems that only temporarily the other items move out of the way. But then they ignore the own dragged, which results in that it usually jumps to the end.
Any idea what I am doing wrong?
Here is how it looks:

Seems that as my SortableContainer was somewhere nested further down in the DOM, setting
useWindowAsScrollContainer=true fixed the issue.

I spend weeks on this undocumented feature, I finally found a working answer here: https://codesandbox.io/embed/0nnq0kv2l


EP-483: Checkout Amount Total Property

📲 What
Android QA: Checkout Amount Total Property
🤔 Why
The total amount of my checkout was $602 AUD. Web correctly converted that to 459.51, but Android is sending<PHONE_NUMBER>062881. I don’t know how this is being calculated on Android, but it should be the total pledge amount at checkout, converted to USD from the project currency.
Should also be a decimal, not a float.
🛠 How

Changed
fun CheckoutData.totalAmount(usdRate: Float) = this.totalAmount() * usdRate
to
fun CheckoutData.totalAmount(usdRate: Float) = this.amount() * usdRate

This is because this.amount() is already pledgeAmount + addOns + shippingAmount + bonus.

Rounded up to 2dp decimal

 put("amount", checkoutData.amount().round())
 put("amount_total_usd", checkoutData.totalAmount(project.staticUsdRate()).round())

Adjusted segment tests cases

👀 See

Before 🐛
After 🦋

📋 QA
Story 📖
https://kickstarter.atlassian.net/browse/EP-483

Oh, one small thing, if you could attach a screenshot of the segment output that shows the correct value for checkout_amount_total_usd, that would be great :)


Interceptor didn't work correctly.

Interceptor didn't update header immediately, update header after one request.
Ex:
//Interceptor.

axios.interceptors.request.use((config) => {
  let token = localStorage.getItem('token');

  //console is showing token but didn't add in request header
  console.log("token", token)

  if(token){
    axios.defaults.headers.common['Authorization'] = `Bearer ${token}`;
  }
  return config;

}, (error) => {
  return Promise.reject(error);

});

//Login Request.
return axios.post(`auth/local`, obj)
    .then(res => {
      localStorage.setItem('token', res.data.token);

      //get user data request.
      return axios.get(`users/me`)
  })
    .then(user => user.data)
    .catch((error) => {
      return error
    });

I am with the same problem, my interceptor not update headers, only if I refresh page

You should add the header to the config object. What you did is add them to the global axios object, so they will only be used on the next request.

axios.interceptors.request.use((config) => {
let token = localStorage.getItem('token');
//console is showing token but didn't add in request header
console.log("token", token)
if(token){
return {
...config,
headers: {
...config.headers,
'Authorization': Bearer ${token}
}
}
}
}, (error) => {
return Promise.reject(error);
});

I am having the same issue. We have multiple axios services so we made a redux middleware function to take in the service, get the token out of the action or the state depending on whether they simply log in or check the "Remember Me" box as they log in, and to set the headers on the axios client accordingly.

const {types: {PARSED_TOKEN, LOGIN}} = authDux

export default service => ({getState}) => next => action => {
    if ([PARSED_TOKEN, LOGIN].includes(action.type)) {
        service.interceptors.request.use(config => ({
            ...config,
            headers: {
                ...config.headers,
                Authorization: `Bearer ${
                    path(['user', 'token', 'access_token'], action) ||
                    path(['auth', 'parsed_token'], getState())
                }`
            }
        }))
    }
    next(action)
}

The behavior is inconsistent for each subsequent axios call. On some pages that use the very same client, I get 401s until I refresh the page.


Disambiguate "Priority Queue" instructions

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

For future reference, should I have combined #38803, #38804, and this one into a single PR?

I have blocked this for now. See the pinned code freeze issue

Limitations removed on Coding Interview Prep Challenges.

For future reference, should I have combined #38803, #38804, and this one into a single PR?

There's no hard and fast rule for PRs, but each commit should be 'atomic' in the sense that it should be reasonably small and easy to understand.  In this case it seems better to have those separate, because one changes style, the other meaning.  It would be easy to miss the typo fix in the middle of all the changes to indentation.
As for this PR, I see that there's a reference to front (i.e. peek, I assume), but that does not appear in the tests or solution.  Could you remove references to it or add a test?  Either is fine.

@ojeytonwilliams Added front() to the solution and a couple of tests along with it! Here's the prettified version of the second new test:
assert(
  (function () {
    var test = new PriorityQueue();
    test.enqueue(["David Brown", 2]);
    var front1 = text.front();
    test.enqueue(["Jon Snow", 1]);
    var front2 = test.front();
    test.dequeue();
    test.enqueue(["A", 3]);
    var front3 = test.front();
    test.enqueue(["B", 3]);
    test.enqueue(["C", 3]);
    test.dequeue();
    var front4 = test.front();
    return (
      front1 === "David Brown" &&
      front2 === "Jon Snow" &&
      front3 === "David Brown" &&
      front4 === "A"
    );
  })()
);

Whoops, thanks for catching that, @RandellDawson! Tests passing now 👌🏼 Thanks for your help!

LGTM  @moT01 @ojeytonwilliams Merge if you agree.


Durability

I am able to launch Minecraft with it open but it doesn't show the durability for the tool on the hot bar. I don't know how to fix this.

Oh, that's a bit strange. Let's try the easy option first: Does it show up if you press B in-game? That's the default keybind to toggle the number on and off.
If that doesn't fix it, I'd like to know a bit more:

What version of the mod and Minecraft are you using?
What other mods, if any, are you using? And if you're using Mod Menu, does Show Durability show up there?
Can you send a Minecraft log file? Doesn't have to be anything specific; just a log file from a session where you start the game and load into a world should be fine.


How To make the list items Clickable

I need to start a new activity by clicking on one of the list items.Please provide any code regarding this.It would be a lot of helpful to me

simply just add  ItemClickListener  then Get position and start your new activity
This Link May Be Usefull : http://paste.ubuntu.com/14128748/

@mamakar333  go through this adapter. The MainActivity uses this to detect clicks.
https://github.com/Suleiman19/Android-Material-Design-for-pre-Lollipop/blob/master/MaterialSample/app/src/main/java/com/suleiman/material/adapter/SimpleRecyclerAdapter.java


Add support for a DeviceAttestationDelegate

With the DeviceAttestationDelegate, device attestation failure is simply ignored. Commissioning non-test Matter devices will now work.
Fix for issues: #83 #86

@pierredelisle
Thank you for PR.
But the problem is not fixed in my environment.
Does not work setDeviceAttestationDelegate.
I check the break point.But it's not stopped.

I attached log.
log.txt
Thank you for confirming.

@TadashiSatoo Please provide complete logs so we can investigate.

@pierredelisle
Thank you for reply.
Sorry...The information I provided was not completed.
I provide debug information.
Could you confirm it.

GHSAFM version

1.3.0-defult
commit 0eb3c6d2c126c020619d3fe40026c9fa52e1c80c

Which device you used to test, and the release version of its firmware.

Google Pixel5 :  Android13(latest firmware)
nRF52840 DK bord.(Nordic)

Google Play Services version

com.google.android.gms.optional_home [230617065] [23.06.17 (100400-0)] [Download:00000141/dl-Home.optional_230617100400.apk] [download:/Home.optional/230617100400:Home.optional:230617100400]

Phone log
logs.txt

Device log
device_logs.txt


[webview_flutter] support onReceivedHttpAuthRequest for Basic Auth

My webview needs a proxy to access it. My proxy has basic authen.
I tried using InAppWebView with the following config and it works ok:
 onReceivedHttpAuthRequest: (controller, challenge) async {
        return HttpAuthResponse(
          username: "xxxx",
          password: "xxxx",
          action: HttpAuthResponseAction.PROCEED,
          permanentPersistence: true,
        );
      }

I'd be grateful if the team adds this feature.

Hello @letrungdo. If you're using Basic Auth, can't you just change the url to include the username and password like<EMAIL_ADDRESS>

@exaby73 You're right, I tried but it doesn't work.

Could you provide a complete and minimal, reproducible example so that I may verify this issue?

@exaby73 Sorry I'm pretty busy right now so I can't send you an example repo. I have now switched to using a proxy without basic authen. Since webview_flutter does not yet have an implementation of onReceivedHttpAuthRequest, you can refer to this document:
https://developer.android.com/reference/android/webkit/WebViewClient#onReceivedHttpAuthRequest(android.webkit.WebView, android.webkit.HttpAuthHandler, java.lang.String, java.lang.String)

Okay thanks for the docs. I'll label this issue as a valid proposal

Duplicate of https://github.com/flutter/flutter/issues/83556