│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CLIENTID from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_SESSIONS from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_PNFS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RW_LOCK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NLM from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RPC from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_TIRPC from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_CB from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_THREAD from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_ACL from NIV_EVENT to NIV_INFO                                                               │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_STATE from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P from NIV_EVENT to NIV_INFO                                                                       │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P_DISPATCH from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL_UP from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DBUS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_MSK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_fds_limit :INODE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :State lock layer successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs4_recovery_init :CLIENT ID :INFO :Recovery Backend Init for longhorn                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_init :CLIENT ID :EVENT :Initialize recovery backend 'share-manager-jupyter-shared-volume'                                                                  │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_read_recov_clids :CLIENT ID :EVENT :Read clients from recovery backend share-manager-jupyter-shared-volume                                                       │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] read_clids :CLIENT ID :EVENT :response={"actions":{},"clients":[],"hostname":"share-manager-jupyter-shared-volume","id":"share-manager-jupyter-shared-volume","links":{"s │
│                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :EVENT :End grace for recovery backend 'share-manager-jupyter-shared-volume' version F6J8KV3Q                                         │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] http_call :CLIENT ID :EVENT :HTTP error: 500 (url=http://longhorn-recovery-backend:9600/v1/recoverybackend/share-manager-jupyter-shared-volume, payload={"version": "F6J8 │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :FATAL :HTTP call error: res=-1 ((null))                                                                                │
│ time="2023-01-02T16:52:12Z" level=error msg="NFS server exited with error" encrypted=false error="ganesha.nfsd failed with error: exit status 2, output: " volume=jupyter-shared-volume                                                                                │
│ W0102 16:52:12.397770       1 mount_helper_common.go:133] Warning: "/export/jupyter-shared-volume" is not a mountpoint, deleting                                                                                │
│ time="2023-01-02T16:52:12Z" level=debug msg="Device /dev/mapper/jupyter-shared-volume is not an active LUKS device" error="failed to run cryptsetup args: [status jupyter-shared-volume] output:  error: exit status 4"

Can you provide the latest support bundle? Thank you.
Jinserk Baik @.***>於 2023年1月3日 週二，上午12:52寫道：

I applied it and lauching pod to mount the volume, but I am watching
share-manager-jupyter-shared-volume pod is repeatedly containercreating
-> running -> completed with the following logs:
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FILEHANDLE from NIV_EVENT to NIV_INFO                                                               │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DISPATCH from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE_LRU from NIV_EVENT to NIV_INFO                                                          │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE from NIV_EVENT to NIV_INFO                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE_CACHE from NIV_EVENT to NIV_INFO                                                          │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DUPREQ from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_INIT from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MAIN from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_IDMAPPER from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_READDIR from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_LOCK from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CONFIG from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CLIENTID from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_SESSIONS from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_PNFS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RW_LOCK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NLM from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RPC from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_TIRPC from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_CB from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_THREAD from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_ACL from NIV_EVENT to NIV_INFO                                                               │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_STATE from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P from NIV_EVENT to NIV_INFO                                                                       │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P_DISPATCH from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL_UP from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DBUS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_MSK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_fds_limit :INODE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :State lock layer successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs4_recovery_init :CLIENT ID :INFO :Recovery Backend Init for longhorn                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_init :CLIENT ID :EVENT :Initialize recovery backend 'share-manager-jupyter-shared-volume'                                                                  │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_read_recov_clids :CLIENT ID :EVENT :Read clients from recovery backend share-manager-jupyter-shared-volume                                                       │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] read_clids :CLIENT ID :EVENT :response={"actions":{},"clients":[],"hostname":"share-manager-jupyter-shared-volume","id":"share-manager-jupyter-shared-volume","links":{"s │
│                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :EVENT :End grace for recovery backend 'share-manager-jupyter-shared-volume' version F6J8KV3Q                                         │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] http_call :CLIENT ID :EVENT :HTTP error: 500 (url=http://longhorn-recovery-backend:9600/v1/recoverybackend/share-manager-jupyter-shared-volume, payload={"version": "F6J8 │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :FATAL :HTTP call error: res=-1 ((null))                                                                                │
│ time="2023-01-02T16:52:12Z" level=error msg="NFS server exited with error" encrypted=false error="ganesha.nfsd failed with error: exit status 2, output: " volume=jupyter-shared-volume                                                                                │
│ W0102 16:52:12.397770       1 mount_helper_common.go:133] Warning: "/export/jupyter-shared-volume" is not a mountpoint, deleting                                                                                │
│ time="2023-01-02T16:52:12Z" level=debug msg="Device /dev/mapper/jupyter-shared-volume is not an active LUKS device" error="failed to run cryptsetup args: [status jupyter-shared-volume] output:  error: exit status 4"
—
Reply to this email directly, view it on GitHub
https://github.com/longhorn/longhorn/issues/5183#issuecomment-1369088636,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC7SNAMEUASVQWETKT6HQLDWQMBV3ANCNFSM6AAAAAATNYHBOU
.
You are receiving this because you were mentioned.Message ID:
@.***>

supportbundle_1b5d7a33-0e17-4e75-b3e3-b6ad7556e43c_2023-01-02T17-04-44Z.zip

Wow it looks working! It is amazing! Did you resolve the 4-yr long issue?

No, my colleague added the verification for some resources creation operations for a workaound. I just thought of the verification and added it to the share-manager's configmap creation flow. :) Look forward to your test results. Many thanks for your patience and help.
From the issue, I think we should verify all creations for all resource creation for avoiding any accident. cc @innobead

etcd is strong consistency so the inconsistency should be related to Kubernetes client design, https://github.com/longhorn/longhorn/issues/133#issuecomment-419229569

Here is the latest one: supportbundle_1b5d7a33-0e17-4e75-b3e3-b6ad7556e43c_2023-01-02T17-04-44Z.zip

Thank you. The error in create operation is fixed. The error now is from update operation. Checking how to fix it.

@

I applied it and lauching pod to mount the volume, but I am watching share-manager-jupyter-shared-volume pod is repeatedly containercreating -> running -> completed with the following logs:
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FILEHANDLE from NIV_EVENT to NIV_INFO                                                               │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DISPATCH from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE_LRU from NIV_EVENT to NIV_INFO                                                          │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE from NIV_EVENT to NIV_INFO                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE_CACHE from NIV_EVENT to NIV_INFO                                                          │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DUPREQ from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_INIT from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MAIN from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_IDMAPPER from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_READDIR from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_LOCK from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CONFIG from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CLIENTID from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_SESSIONS from NIV_EVENT to NIV_INFO                                                                 │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_PNFS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RW_LOCK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NLM from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RPC from NIV_EVENT to NIV_INFO                                                                      │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_TIRPC from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_CB from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_THREAD from NIV_EVENT to NIV_INFO                                                                   │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_ACL from NIV_EVENT to NIV_INFO                                                               │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_STATE from NIV_EVENT to NIV_INFO                                                                    │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P from NIV_EVENT to NIV_INFO                                                                       │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P_DISPATCH from NIV_EVENT to NIV_INFO                                                              │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL_UP from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DBUS from NIV_EVENT to NIV_INFO                                                                     │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_MSK from NIV_EVENT to NIV_INFO                                                                  │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_fds_limit :INODE LRU :EVENT :Setting the system-imposed limit on FDs to 1048576.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :State lock layer successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs4_recovery_init :CLIENT ID :INFO :Recovery Backend Init for longhorn                                                                                │
│ 02/01/2023 16:52:11 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_init :CLIENT ID :EVENT :Initialize recovery backend 'share-manager-jupyter-shared-volume'                                                                  │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_read_recov_clids :CLIENT ID :EVENT :Read clients from recovery backend share-manager-jupyter-shared-volume                                                       │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] read_clids :CLIENT ID :EVENT :response={"actions":{},"clients":[],"hostname":"share-manager-jupyter-shared-volume","id":"share-manager-jupyter-shared-volume","links":{"s │
│                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)                                                                                │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :EVENT :End grace for recovery backend 'share-manager-jupyter-shared-volume' version F6J8KV3Q                                         │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] http_call :CLIENT ID :EVENT :HTTP error: 500 (url=http://longhorn-recovery-backend:9600/v1/recoverybackend/share-manager-jupyter-shared-volume, payload={"version": "F6J8 │
│ 02/01/2023 16:52:12 : epoch 63b30bbb : share-manager-jupyter-shared-volume : nfs-ganesha-30[main] longhorn_recov_end_grace :CLIENT ID :FATAL :HTTP call error: res=-1 ((null))                                                                                │
│ time="2023-01-02T16:52:12Z" level=error msg="NFS server exited with error" encrypted=false error="ganesha.nfsd failed with error: exit status 2, output: " volume=jupyter-shared-volume                                                                                │
│ W0102 16:52:12.397770       1 mount_helper_common.go:133] Warning: "/export/jupyter-shared-volume" is not a mountpoint, deleting                                                                                │
│ time="2023-01-02T16:52:12Z" level=debug msg="Device /dev/mapper/jupyter-shared-volume is not an active LUKS device" error="failed to run cryptsetup args: [status jupyter-shared-volume] output:  error: exit status 4"

This is the log from longhorn-manager:
time="2023-01-02T16:53:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856035\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:37Z"
time="2023-01-02T16:53:37Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:53:38Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:38Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:38Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:38Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856080\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:45Z"
time="2023-01-02T16:53:45Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:53:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856131\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:45Z"
time="2023-01-02T16:53:45Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:45Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:45Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856141\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:51Z"
time="2023-01-02T16:53:51Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:53:51Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:51Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856185\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:51Z"
time="2023-01-02T16:53:51Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856220\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:56Z"
time="2023-01-02T16:53:56Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:53:57Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:57Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:53:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856240\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:53:57Z"
time="2023-01-02T16:53:57Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856255\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:54:02Z"
time="2023-01-02T16:54:02Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:54:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856359\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:54:03Z"
time="2023-01-02T16:54:03Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:03Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:03Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856378\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:54:09Z"
time="2023-01-02T16:54:09Z" level=debug msg="starting share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=error volume=jupyter-shared-volume
time="2023-01-02T16:54:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"jupyter-shared-volume\", UID:\"c89b0857-7727-4f89-aa51-d9164a186c27\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"856483\", FieldPath:\"\"}): type: 'Normal' reason: 'Remount' Volume jupyter-shared-volume requested remount at 2023-01-02T16:54:09Z"
time="2023-01-02T16:54:09Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=kyu-gpu2 owner=kyu-gpu2 pod=share-manager-jupyter-shared-volume shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:09Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume
time="2023-01-02T16:54:09Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=kyu-gpu2 owner=kyu-gpu2 shareManager=jupyter-shared-volume state=starting volume=jupyter-shared-volume

Hello @jinserk
I added the retry of confimap update on conflict in image derekbit/longhorn-manager:v1.4.0-update. Could you please try again? Thank you.
@longhorn/qa @chriscchien  I cannot reproduce the issue in my env. Can you help check if you can reproduce it?

@chriscchien lets use 1.4.0 to reproduce it instead of @derekbit patch images.

The update and create operations need retries for avoiding the issue, because https://github.com/longhorn/longhorn/issues/133#issuecomment-419229569.
cc @innobead @shuo-wu

One quick question. Does not the fix work?

I applied it and lauching pod to mount the volume, but I am watching share-manager-jupyter-shared-volume pod is repeatedly containercreating -> running -> completed with the following logs:

Hello @jinserk I added the retry of confimap update on conflict in image derekbit/longhorn-manager:v1.4.0-update. Could you please try again? Thank you.
@longhorn/qa @chriscchien I cannot reproduce the issue in my env. Can you help check if you can reproduce it using v1.4.0 images?

Hi @derekbit @longhorn/qa
I had tried to reproduce the issue on AWS yesterday, but I couldn't reproduce it either.

One quick question. Does not the fix work?

@shuo-wu
Yeah, the retris fixes the create operation. Added it to update operation as well. Waiting @jinserk's update.
BTW, do you know how to trigger it?

I see, the functions in datastore/kubernetes do not contain verifyCreate or verifyUpdate
No... I just knew the comments Sheng left in the ticket...

@derekbit
Sorry for late reply. I tested your latest update image derekbit/longhorn-manager:v1.4.0-update, with the https://github.com/longhorn/longhorn/issues/5183#issuecomment-1368465846 and it looks working as charm!
By the way, my JupyterHub singleuser pod repeatedly failed to be spawned, that doesn't look caused by this issue:
Events:
  Type     Reason                  Age    From                       Message
  ----     ------                  ----   ----                       -------
  Normal   Scheduled               10m    jupyterhub-user-scheduler  Successfully assigned jupyterhub/jupyter-jinserk-2ebaik to kyu-gpu1
  Normal   SuccessfulAttachVolume  10m    attachdetach-controller    AttachVolume.Attach succeeded for volume "jinserk-baik-volume"
  Normal   SuccessfulAttachVolume  9m59s  attachdetach-controller    AttachVolume.Attach succeeded for volume "jupyter-shared-volume"
  Warning  FailedMount             8m11s  kubelet                    Unable to attach or mount volumes: unmounted volumes=[volume-jinserk-2ebaik], unattached volumes=[volume-jinserk-2ebaik shm-volume shared-volume docker-graph-storage]: timed out waiting for the condition
  Warning  FailedMount             5m55s  kubelet                    Unable to attach or mount volumes: unmounted volumes=[volume-jinserk-2ebaik], unattached volumes=[docker-graph-storage volume-jinserk-2ebaik shm-volume shared-volume]: timed out waiting for the condition
  Warning  FailedMount             3m39s  kubelet                    Unable to attach or mount volumes: unmounted volumes=[volume-jinserk-2ebaik], unattached volumes=[shared-volume docker-graph-storage volume-jinserk-2ebaik shm-volume]: timed out waiting for the condition

I have suffered from the "failedmount" warning for a long time, so I and my collegues just try to spawn and if works we don't terminate the pod. I guess this is related to the virtual device /dev/sd? when a longhorn volume mounted, and it's not cleared by pod terminating. Anyway, this is not related to this issue, so I don't want to discuss it here.
Here is the support bundle for the above test:
supportbundle_1b5d7a33-0e17-4e75-b3e3-b6ad7556e43c_2023-01-03T15-03-25Z.zip

Suspect i have the same issue. Can see sharemanager pod restarting regularly. Using RWX with 3 pods

Suspect i have the same issue. Can see sharemanager pod restarting regularly. Using RWX with 3 pods

@tbertenshaw
If available, you can replace the longhorn-manager with derekbit/longhorn-manager:v1.4.0-update and check if it can solve it. This image is v1.4.0 with the patch.

@derekbit Sorry for late reply. I tested your latest update image derekbit/longhorn-manager:v1.4.0-update, with the #5183 (comment) and it looks working as charm!
By the way, my JupyterHub singleuser pod repeatedly failed to be spawned, that doesn't look caused by this issue:

Hello @jinserk,
Thank you for the update.
Can you provide the volume name and the timing for further investigation?
Yeah, from the support bundle, it is not releated to the RWX volume's recovery backend. The recovery backend works as expected for now.

@jinserk Can you provide the kubelet log as well? Thank you.

@derekbit I'm not sure these are the logs you want: I used the cmd to get this:
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu1.log    # on kyu-gpu1 node
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu2.log    # on kyu-gpu2 node

kyu-gpu1.log
kyu-gpu2.log
If you're digging the long-time FailedMounting issue, it will be super helpful for me!! Thank you so much again.

derekbit/longhorn-manager:v1.4.0-update

applied this to the 3 places of the longhornmanager daemonset. I'm still seeing the same restart of the sharemanager pod every 2 mins or so.

@timbo
Can you provide the support bundle?
timbo @.***>於 2023年1月4日 週三，上午3:02寫道：

derekbit/longhorn-manager:v1.4.0-update
applied this to the 3 places of the longhornmanager daemonset. I'm still
seeing the same restart of the sharemanager pod every 2 mins or so.
—
Reply to this email directly, view it on GitHub
https://github.com/longhorn/longhorn/issues/5183#issuecomment-1370122182,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC7SNALXCICGF4ZYCBDHHW3WQRZTNANCNFSM6AAAAAATNYHBOU
.
You are receiving this because you were mentioned.Message ID:
@.***>

@timbo

applied this to the 3 places of the longhornmanager daemonset. I'm still
seeing the same restart of the sharemanager pod every 2 mins or so.

Sorry, I didn't explain clearly. Should replace the longhorn-recovery-backend deployment's image to derekbit/longhorn-manager:v1.4.0-update

@derekbit I'm not sure these are the logs you want: I used the cmds to get them:
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu1.log    # on kyu-gpu1 node
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu2.log    # on kyu-gpu2 node

kyu-gpu1.log kyu-gpu2.log
If you're digging the long-time FailedMount issue, it will be super helpful for me!! Thank you so much again. I found #3207 which looks like related to this.

@tbertenshaw Can you try the workaround mentioned in https://github.com/longhorn/longhorn/issues/3207?
I will continue digging in the new issue. BTW, does the issue happen in v1.3.x as well?

derekbit/longhorn-manager:v1.4.0-update

So update the longhorn-manager daemonset and the longhorn-recovery-backend deployment to use derekbit/longhorn-manager:v1.4.0-update? Just done that now and can see that the share-manager-pvc pod is still restarting every few minutes

derekbit/longhorn-manager:v1.4.0-update

So update the longhorn-manager daemonset and the longhorn-recovery-backend deployment to use derekbit/longhorn-manager:v1.4.0-update? Just done that now and can see that the share-manager-pvc pod is still restarting every few minutes

Can you provide us the support bundle? A newly created volume also encounters the same issue?

derekbit/longhorn-manager:v1.4.0-update

So update the longhorn-manager daemonset and the longhorn-recovery-backend deployment to use derekbit/longhorn-manager:v1.4.0-update? Just done that now and can see that the share-manager-pvc pod is still restarting every few minutes

@tbertenshaw Can you provide us the support bundle? A newly created volume also encounters the same issue?

share-manager-pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191.log

AKS 1.25.2
longhorn version 1.4.0 (previously had 1.3.2 only spotted this issue since upgrade)
3 nodes (2core x 8Gb) essentially only running longhorn and moodle with 3 moodle pods RWX.

Any more detail needed?

@tbertenshaw Thanks for the info, but we need the support bundle with more info. You can follow the instruction at https://longhorn.io/docs/1.4.0/advanced-resources/support-bundle/ to generate it.

@tbertenshaw
The share-manager pod looks normal. Can you provide the support bundle for further information? You can follow the instruction at https://longhorn.io/docs/1.4.0/advanced-resources/support-bundle/ to generate it and send to<EMAIL_ADDRESS>or attach here.

Incidentally just trying to run the support bundle but suspect it will fail as it is trying to schedule a support-bundle-agent on a virtual node which is cordoned

@tbertenshaw
I see. Can you provide the recovery-backend pods and longhorn-manager pods 'logs?

ok removed virtual node as not using it :)
@innobead @derekbit managed to create the support bundle. Is it Safe to post here? i.e. doesn't grab secret things?

Also would it make more sense if i scaled my pods up so the restarting is still occurring then get a support bundle from that profile? Or will a support bundle  be ok? i.e. the share manager has now be running for 103 minutes successfully since scaling moodle down to 1 pod.

Also would it make more sense if i scaled my pods up so the restarting is still occurring then get a support bundle from that profile?

It makes sense. And also provide the volume name. Thank you.
The steps to reproduce are appreciated as well.

supportbundle_cf947ccc-9d7e-490f-8df5-f22db6f413f7_2023-01-04T10-35-26Z.zip
@derekbit here is the current support bundle. Not in a position to scale up and break students sessions. Will do later. When you ask for steps to reproduce. I havent tested a new test deployment this is an existing helm deployment so not really gonna have the steps, unless you want some other detail?

@derekbit I'm not sure these are the logs you want: I used the cmds to get them:
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu1.log    # on kyu-gpu1 node
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu2.log    # on kyu-gpu2 node

kyu-gpu1.log kyu-gpu2.log
If you're digging the long-time FailedMount issue, it will be super helpful for me!! Thank you so much again. I found #3207 which looks like related to this.

@jinserk Can you try the workaround mentioned in #3207? I will continue digging in the new issue. BTW, does the issue happen in v1.3.x as well?

Yes I used longhorn from v1.3.1, and experienced this very frequently, especially everytime I reinstall the k3s cluster.

@derekbit I'm not sure these are the logs you want: I used the cmds to get them:
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu1.log    # on kyu-gpu1 node
$ sudo journalctl -u k3s --since "2023-01-03" > kyu-gpu2.log    # on kyu-gpu2 node

kyu-gpu1.log kyu-gpu2.log
If you're digging the long-time FailedMount issue, it will be super helpful for me!! Thank you so much again. I found #3207 which looks like related to this.

@jinserk Can you try the workaround mentioned in #3207? I will continue digging in the new issue. BTW, does the issue happen in v1.3.x as well?

Yes I used longhorn from v1.3.1, and experienced this very frequently, especially everytime I reinstall the k3s cluster.
I already applied the #3207 workaround, and found that it works in my case too. Every volumes in my use has 2 replicas on both nodes, which hit the case in #3207

@jinserk
Got it. Can you help open a ticket? Then, we can check it in that ticket. Thank you.

@innobead @derekbit scaled workload to 3 pods and this is the support bundle after an hour of this config.
supportbundle_cf947ccc-9d7e-490f-8df5-f22db6f413f7_2023-01-04T18-53-12Z.zip

@tbertenshaw
I just checked the support bundle. There are two rwx volumes,  pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 and test-volume. pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 with 3 replicas works correctly and is running from 2023-01-04T11:32:08.529676404Z . It is used by the 3 application pods.
But I saw a lot of warning messages before 2023-01-04T11:32:08.529676404Z.
2023-01-04T11:32:05.350511341Z time="2023-01-04T11:32:05Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 pod=share-manager-pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.350541588Z time="2023-01-04T11:32:05Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.360134624Z time="2023-01-04T11:32:05Z" level=debug msg="Requeue share manager due to conflict" controller=longhorn-share-manager error="Operation cannot be fulfilled on sharemanagers.longhorn.io \"pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191\": the object has been modified; please apply your changes to the latest version and try again" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=error volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430476880Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430498000Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430572169Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430624078Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430698826Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430745183Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430829984Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430885085Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430932985Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431016262Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431024306Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191

@tbertenshaw
I just checked the support bundle. There are two rwx volumes,  pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 and test-volume. pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 with 3 replicas works correctly and is running from 2023-01-04T11:32:08.529676404Z . It is used by the 3 application pods.
But I saw a lot of warning messages before 2023-01-04T11:32:08.529676404Z.
2023-01-04T11:32:05.350511341Z time="2023-01-04T11:32:05Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 pod=share-manager-pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.350541588Z time="2023-01-04T11:32:05Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.360134624Z time="2023-01-04T11:32:05Z" level=debug msg="Requeue share manager due to conflict" controller=longhorn-share-manager error="Operation cannot be fulfilled on sharemanagers.longhorn.io \"pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191\": the object has been modified; please apply your changes to the latest version and try again" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=error volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430476880Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430498000Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430572169Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430624078Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430698826Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430745183Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430829984Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430885085Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430932985Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431016262Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431024306Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191

The pod.spec.nodeName looks somehow are updated.
@joshimoo Do you have any idea for the error?

@innobead @derekbit scaled workload to 3 pods and this is the support bundle after an hour of this config.
supportbundle_cf947ccc-9d7e-490f-8df5-f22db6f413f7_2023-01-04T18-53-12Z.zip

@tbertenshaw
Can you provide the kubelet log? There are probably some clues for the error cannot check IsNodeDownOrDeleted() when syncShareManagerPod.

@tbertenshaw I just checked the support bundle. There are two rwx volumes, pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 and test-volume. pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 with 3 replicas works correctly and is running from 2023-01-04T11:32:08.529676404Z . It is used by the 3 application pods.
But I saw a lot of warning messages before 2023-01-04T11:32:08.529676404Z.
2023-01-04T11:32:05.350511341Z time="2023-01-04T11:32:05Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 pod=share-manager-pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.350541588Z time="2023-01-04T11:32:05Z" level=warning msg="cannot check IsNodeDownOrDeleted() when syncShareManagerPod" controller=longhorn-share-manager error="no node name provided to check node down or deleted" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.360134624Z time="2023-01-04T11:32:05Z" level=debug msg="Requeue share manager due to conflict" controller=longhorn-share-manager error="Operation cannot be fulfilled on sharemanagers.longhorn.io \"pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191\": the object has been modified; please apply your changes to the latest version and try again" node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=error volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430476880Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430498000Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430572169Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430624078Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430698826Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430745183Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 replica candidate for auto-balance" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceOption=least-effort shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430829984Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, volume replicas are running on different nodes" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=node shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430885085Z time="2023-01-04T11:32:05Z" level=debug msg="Found 3 use zones [northeurope-1 northeurope-3 northeurope-2]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.430932985Z time="2023-01-04T11:32:05Z" level=debug msg="Found 2 use nodes [aks-nodepoolres-14028323-vmss000002 aks-nodepoolres-14028323-vmss000001]" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431016262Z time="2023-01-04T11:32:05Z" level=debug msg="Balanced, 3 volume replicas are running on different zones" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 replicaAutoBalanceType=zone shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
2023-01-04T11:32:05.431024306Z time="2023-01-04T11:32:05Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwx controller=longhorn-volume frontend=blockdev migratable=false node=aks-nodepoolres-14028323-vmss000000 owner=aks-nodepoolres-14028323-vmss000000 shareEndpoint= shareState=error state=attached volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191

The pod.spec.nodeName looks somehow are not updated. @joshimoo Do you have any idea for the error cannot check IsNodeDownOrDeleted() when syncShareManagerPod?

The volume pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 is the one that is not running correctly now since the upgrade. This is the one where the share manage pod is restarted regularly

journal_kubelet1.zip
i grabbed the last 600 lines from kubelet logs on two nodes.

d.spec.nodeNamelooks somehow are not updated. @joshimoo Do you have any idea for the errorcannot check IsNodeDownOrDeleted() when syncShareManagerPod`?

The volume pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 is the one that is not running correctly now since the upgrade. This is the one where the share manage pod is restarted regularly.

@tbertenshaw
Sorry, my bad. Checked the wrong node before.
Yes, as you mentioned. The share-manager pod is created and deleted repeatedly. The share-manager controller complains cannot check IsNodeDownOrDeleted() when syncShareManagerPod because of  the errorno node name provided to check node down or deleted. I also checked the log, and the share-manager pod's spec.nodeName is actually set to aks-nodepoolres-14028323-vmss000001 which is ready.
I suspect this is probably caused by that immediately checking pod.spec.nodeName after creating a share-manager pod.
I'm going to work on a patch, and I would appreciate it if you help check it.

@

The volume pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 is the one that is not running correctly now since the upgrade. This is the one where the share manage pod is restarted regularly.

@tbertenshaw Sorry, my bad. Checked the wrong node before.
Yes, as you mentioned. The share-manager pod is created and deleted repeatedly. The share-manager controller complains cannot check IsNodeDownOrDeleted() when syncShareManagerPod because of the errorno node name provided to check node down or deleted. I also checked the log, and the share-manager pod's spec.nodeName is actually set to aks-nodepoolres-14028323-vmss000001 which is ready.
I suspect this is probably caused by that immediately checking pod.spec.nodeName after creating a share-manager pod.
I'm going to work on a patch, and I would appreciate it if you help check it.

@tbertenshaw
Can you replace the longhorn-manager daemonset's image with the customized one derekbit/longhorn-manager:v1.4.0-fix-share-pod-recreate? Thank you.

I suspect this is probably caused by that immediately checking pod.spec.nodeName after creating a share-manager pod.
I'm going to work on a patch, and I would appreciate it if you help check it.

@tbertenshaw Can you replace the longhorn-manager daemonset's image with the customized one derekbit/longhorn-manager:v1.4.0-fix-share-pod-recreate? Thank you.

just applied and scaled up to 3 pods again. oops can see share manager has restarted.  Will see if i can get some kubelet logs again after enough time for sufficient logs

st applied and scaled up to 3 pods again. oops can see share manager has restarted. Will see if i can get some kubelet logs again after enough time for sufficient logs

Support bundle is needed as well. Thanks.

st applied and scaled up to 3 pods again. oops can see share manager has restarted. Will see if i can get some kubelet logs again after enough time for sufficient logs

Support bundle is needed as well. Thanks.

journal_kubelet.zip
supportbundle_cf947ccc-9d7e-490f-8df5-f22db6f413f7_2023-01-05T14-19-09Z.zip
@derekbit Journals from all 3 nodes with last 600 entries from kubelet

@tbertenshaw
Can you please try the image derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename? Thanks.

just deployed, rescaled back to 3 pods. Can see after a few minutes the share manager has restarted. Will check back later and get some logs. Not able to at the moment.

@jinserk
I just wrapped up the patch for RWX volume. If you are available, can you try the image derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename? Many thanks.

longhorn-manager-x5p6j.log
as well as the share manager restarting, i can see that one of the longhorn-manager pods has restarted 123 times
longhorn-manager-x5p6j.log

@derekbit longhorn-manager-x5p6j.log as well as the share manager restarting, i can see that one of the longhorn-manager pods has restarted 123 times longhorn-manager-x5p6j.log

time="2023-01-06T05:50:06Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=aks-nodepoolres-14028323-vmss000001 owner=aks-nodepoolres-14028323-vmss000001 pod=share-manager-pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 shareManager=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191 state=starting volume=pvc-f5952e3a-d202-4e8f-9fcf-277d70bec191
E0106 05:50:06.561221       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 777 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1d7e300, 0x3567c50})
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0017e35c0})
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x1d7e300, 0x3567c50})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).syncShareManagerPod(0xc000149300, 0xc00224a2c0)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:605 +0x246
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).syncShareManager(0xc000149300, {0xc0018a7580, 0x38})
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:330 +0x92b
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).processNextWorkItem(0xc000149300)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:232 +0xdb
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).worker(...)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:222
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f32dede4740)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x234eb00, 0xc002b23140}, 0x1, 0xc0000d6180)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).Run
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:216 +0x20b
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1d8 pc=0x193a066]

goroutine 777 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0017e35c0})
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x1d7e300, 0x3567c50})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).syncShareManagerPod(0xc000149300, 0xc00224a2c0)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:605 +0x246
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).syncShareManager(0xc000149300, {0xc0018a7580, 0x38})
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:330 +0x92b
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).processNextWorkItem(0xc000149300)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:232 +0xdb
github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).worker(...)
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:222
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f32dede4740)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x234eb00, 0xc002b23140}, 0x1, 0xc0000d6180)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
	/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/longhorn/longhorn-manager/controller.(*ShareManagerController).Run
	/go/src/github.com/longhorn/longhorn-manager/controller/share_manager_controller.go:216 +0x20b

@innobead  From the patched image.
@tbertenshaw You can fall back to longhornio/longhorn-manager:v1.4.0. Looks the patched one has some issues.

@tbertenshaw
Sorry... my bad. The variable scope leads to the issue. I will build a new one for testing.

@tbertenshaw
Sorry... my bad. The variable scope leads to the issue. I will build a new one for testing.
You can please use the image derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename-v2 instead. Thank you.

Thanks, @tbertenshaw to cooperate with us for testing :+1:

@innobead @derekbit Just updated the image. (fingerscrossed) seems sofar to be positive. No restarted share-manager or longhorn-manager pods. Will keep an eye for next couple hours and update on status.

@innobead @derekbit Just updated the image. (fingerscrossed) seems sofar to be positive. No restarted share-manager or longhorn-manager pods. Will keep an eye for next couple hours and update on status.

@tbertenshaw Thanks for the good news. Appreciated your help! 👍

@derekbit spoke too soon. :( can see the longhorn-manager pod/s that are on the same vm as the volume has restarted twice. and also the share-manager has also. here are the logs of of one of the restarted longhorn-manager pods.
longhorn-manager-fzq9t.log
do you want any more logs?

Added log for 3rd restarted pod as it took some time to start and has warnings
longhorn-manager-fzq9t.log

@derekbit spoke too soon. :( can see the longhorn-manager pod/s that are on the same vm as the volume has restarted twice. and also the share-manager has also. here are the logs of of one of the restarted longhorn-manager pods.
longhorn-manager-fzq9t.log
do you want any more logs?

You can revert to longhornio/longhorn-manager:v1.4.0. Let me check it. Have no iead for the panic for now. :(

@tbertenshaw
I added more logs for tracing the error. Can you help try the image derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename-v3 again? Many thanks. :)

@derekbit have applied this. Will let it run for a while to get some logs :)

longhorn-manager-4xkdg.log
log from last terminated longhorn-manager pod on same node as pvc

longhorn-manager-4xkdg.log
one more.

longhorn-manager-4xkdg.log log from last terminated longhorn-manager pod on same node as pvc

Hello @tbertenshaw
Just figured out the root cause, can you try derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename-v4? Many thanks.

applied.  The share-manager has restarted about 5 times, but no longhorn-manager restarts as of yet. Any particular logs? Here are the logs from the longhorn-manager on the same node as th
longhorn-manager-4xkdg.log
e  share-manager (interestingly the share-manager seems to be created on the same node now when it restarts)

Hi @derekbit applied. The share-manager has restarted about 5 times, but no longhorn-manager restarts as of yet. Any particular logs? Here are the logs from the longhorn-manager on the same node as the share-manager (interestingly the share-manager seems to be created on the same node now when it restarts) longhorn-manager-4xkdg.log

Hmm, why is the error same as https://github.com/longhorn/longhorn/issues/5183#issuecomment-1373362040 and is not matched the codes...
Looks the image is not updated yet.

@derekbit checked and yeah using your new image

@derekbit checked and yeah using your new image

I see. This is weird...
I can confirm the issue is caused by the immediate check after creating the share-manager pod from the messges. But the last one make me confused if the fix took place. I added more log messages for distingushing the images in derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename-v5, and you can try it. Sorry for taking so much of your time.

@derekbit apologies i might have reuploaded the same logs. Let me patch with your new image and then reupload.

@derekbit longhorn-manager.zip
ok can see at least 3 restarts of share-manager during watching. longhorn-manager not restarting yet so seems stable in that regard. Uploaded logs from the 3 longhorn-manager pods. p42sv is the one on the same node as the share-manager

@derekbit longhorn-manager.zip ok can see at least 3 restarts of share-manager during watching. longhorn-manager not restarting yet so seems stable in that regard. Uploaded logs from the 3 longhorn-manager pods. p42sv is the one on the same node as the share-manager
Hope that helps :)

Cool @tbertenshaw. The fix for checking pod.spec.nodeName took place. I can confirm the direction is correct.
The last issue is that pod.status.phase is pending and share-manager controller marks it as error. Then, repeatedly creation and deletion.
Same retry method (waiting for the pod becomes running) is applied. You can try derekbit/longhorn-manager:v1.4.0-rwx-verify-pod-nodename-v6. Thank you.

have applied and can see quite a few restarts already.
longhorn-manager-2.zip
qq2l7 is the pod on the same node as share-manager

have applied and can see quite a few restarts already. longhorn-manager-2.zip qq2l7 is the pod on the same node as share-manager

If you scale down the workloads using the RWX volume and let the volume be detached, then scale up the workloads. Will it stop restarts?

No such luck, did 3 => 0 =>1 =>3 and still restarting share-manager

supportbundle_cf947ccc-9d7e-490f-8df5-f22db6f413f7_2023-01-06T14-28-35Z.zip
@derekbit :)

@tbertenshaw
Because your issue is different than the issue @jinserk encountered,  can you open a ticket for tracking what you encountered? Provide the env, steps and the latest support bundle. We will handle it in the new ticket. Thank you.

done 👍

I can not reproduce this issue in my test environment, using Longhorn manager 91d7a7 create a RWX volume with 7 pods attached to, the share manager pod already running for 18 hours without restart  and data in mount point kept consistent. Close this ticket and will keep following https://github.com/longhorn/longhorn/issues/5204


how to add ChromeWebDriver path in kali linux with chrome

hey friends can anybody help me out !!!
i installed the required Dependencies
as u said for  for Automatic Install
$ pip install -r requirements.txt
this didnt work bcoz you need to install pyqt5 for python3, not python 2.
So i tried  with
$apt install python3-pyqt5
then i did
$ pip install selenium
and all goes well
and it worked and rest all work but i am not getting how to add ChromeWebDriver path in kali linux with chrome.
can anyone help me out and explain the step by step process.!!!!

@vishasangwa  copy the driver file to any one of the following location

/usr/bin/
/usr/$USER/.local/bin
/bin


Fix NodeLoadBalancer health check

Fix for https://github.com/Mashape/mashape-distributions/issues/86. Updated ELB health to ping port 4243

You'll need to adjust the AWS::AutoScaling::AutoScalingGroup for node and storage to include "HealthCheckType": "ELB",
Might be better off creating a new (internal only) ELB. The only purpose of this ELB is to health check the nodes so it doesn't need to be public

did not get you, do you want two proxy? one which you have currently and another for internal use? If yes what ASG would be using it?


[BEAM-2899] Use Docker in the ReferenceRunner

Wire up the Docker EnvironmentFactory, and make the environment choice configurable.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

self merge on green


futurebuilder  how to pass params  in initState

`  @override
void initState() {
super.initState();
new Future.delayed(Duration.zero, () {

  StoreProvider.of<AppState>(context)
      .dispatch(asyncGetProductData(widget.arguments));
});
 _productAdsFuture= ApiRepository.getProductAd();
-----------how to  get product parms -----------------
// _setDefaultPreviewImage(product,defaultSku);

}`

Hi @wangxingxing123654
This platform is not meant for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this, you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/ .
Thank You

Hi @wangxingxing123654
It seems you are asking for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this, you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments and I will reopen it.
Thank you