(CustomSageMakerEndpoint): (node.add_dependency not working)

Describe the bug
The generated CloudFormation stack for the below 2 resources fails to deploy
        iam_role = iam.Role(
            self,
            id="sagemaker_execution_role",
            assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com"),
            managed_policies=[iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSageMakerFullAccess")],
            role_name=f"{config_vars.stage}-{config_vars.project}-{model_name}-execution-role",
        )

        custom_sagemaker_endpoint = CustomSageMakerEndpoint(
            self,
            id=model_name,
            container=ContainerImage.from_ecr_repository(repository=self.whisper_ecr_repo, tag=config_vars.image_tag),
            enable_operational_metric=True,
            endpoint_name=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            instance_count=1,
            instance_type=SageMakerInstanceType.ML_G5_2_XLARGE  # type: ignore
            model_data_url=f"s3://{self.model_data_s3_bucket.bucket_name}/model.tar.gz",
            model_id=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            role=iam_role,
            environment={"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": f"whisper-{model_name}"},
        )

The stack is rolled back because the iam_role is not ready for custom_sagemaker_endpoint to be used.
Adding explicit dependency via the below code is also not helping
custom_sagemaker_endpoint.node.add_dependency(iam_role)

as the generated cloudformation output is missing DependsOn section in the resource manifest for "AWS::SageMaker::EndpointConfig"
Expected Behavior
The stack deployment will not fail due to missing/not ready IAM Role.
Current Behavior
Could not access model data at s3://dev-backend-whisper-model-data/model.tar.gz. Please ensure that the role "arn:aws:iam::112233:role/dev-backend-whisper-large-v2-vanilla-exe
cution-role" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com". Also ensure that the role has "s3:
GetObject" permissions and that the object is located in us-east-1. If your Model uses multiple models or uncompressed models, please ensure that the role has "s3:ListBucket" permis
sion. (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 7570c458-14c4-4c9a-bc97-a5287e74faaa; Proxy: null)

Reproduction Steps
Using the following code
        iam_role = iam.Role(
            self,
            id="sagemaker_execution_role",
            assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com"),
            managed_policies=[iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSageMakerFullAccess")],
            role_name=f"{config_vars.stage}-{config_vars.project}-{model_name}-execution-role",
        )

        custom_sagemaker_endpoint = CustomSageMakerEndpoint(
            self,
            id=model_name,
            container=ContainerImage.from_ecr_repository(repository=self.whisper_ecr_repo, tag=config_vars.image_tag),
            enable_operational_metric=True,
            endpoint_name=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            instance_count=1,
            instance_type=SageMakerInstanceType.ML_G5_2_XLARGE  # type: ignore
            model_data_url=f"s3://{self.model_data_s3_bucket.bucket_name}/model.tar.gz",
            model_id=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            role=iam_role,
            environment={"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": f"whisper-{model_name}"},
        )

generate output and deploy via cdk deploy
Possible Solution
the method add_dependency will update the DependsOn section in the generated template correctly
Additional Information/Context
No response
CDK CLI Version
2.140.0 (build 46168aa)
Framework Version
No response
Node.js Version
v18.20.2
OS
macOS Sonoma 14.4.1
Language
Python
Language Version
No response
Region experiencing the issue
us-east-1
Code modification
N/A
Other information
No response
Service quota

[X] I have reviewed the service quotas for this construct

Hi @airmonitor , thank you for reporting this. Did you have a look at the associated sample we have for this construct ? https://github.com/aws-samples/generative-ai-cdk-constructs-samples/blob/main/samples/sagemaker_custom_endpoint/lib/sagemaker_custom_endpoint-stack.ts

Hello @krokoko
Thx for reaching out.
I will try with your suggestion.
Do you have also a working example for adding application autoscaling?
I need to have scalability added there so the number of endpoint instances will scale base on metric.
I've tried with application autoscaling but it was failing due to not working add_dependency method.
In short the endpoint creation was still in progress when application autoscaling tried to add required functionality.
AWS support added DependsOn section to the synthesised template but I was not able to implement the same using CDK due to missing add dependency method.
Regards
Tom

@krokoko
I've tested the proposed solution to use the IAM role that is created with CustomSageMakerEndpoint but apparently, I'm doing something wrong here
For the below scenario I'm getting errors from the cloudformation:
        custom_sagemaker_endpoint = CustomSageMakerEndpoint(
            self,
            id=model_name,
            container=ContainerImage.from_ecr_repository(repository=self.whisper_ecr_repo, tag=config_vars.image_tag),
            enable_operational_metric=True,
            endpoint_name=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            instance_count=1,
            instance_type=SageMakerInstanceType.ML_G5_2_XLARGE  # type: ignore
            if config_vars.stage == "prod"
            else SageMakerInstanceType.ML_G5_XLARGE,  # type: ignore
            model_data_url=f"s3://{self.model_data_s3_bucket.bucket_name}/model.tar.gz",
            model_id=f"{config_vars.stage}-{config_vars.project}-{model_name}",
            environment={"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": f"whisper-{model_name}"},
        )

        self.whisper_ecr_repo.grant_pull(custom_sagemaker_endpoint.role) # ECR Repository
        self.model_data_s3_bucket.grant_read_write(custom_sagemaker_endpoint.role) # S3 bucket

Error from CloudFormation
9:56:57 AM | CREATE_FAILED        | AWS::SageMaker::Model                       | devbackendwhisperl...odellargev2vanilla
Could not access model data at s3://s3-bucket/model.tar.gz. Please ensure that the role "arn:aws:iam::11223344:role/dev-backend-whisper-core--largev2vanilla
RoleC7CE1EE-xaSDM2XaG6fE" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com". Also ensure that the
role has "s3:GetObject" permissions and that the object is located in us-east-1. If your Model uses multiple models or uncompressed models, please ensure that the role has "s3:ListB
ucket" permission. (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 591d86b1-0e24-4d83-a7ec-f3f12b30402d; Proxy: null)

 ❌  backend-whisper-pipeline/dev-backend-whisper-core-stage/core-stack (dev-backend-whisper-core-stage-core-stack) failed: Error: The stack named dev-backend-whisper-core-stage-core-stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Could not access model data at s3://s3-bucket/model.tar.gz. Please ensure that the role "arn:aws:iam::11223344:role/dev-backend-whisper-core--largev2vanillaRoleC7CE1EE-xaSDM2XaG6fE" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com". Also ensure that the role has "s3:GetObject" permissions and that the object is located in us-east-1. If your Model uses multiple models or uncompressed models, please ensure that the role has "s3:ListBucket" permission. (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 591d86b1-0e24-4d83-a7ec-f3f12b30402d; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:435:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:438:199515)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:438:181237

 ❌ Deployment failed: Error: The stack named dev-backend-whisper-core-stage-core-stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Could not access model data at s3://s3-bucket/model.tar.gz. Please ensure that the role "arn:aws:iam::11223344:role/dev-backend-whisper-core--largev2vanillaRoleC7CE1EE-xaSDM2XaG6fE" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com". Also ensure that the role has "s3:GetObject" permissions and that the object is located in us-east-1. If your Model uses multiple models or uncompressed models, please ensure that the role has "s3:ListBucket" permission. (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 591d86b1-0e24-4d83-a7ec-f3f12b30402d; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:435:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:438:199515)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:438:181237

The stack named dev-backend-whisper-core-stage-core-stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Could not access model data at s3://s3-bucket/model.tar.gz. Please ensure that the role "arn:aws:iam::11223344:role/dev-backend-whisper-core--largev2vanillaRoleC7CE1EE-xaSDM2XaG6fE" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "sagemaker.amazonaws.com". Also ensure that the role has "s3:GetObject" permissions and that the object is located in us-east-1. If your Model uses multiple models or uncompressed models, please ensure that the role has "s3:ListBucket" permission. (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 591d86b1-0e24-4d83-a7ec-f3f12b30402d; Proxy: null)

Hi @airmonitor , did you provide your role with permissions to access the S3 bucket storing your artifacts ? We have this demonstrated in the sample here: https://github.com/aws-samples/generative-ai-cdk-constructs-samples/blob/main/samples/sagemaker_custom_endpoint/lib/sagemaker_custom_endpoint-stack.ts#L43C5-L57C7

Hey @krokoko.
I'm not sure if I get your intention here.
The example for extending IAM permissions is using non optimal way for granting necessary permissions
    customEndpoint.addToRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: [
          's3:GetObject',
          's3:GetObject*',
          's3:GetBucket*',
          's3:List*',
        ],
        resources: [
          'BUCKET_ARN',
          'BUCKET_ARN/*',
        ],
      }),
    );

In CDK we have a better way, shorter way to achieve the required results using the following examples
        self.whisper_ecr_repo.grant_pull(custom_sagemaker_endpoint.role) # ECR Repository
        self.model_data_s3_bucket.grant_read_write(custom_sagemaker_endpoint.role) # S3 bucket

grant_pull, grant_read_write method for ecr repository and s3 bucket should do the trick.
I'm not sure why we need to get back mentally a few years later when those methods weren't available in CDK?
Regards
Tom

Hi @airmonitor , sorry I just missed the last line of your previous example, I was thinking that permissions were missing on the role to access the model artifacts in S3.
Is the endpoint created in the same region where the bucket was created ?

Hello,
Yes, all resources are deployed in us-east-1

I am not able to reproduce the issue. Tested both with the provided role by the construct, and by creating a role outside of the construct and passing it through constructor:
const roleSG = new iam.Role(this, 'Role', {
      assumedBy: new iam.ServicePrincipal('sagemaker.amazonaws.com'),
      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSageMakerFullAccess'),
      ],
    });

const endpoint = new genai.CustomSageMakerEndpoint(this, 'testllavaendpoint', {
...
role: roleSG
...
}

In both cases, the construct is successfully deployed.
One thing is for the samples we have and the tests I did, I am using the default bucket created by the SageMaker SDK (see https://sagemaker.readthedocs.io/en/stable/api/utility/session.html, default_bucket) when preparing the model. It might be worth on your end to create the same default session bucket and compare it with the bucket you provide to the construct, and see if any configuration difference stands out.
I will close the issue, if you get more details and/or can upload a sample stack to a public repo which we can use to consistently reproduce the issue, please reopen this ticket. Thank you !


TypeError: a is null in project-progress-createapp-nextstep.component.js

https://errortracking.prod-preview.openshift.io/openshift_io/fabric8-launcher-frontend/issues/7060/
TypeError: a is null
  at listener (./node_modules/ngx-launcher/src/app/launcher/create-app/project-progress-createapp-nextstep/project-progress-createapp-nextstep.component.js:36:59)
  at func (./node_modules/zone.js/dist/zone.js:1188:1)
  at wrapped (./node_modules/raven-js/src/raven.js:377:1)
  at delegate (./node_modules/zone.js/dist/zone.js:421:1)
  at this (./node_modules/@angular/core/fesm5/core.js:3816:1)
...
(5 additional frame(s) were not displayed)

https://errortracking.prod-preview.openshift.io/openshift_io/fabric8-launcher-frontend/issues/7096/


[Tue Dec 13 11:06:28 2022] blk_update_request: I/O error, dev sdc, sector 397616 op 0x1:(WRITE) flags 0x0 phys_seg 218 prio class 0

sudo dmesg -T logs
minimal reproducible steps on k3s:-
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack --set storage.tsdb.retention.time=3GB -f ./values.yml

Resources:-
values.yml
https://github.com/prometheus-community/helm-charts

@derekbit
Spoken to fast, problem happened again even using longhorn version 1.3.0.
Dec 13 11:03:29 kube03 iscsid[1752]: iscsid: connect to <IP_ADDRESS>:3260 failed (No route to host)
Dec 13 11:02:22 kube03 iscsid[1752]: iscsid: Kernel reported iSCSI connection 37:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)

I will try updating the system packages on all of our nodes
Upgrading:
 containerd.io                                            x86_64                                            1.6.12-3.1.el9                                            docker-ce-stable                                                          32 M
 tzdata                                                   noarch                                            2022g-1.el9_1                                             rhel-9-for-x86_64-baseos-rpms                                            839 k

@iMikeG6  Thank you for the update.
@proteche I'm investigating the case, will update the result soon. Intuitively, the two cases look triggered by different root causes.

@iMikeG6 @derekbit i can confirm that i am not facing this issue anymore, for me the main culprit was prometheus-graphana installation  using helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack  with longhorn as specified here as minimal reproducible step. Thanks for help.

Just leaving some tips here:

reduce the vm.dirty_ratio and related parameters
https://www.suse.com/support/kb/doc/?id=000017857
setup priority classes, so that in case of resource constraints you guard against the scheduler evicting the longhorn pods and allow for preempting other lower priority pods in case of failover
https://longhorn.io/docs/1.4.0/advanced-resources/deploy/priority-class/
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

@iMikeG6 how are the disks attached to the node local/icsi? Are these ssd drives or old spinning rust?
Make sure to lower the vm.dirty settings to have more consistent io flushing.
The iptables stuff is concerning it could be an incompatibility of iptables/nftables, flannel cni had something similar.
REF: https://github.com/flannel-io/flannel/issues/1415
I don't know anything about Network CNI: Antrea version 1.7
@Oats87 @brandond Do you know anything about the Antrea cni version 1.7 in combination with RKE1 usage?
kube01 kernel: iptables[3853612]: segfault at 80 ip 00007f1bfb5f2964 sp 00007fff90e741a8 error 4 in libnftnl.so.11.2.0[7f1bfb5ed000+19000]

@brandond Do you know anything about the Antrea cni version 1.7 in combination with RKE1 usage?

I know nothing about either Antrea nor RKE1, sorry.

Hi @joshimoo
thank you very much for this hint. Indeed, I had never though about pod eviction and the vm.dirty settings, which would make sense. The drives are old SAS 10K spinning disks. I gonna try tweaking the setting and add priority class to longhorn and see if that makes a difference.
@brandond Thanks for your help. Though, when the cluster has the 5 vcluster shutdown, only Prometheus and Grafana volumes are attached to LH, there's almost no error.
We also tried to update the system packages, docker, etc, but the problem persist.

After setting vm.dirty_ratio=10 and vm.dirty_background_ratio=5 and also set LH priority class to system-node-critical the problem still occur

I am currently inclined to the 2 directions as @brandond mentioned above:

The disks on host are slow. It is affecting ETCD as well: https://gist.github.com/iMikeG6/f9df95394426ad6a68e44263e3132d8a#file-dmesg-log-22-12-08-L715
Problem with docker runtime that restarts all containers of all pods on the nodes . For example, all containers on node kube01 are restarted at:

➜  longhorn-support-bundle_15e21d6b-8903-4918-b006-631f01847fb5_2022-12-08T08-55-40Z cat longhorn-support-bundle_15e21d6b-8903-4918-b006-631f01847fb5_2022-12-08T08-55-40Z/yamls/kubernetes/pods.yaml| grep -i "nodename: kube01" -A 200 | grep -i "running" -A 2 |grep -i "startedAt:"
          startedat: "2022-12-07T09:12:02Z"
          startedat: "2022-12-07T09:11:17Z"
          startedat: "2022-12-07T09:11:52Z"
          startedat: "2022-12-07T09:10:50Z"
          startedat: "2022-12-07T09:10:38Z"
          startedat: "2022-12-07T09:11:17Z"
          startedat: "2022-12-07T09:10:23Z"
          startedat: "2022-12-07T09:11:18Z"
          startedat: "2022-12-07T09:10:26Z"
          startedat: "2022-12-07T09:09:55Z"
          startedat: "2022-12-07T09:12:27Z"
          startedat: "2022-12-07T09:12:28Z"
          startedat: "2022-12-07T09:10:44Z"
          startedat: "2022-12-07T09:10:15Z"
          startedat: "2022-12-07T09:11:55Z"
          startedat: "2022-12-07T09:11:55Z"
          startedat: "2022-12-07T09:11:58Z"
          startedat: "2022-12-07T09:12:23Z"
          startedat: "2022-12-07T09:11:24Z"

After setting vm.dirty_ratio=10 and vm.dirty_background_ratio=5 and also set LH priority class to system-node-critical the problem still occur

@iMikeG6 Could you send us an updated support bundle for this?

After setting vm.dirty_ratio=10 and vm.dirty_background_ratio=5 and also set LH priority class to system-node-critical the problem still occur

@iMikeG6 Regarding to this case, could you send us an updated support bundle, docker service logs on all nodes, dmesg on all nodes?

Wow yeah, you had etcd blocked for more than two minutes on a fsync call (ext4_sync_file).
[Thu Dec  8 20:36:41 2022] INFO: task etcd:2094053 blocked for more than 122 seconds.
[Thu Dec  8 20:36:41 2022]       Tainted: G          I    X --------- ---  5.14.0-162.6.1.el9_1.x86_64 #1
[Thu Dec  8 20:36:41 2022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Dec  8 20:36:41 2022] task:etcd            state:D stack:    0 pid:2094053 ppid:2094011 flags:0x00000004
[Thu Dec  8 20:36:41 2022] Call Trace:
[Thu Dec  8 20:36:41 2022]  __schedule+0x206/0x580
[Thu Dec  8 20:36:41 2022]  schedule+0x43/0xa0
[Thu Dec  8 20:36:41 2022]  io_schedule+0x42/0x70
[Thu Dec  8 20:36:41 2022]  folio_wait_bit_common+0x147/0x380
[Thu Dec  8 20:36:41 2022]  ? find_get_pages_range_tag+0x199/0x1e0
[Thu Dec  8 20:36:41 2022]  ? file_fdatawait_range+0x20/0x20
[Thu Dec  8 20:36:41 2022]  folio_wait_writeback+0x28/0x80
[Thu Dec  8 20:36:41 2022]  __filemap_fdatawait_range+0x81/0xf0
[Thu Dec  8 20:36:41 2022]  ? __filemap_fdatawrite_range+0xc0/0xe0
[Thu Dec  8 20:36:41 2022]  file_write_and_wait_range+0x72/0xa0
[Thu Dec  8 20:36:41 2022]  ext4_sync_file+0x73/0x370 [ext4]
[Thu Dec  8 20:36:41 2022]  __x64_sys_fdatasync+0x43/0x80
[Thu Dec  8 20:36:41 2022]  do_syscall_64+0x59/0x90
[Thu Dec  8 20:36:41 2022]  ? exc_page_fault+0x62/0x150
[Thu Dec  8 20:36:41 2022]  entry_SYSCALL_64_after_hwframe+0x63/0xcd

Etcd calls fsync to commit EVERY write so this will happen a lot. Note that if you have etcd and longhorn on the same backing disks, the etcd fsync calls will also need to flush the LH writes as well, so I would recommend against having them on the same physical disks. I would also try to confirm that you don't have some problem with your disks or storage controller; even on a pathologically overloaded system I wouldn't expect a fsync call to hang for that long unless something else is broken.

I can confirm that even with placing etcd on a separate physical NVME disk, that the situation of using longhorn coupled with kube-prometheus-stack does exhibit this issue and actually CAUSES delays that brings the k3s node offline (places it into a NotReady) status.
I'm using v1.4.1.

@CJCShadowsan Can you provide more details including support bundle, setup information, setup, dmesg, uname -r of each nodes for further investigation when you are running into the issue?

I've also reduced the number of recovery-backends to my deployment down to 1 because I did see an NFS-ganesha error, which ended up showing itself and led me to #5183 - Which is still present on v1.4.1 - But I am not certain whatever fix went in resolved itself so i've done that as a stop-gap.

I've also reduced the number of recovery-backends to my deployment down to 1 because I did see an NFS-ganesha error, which ended up showing itself and led me to #5183 - Which is still present on v1.4.1 - But I am not certain whatever fix went in resolved itself so i've done that as a stop-gap.

What's the error message?

@CJCShadowsan Can you provide more details including

support bundle
setup information
dmesg, uname -r of each nodes
how to trigger the issue
for further investigation when you are running into the issue?

I'm not certain when i'm running into the issue, only that this has happened... 5 times in the last 25 days or so. There doesn't seem to be a pattern.
[root@cca8102 ~]# uname -a
Linux cca8102 4.18.0-425.3.1.el8.x86_64 #1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@cca8102 ~]# uname -r
4.18.0-425.3.1.el8.x86_64

dmesg from the node in question:
[29416.542681] perf: interrupt took too long (4038 > 3865), lowering kernel.perf_event_max_sample_rate to 49000
[252372.667269] perf: interrupt took too long (5795 > 5047), lowering kernel.perf_event_max_sample_rate to 34000
[478497.628083] sd 19:0:0:1: [sdf] tag#54 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628087] sd 19:0:0:1: [sdf] tag#54 Sense Key : Medium Error [current]
[478497.628089] sd 19:0:0:1: [sdf] tag#54 Add. Sense: Unrecovered read error
[478497.628090] sd 19:0:0:1: [sdf] tag#54 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478497.628091] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478497.628122] Buffer I/O error on dev sdf, logical block 0, lost async page write
[478497.628140] Buffer I/O error on dev sdf, logical block 1, lost async page write
[478497.628161] sd 19:0:0:1: [sdf] tag#55 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628163] sd 19:0:0:1: [sdf] tag#55 Sense Key : Medium Error [current]
[478497.628164] sd 19:0:0:1: [sdf] tag#55 Add. Sense: Unrecovered read error
[478497.628165] sd 19:0:0:1: [sdf] tag#55 CDB: Write(10) 2a 00 00 40 00 08 00 00 08 00
[478497.628165] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628184] Buffer I/O error on dev sdf, logical block 524289, lost async page write
[478497.628201] sd 19:0:0:1: [sdf] tag#57 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628202] sd 19:0:0:1: [sdf] tag#57 Sense Key : Medium Error [current]
[478497.628203] sd 19:0:0:1: [sdf] tag#57 Add. Sense: Unrecovered read error
[478497.628204] sd 19:0:0:1: [sdf] tag#57 CDB: Write(10) 2a 00 00 40 01 08 00 00 08 00
[478497.628204] blk_update_request: critical medium error, dev sdf, sector 4194568 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628222] Buffer I/O error on dev sdf, logical block 524321, lost async page write
[478497.628237] sd 19:0:0:1: [sdf] tag#56 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628237] sd 19:0:0:1: [sdf] tag#56 Sense Key : Medium Error [current]
[478497.628238] sd 19:0:0:1: [sdf] tag#56 Add. Sense: Unrecovered read error
[478497.628239] sd 19:0:0:1: [sdf] tag#56 CDB: Write(10) 2a 00 00 40 00 80 00 00 08 00
[478497.628239] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628257] Buffer I/O error on dev sdf, logical block 524304, lost async page write
[478497.628272] sd 19:0:0:1: [sdf] tag#58 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628272] sd 19:0:0:1: [sdf] tag#58 Sense Key : Medium Error [current]
[478497.628273] sd 19:0:0:1: [sdf] tag#58 Add. Sense: Unrecovered read error
[478497.628274] sd 19:0:0:1: [sdf] tag#58 CDB: Write(10) 2a 00 00 40 01 a8 00 00 08 00
[478497.628274] blk_update_request: critical medium error, dev sdf, sector 4194728 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628292] Buffer I/O error on dev sdf, logical block 524341, lost async page write
[478497.628307] sd 19:0:0:1: [sdf] tag#59 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628307] sd 19:0:0:1: [sdf] tag#59 Sense Key : Medium Error [current]
[478497.628308] sd 19:0:0:1: [sdf] tag#59 Add. Sense: Unrecovered read error
[478497.628309] sd 19:0:0:1: [sdf] tag#59 CDB: Write(10) 2a 00 00 40 04 c0 00 00 08 00
[478497.628309] blk_update_request: critical medium error, dev sdf, sector 4195520 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628327] Buffer I/O error on dev sdf, logical block 524440, lost async page write
[478497.628343] sd 19:0:0:1: [sdf] tag#60 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Sense Key : Medium Error [current]
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Add. Sense: Unrecovered read error
[478497.628345] sd 19:0:0:1: [sdf] tag#60 CDB: Write(10) 2a 00 00 41 01 98 00 00 08 00
[478497.628345] blk_update_request: critical medium error, dev sdf, sector 4260248 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628363] Buffer I/O error on dev sdf, logical block 532531, lost async page write
[478512.835388] sd 17:0:0:1: [sdd] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.835392] sd 17:0:0:1: [sdd] tag#42 Sense Key : Medium Error [current]
[478512.835394] sd 17:0:0:1: [sdd] tag#42 Add. Sense: Unrecovered read error
[478512.835395] sd 17:0:0:1: [sdd] tag#42 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478512.835396] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478512.835424] Buffer I/O error on dev sdd, logical block 0, lost async page write
[478512.835442] Buffer I/O error on dev sdd, logical block 1, lost async page write
[478512.900796] sd 17:0:0:1: [sdd] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.900800] sd 17:0:0:1: [sdd] tag#43 Sense Key : Medium Error [current]
[478512.900802] sd 17:0:0:1: [sdd] tag#43 Add. Sense: Unrecovered read error
[478512.900803] sd 17:0:0:1: [sdd] tag#43 CDB: Write(10) 2a 00 00 00 04 10 00 00 10 00
[478512.900804] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0
[478512.900837] Buffer I/O error on dev sdd, logical block 130, lost async page write
[478512.900854] Buffer I/O error on dev sdd, logical block 131, lost async page write
[478512.936104] sd 17:0:0:1: [sdd] tag#44 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.936107] sd 17:0:0:1: [sdd] tag#44 Sense Key : Medium Error [current]
[478512.936109] sd 17:0:0:1: [sdd] tag#44 Add. Sense: Unrecovered read error
[478512.936110] sd 17:0:0:1: [sdd] tag#44 CDB: Write(10) 2a 00 00 00 04 90 00 00 08 00
[478512.936111] blk_update_request: critical medium error, dev sdd, sector 1168 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.936136] Buffer I/O error on dev sdd, logical block 146, lost async page write
[478512.970808] sd 17:0:0:1: [sdd] tag#45 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.970811] sd 17:0:0:1: [sdd] tag#45 Sense Key : Medium Error [current]
[478512.970812] sd 17:0:0:1: [sdd] tag#45 Add. Sense: Unrecovered read error
[478512.970812] sd 17:0:0:1: [sdd] tag#45 CDB: Write(10) 2a 00 00 00 04 b8 00 00 08 00
[478512.970813] blk_update_request: critical medium error, dev sdd, sector 1208 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.970835] Buffer I/O error on dev sdd, logical block 151, lost async page write
[478514.391586] sd 17:0:0:1: [sdd] tag#46 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.391591] sd 17:0:0:1: [sdd] tag#46 Sense Key : Medium Error [current]
[478514.391593] sd 17:0:0:1: [sdd] tag#46 Add. Sense: Unrecovered read error
[478514.391594] sd 17:0:0:1: [sdd] tag#46 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.391595] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.391638] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.391675] EXT4-fs (sdd): previous I/O error to superblock detected
[478514.466483] sd 17:0:0:1: [sdd] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.466488] sd 17:0:0:1: [sdd] tag#51 Sense Key : Medium Error [current]
[478514.466489] sd 17:0:0:1: [sdd] tag#51 Add. Sense: Unrecovered read error
[478514.466492] sd 17:0:0:1: [sdd] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.466493] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.466534] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.467017] EXT4-fs (sdd): I/O error while writing superblock
[478514.467314] EXT4-fs error (device sdd) in ext4_free_blocks:5130: IO failure
[478514.508393] sd 17:0:0:1: [sdd] tag#47 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.508396] sd 17:0:0:1: [sdd] tag#47 Sense Key : Medium Error [current]
[478514.508397] sd 17:0:0:1: [sdd] tag#47 Add. Sense: Unrecovered read error
[478514.508398] sd 17:0:0:1: [sdd] tag#47 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.508399] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.508727] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.508892] EXT4-fs (sdd): I/O error while writing superblock
[478514.577746] sd 17:0:0:1: [sdd] tag#32 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.577749] sd 17:0:0:1: [sdd] tag#32 Sense Key : Medium Error [current]
[478514.577751] sd 17:0:0:1: [sdd] tag#32 Add. Sense: Unrecovered read error
[478514.577752] sd 17:0:0:1: [sdd] tag#32 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.577753] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.578092] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.649212] sd 17:0:0:1: [sdd] tag#33 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.649215] sd 17:0:0:1: [sdd] tag#33 Sense Key : Medium Error [current]
[478514.649216] sd 17:0:0:1: [sdd] tag#33 Add. Sense: Unrecovered read error
[478514.649218] sd 17:0:0:1: [sdd] tag#33 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.649218] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.649915] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.650203] EXT4-fs (sdd): I/O error while writing superblock
[478514.650462] EXT4-fs (sdd): Delayed block allocation failed for inode 97 at logical offset 0 with max blocks 1 with error 5
[478514.654098] EXT4-fs (sdd): This should not happen!! Data will be lost

[478520.086258] sd 17:0:0:1: [sdd] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.086262] sd 17:0:0:1: [sdd] tag#19 Sense Key : Medium Error [current]
[478520.086264] sd 17:0:0:1: [sdd] tag#19 Add. Sense: Unrecovered read error
[478520.086265] sd 17:0:0:1: [sdd] tag#19 CDB: Write(10) 2a 00 00 10 75 e0 00 00 18 00
[478520.086266] blk_update_request: critical medium error, dev sdd, sector 1078752 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[478520.087016] Aborting journal on device sdd-8.
[478520.123617] sd 17:0:0:1: [sdd] tag#34 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.123620] sd 17:0:0:1: [sdd] tag#34 Sense Key : Medium Error [current]
[478520.123622] sd 17:0:0:1: [sdd] tag#34 Add. Sense: Unrecovered read error
[478520.123624] sd 17:0:0:1: [sdd] tag#34 CDB: Write(10) 2a 08 00 10 00 00 00 00 08 00
[478520.123625] blk_update_request: critical medium error, dev sdd, sector 1048576 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478520.124050] Buffer I/O error on dev sdd, logical block 131072, lost sync page write
[478520.124270] JBD2: Error -5 detected when updating journal superblock for sdd-8.
[478521.695178] sd 19:0:0:1: [sdf] tag#61 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.695182] sd 19:0:0:1: [sdf] tag#61 Sense Key : Medium Error [current]
[478521.695184] sd 19:0:0:1: [sdf] tag#61 Add. Sense: Unrecovered read error
[478521.695185] sd 19:0:0:1: [sdf] tag#61 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478521.695186] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.695977] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478521.698348] EXT4-fs (sdf): previous I/O error to superblock detected
[478521.805999] sd 19:0:0:1: [sdf] tag#62 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.806005] sd 19:0:0:1: [sdf] tag#62 Sense Key : Medium Error [current]
[478521.806007] sd 19:0:0:1: [sdf] tag#62 Add. Sense: Unrecovered read error
[478521.806009] sd 19:0:0:1: [sdf] tag#62 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478521.806010] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478521.806906] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478521.807265] EXT4-fs (sdf): I/O error while writing superblock
[478521.982317] sd 19:0:0:1: [sdf] tag#63 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.982321] sd 19:0:0:1: [sdf] tag#63 Sense Key : Medium Error [current]
[478521.982322] sd 19:0:0:1: [sdf] tag#63 Add. Sense: Unrecovered read error
[478521.982324] sd 19:0:0:1: [sdf] tag#63 CDB: Read(10) 28 00 00 40 00 08 00 00 08 00
[478521.982325] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.983051] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 17, block_bitmap = 524289
[478522.059068] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.059071] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478522.059072] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478522.059073] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.059074] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.059503] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.059687] EXT4-fs (sdf): I/O error while writing superblock
[478522.061139] EXT4-fs (sdf): Delayed block allocation failed for inode 139266 at logical offset 0 with max blocks 1 with error 5
[478522.064481] EXT4-fs (sdf): This should not happen!! Data will be lost

[478522.179755] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.179758] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478522.179759] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478522.179760] sd 19:0:0:1: [sdf] tag#49 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478522.179761] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.180091] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.250674] sd 19:0:0:1: [sdf] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.250676] sd 19:0:0:1: [sdf] tag#50 Sense Key : Medium Error [current]
[478522.250678] sd 19:0:0:1: [sdf] tag#50 Add. Sense: Unrecovered read error
[478522.250679] sd 19:0:0:1: [sdf] tag#50 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.250680] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.251358] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.252521] EXT4-fs (sdf): I/O error while writing superblock
[478522.255525] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.322553] sd 19:0:0:1: [sdf] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.322555] sd 19:0:0:1: [sdf] tag#51 Sense Key : Medium Error [current]
[478522.322556] sd 19:0:0:1: [sdf] tag#51 Add. Sense: Unrecovered read error
[478522.322557] sd 19:0:0:1: [sdf] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.322558] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.324001] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.324902] EXT4-fs (sdf): I/O error while writing superblock
[478522.401437] sd 19:0:0:1: [sdf] tag#52 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.401439] sd 19:0:0:1: [sdf] tag#52 Sense Key : Medium Error [current]
[478522.401440] sd 19:0:0:1: [sdf] tag#52 Add. Sense: Unrecovered read error
[478522.401441] sd 19:0:0:1: [sdf] tag#52 CDB: Read(10) 28 00 00 40 00 10 00 00 08 00
[478522.401442] blk_update_request: critical medium error, dev sdf, sector 4194320 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.401910] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 18, block_bitmap = 524290
[478522.475304] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.475489] EXT4-fs (sdf): I/O error while writing superblock
[478522.542490] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139265 (offset 0 size 4096 starting block 622593)
[478522.542496] Buffer I/O error on device sdf, logical block 622592
[478522.823845] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.908908] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.909231] EXT4-fs (sdf): I/O error while writing superblock
[478522.910286] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.986164] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.986323] EXT4-fs (sdf): I/O error while writing superblock
[478523.094949] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478523.136901] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139268 (offset 0 size 36864 starting block 622602)
[478523.136904] Buffer I/O error on device sdf, logical block 622593
[478523.137062] Buffer I/O error on device sdf, logical block 622594
[478523.137203] Buffer I/O error on device sdf, logical block 622595
[478523.137342] Buffer I/O error on device sdf, logical block 622596
[478523.137485] Buffer I/O error on device sdf, logical block 622597
[478523.137617] Buffer I/O error on device sdf, logical block 622598
[478523.137747] Buffer I/O error on device sdf, logical block 622599
[478523.137874] Buffer I/O error on device sdf, logical block 622600
[478523.137998] Buffer I/O error on device sdf, logical block 622601
[478523.178167] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.178312] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478523.267902] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.354850] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139269 (offset 0 size 4096 starting block 622603)
[478523.354865] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139267 (offset 0 size 4096 starting block 622604)
[478528.163725] JBD2: Detected IO errors while flushing file data on sdf-8
[478528.451874] scsi_io_completion_action: 11 callbacks suppressed
[478528.451881] sd 19:0:0:1: [sdf] tag#64 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.451884] sd 19:0:0:1: [sdf] tag#64 Sense Key : Medium Error [current]
[478528.451886] sd 19:0:0:1: [sdf] tag#64 Add. Sense: Unrecovered read error
[478528.451888] sd 19:0:0:1: [sdf] tag#64 CDB: Write(10) 2a 00 00 85 a8 58 00 00 50 00
[478528.451888] print_req_error: 11 callbacks suppressed
[478528.451889] blk_update_request: critical medium error, dev sdf, sector 8759384 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 0
[478528.452451] Aborting journal on device sdf-8.
[478528.525275] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.525278] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478528.525279] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478528.525280] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 84 00 00 00 00 08 00
[478528.525281] blk_update_request: critical medium error, dev sdf, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478528.525533] Buffer I/O error on dev sdf, logical block 1081344, lost sync page write
[478528.525693] JBD2: Error -5 detected when updating journal superblock for sdf-8.
[478544.415585] sd 17:0:0:1: [sdd] tag#35 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478544.415589] sd 17:0:0:1: [sdd] tag#35 Sense Key : Medium Error [current]
[478544.415591] sd 17:0:0:1: [sdd] tag#35 Add. Sense: Unrecovered read error
[478544.415592] sd 17:0:0:1: [sdd] tag#35 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478544.415593] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478544.416217] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478544.416461] EXT4-fs (sdd): I/O error while writing superblock
[478544.417605] EXT4-fs error (device sdd): ext4_journal_check_start:61: Detected aborted journal
[478544.419662] EXT4-fs (sdd): Remounting filesystem read-only
[478581.772011] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478581.772015] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478581.772017] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478581.772018] sd 19:0:0:1: [sdf] tag#49 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478581.772019] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478581.772648] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478581.772885] EXT4-fs: 3 callbacks suppressed
[478581.772887] EXT4-fs (sdf): I/O error while writing superblock
[478581.773063] EXT4-fs error (device sdf): ext4_journal_check_start:61: Detected aborted journal
[478581.774506] EXT4-fs (sdf): Remounting filesystem read-only

@CJCShadowsan Can you provide more details including

support bundle
setup information
dmesg, uname -r of each nodes
how to trigger the issue
for further investigation when you are running into the issue?

I'm not certain when i'm running into the issue, only that this has happened... 5 times in the last 25 days or so. There doesn't seem to be a pattern.
[root@cca8102 ~]# uname -a
Linux cca8102 4.18.0-425.3.1.el8.x86_64 #1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@cca8102 ~]# uname -r
4.18.0-425.3.1.el8.x86_64

dmesg from the node in question:
[29416.542681] perf: interrupt took too long (4038 > 3865), lowering kernel.perf_event_max_sample_rate to 49000
[252372.667269] perf: interrupt took too long (5795 > 5047), lowering kernel.perf_event_max_sample_rate to 34000
[478497.628083] sd 19:0:0:1: [sdf] tag#54 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628087] sd 19:0:0:1: [sdf] tag#54 Sense Key : Medium Error [current]
[478497.628089] sd 19:0:0:1: [sdf] tag#54 Add. Sense: Unrecovered read error
[478497.628090] sd 19:0:0:1: [sdf] tag#54 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478497.628091] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478497.628122] Buffer I/O error on dev sdf, logical block 0, lost async page write
[478497.628140] Buffer I/O error on dev sdf, logical block 1, lost async page write
[478497.628161] sd 19:0:0:1: [sdf] tag#55 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628163] sd 19:0:0:1: [sdf] tag#55 Sense Key : Medium Error [current]
[478497.628164] sd 19:0:0:1: [sdf] tag#55 Add. Sense: Unrecovered read error
[478497.628165] sd 19:0:0:1: [sdf] tag#55 CDB: Write(10) 2a 00 00 40 00 08 00 00 08 00
[478497.628165] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628184] Buffer I/O error on dev sdf, logical block 524289, lost async page write
[478497.628201] sd 19:0:0:1: [sdf] tag#57 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628202] sd 19:0:0:1: [sdf] tag#57 Sense Key : Medium Error [current]
[478497.628203] sd 19:0:0:1: [sdf] tag#57 Add. Sense: Unrecovered read error
[478497.628204] sd 19:0:0:1: [sdf] tag#57 CDB: Write(10) 2a 00 00 40 01 08 00 00 08 00
[478497.628204] blk_update_request: critical medium error, dev sdf, sector 4194568 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628222] Buffer I/O error on dev sdf, logical block 524321, lost async page write
[478497.628237] sd 19:0:0:1: [sdf] tag#56 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628237] sd 19:0:0:1: [sdf] tag#56 Sense Key : Medium Error [current]
[478497.628238] sd 19:0:0:1: [sdf] tag#56 Add. Sense: Unrecovered read error
[478497.628239] sd 19:0:0:1: [sdf] tag#56 CDB: Write(10) 2a 00 00 40 00 80 00 00 08 00
[478497.628239] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628257] Buffer I/O error on dev sdf, logical block 524304, lost async page write
[478497.628272] sd 19:0:0:1: [sdf] tag#58 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628272] sd 19:0:0:1: [sdf] tag#58 Sense Key : Medium Error [current]
[478497.628273] sd 19:0:0:1: [sdf] tag#58 Add. Sense: Unrecovered read error
[478497.628274] sd 19:0:0:1: [sdf] tag#58 CDB: Write(10) 2a 00 00 40 01 a8 00 00 08 00
[478497.628274] blk_update_request: critical medium error, dev sdf, sector 4194728 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628292] Buffer I/O error on dev sdf, logical block 524341, lost async page write
[478497.628307] sd 19:0:0:1: [sdf] tag#59 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628307] sd 19:0:0:1: [sdf] tag#59 Sense Key : Medium Error [current]
[478497.628308] sd 19:0:0:1: [sdf] tag#59 Add. Sense: Unrecovered read error
[478497.628309] sd 19:0:0:1: [sdf] tag#59 CDB: Write(10) 2a 00 00 40 04 c0 00 00 08 00
[478497.628309] blk_update_request: critical medium error, dev sdf, sector 4195520 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628327] Buffer I/O error on dev sdf, logical block 524440, lost async page write
[478497.628343] sd 19:0:0:1: [sdf] tag#60 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Sense Key : Medium Error [current]
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Add. Sense: Unrecovered read error
[478497.628345] sd 19:0:0:1: [sdf] tag#60 CDB: Write(10) 2a 00 00 41 01 98 00 00 08 00
[478497.628345] blk_update_request: critical medium error, dev sdf, sector 4260248 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628363] Buffer I/O error on dev sdf, logical block 532531, lost async page write
[478512.835388] sd 17:0:0:1: [sdd] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.835392] sd 17:0:0:1: [sdd] tag#42 Sense Key : Medium Error [current]
[478512.835394] sd 17:0:0:1: [sdd] tag#42 Add. Sense: Unrecovered read error
[478512.835395] sd 17:0:0:1: [sdd] tag#42 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478512.835396] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478512.835424] Buffer I/O error on dev sdd, logical block 0, lost async page write
[478512.835442] Buffer I/O error on dev sdd, logical block 1, lost async page write
[478512.900796] sd 17:0:0:1: [sdd] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.900800] sd 17:0:0:1: [sdd] tag#43 Sense Key : Medium Error [current]
[478512.900802] sd 17:0:0:1: [sdd] tag#43 Add. Sense: Unrecovered read error
[478512.900803] sd 17:0:0:1: [sdd] tag#43 CDB: Write(10) 2a 00 00 00 04 10 00 00 10 00
[478512.900804] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0
[478512.900837] Buffer I/O error on dev sdd, logical block 130, lost async page write
[478512.900854] Buffer I/O error on dev sdd, logical block 131, lost async page write
[478512.936104] sd 17:0:0:1: [sdd] tag#44 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.936107] sd 17:0:0:1: [sdd] tag#44 Sense Key : Medium Error [current]
[478512.936109] sd 17:0:0:1: [sdd] tag#44 Add. Sense: Unrecovered read error
[478512.936110] sd 17:0:0:1: [sdd] tag#44 CDB: Write(10) 2a 00 00 00 04 90 00 00 08 00
[478512.936111] blk_update_request: critical medium error, dev sdd, sector 1168 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.936136] Buffer I/O error on dev sdd, logical block 146, lost async page write
[478512.970808] sd 17:0:0:1: [sdd] tag#45 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.970811] sd 17:0:0:1: [sdd] tag#45 Sense Key : Medium Error [current]
[478512.970812] sd 17:0:0:1: [sdd] tag#45 Add. Sense: Unrecovered read error
[478512.970812] sd 17:0:0:1: [sdd] tag#45 CDB: Write(10) 2a 00 00 00 04 b8 00 00 08 00
[478512.970813] blk_update_request: critical medium error, dev sdd, sector 1208 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.970835] Buffer I/O error on dev sdd, logical block 151, lost async page write
[478514.391586] sd 17:0:0:1: [sdd] tag#46 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.391591] sd 17:0:0:1: [sdd] tag#46 Sense Key : Medium Error [current]
[478514.391593] sd 17:0:0:1: [sdd] tag#46 Add. Sense: Unrecovered read error
[478514.391594] sd 17:0:0:1: [sdd] tag#46 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.391595] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.391638] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.391675] EXT4-fs (sdd): previous I/O error to superblock detected
[478514.466483] sd 17:0:0:1: [sdd] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.466488] sd 17:0:0:1: [sdd] tag#51 Sense Key : Medium Error [current]
[478514.466489] sd 17:0:0:1: [sdd] tag#51 Add. Sense: Unrecovered read error
[478514.466492] sd 17:0:0:1: [sdd] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.466493] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.466534] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.467017] EXT4-fs (sdd): I/O error while writing superblock
[478514.467314] EXT4-fs error (device sdd) in ext4_free_blocks:5130: IO failure
[478514.508393] sd 17:0:0:1: [sdd] tag#47 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.508396] sd 17:0:0:1: [sdd] tag#47 Sense Key : Medium Error [current]
[478514.508397] sd 17:0:0:1: [sdd] tag#47 Add. Sense: Unrecovered read error
[478514.508398] sd 17:0:0:1: [sdd] tag#47 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.508399] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.508727] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.508892] EXT4-fs (sdd): I/O error while writing superblock
[478514.577746] sd 17:0:0:1: [sdd] tag#32 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.577749] sd 17:0:0:1: [sdd] tag#32 Sense Key : Medium Error [current]
[478514.577751] sd 17:0:0:1: [sdd] tag#32 Add. Sense: Unrecovered read error
[478514.577752] sd 17:0:0:1: [sdd] tag#32 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.577753] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.578092] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.649212] sd 17:0:0:1: [sdd] tag#33 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.649215] sd 17:0:0:1: [sdd] tag#33 Sense Key : Medium Error [current]
[478514.649216] sd 17:0:0:1: [sdd] tag#33 Add. Sense: Unrecovered read error
[478514.649218] sd 17:0:0:1: [sdd] tag#33 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.649218] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.649915] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.650203] EXT4-fs (sdd): I/O error while writing superblock
[478514.650462] EXT4-fs (sdd): Delayed block allocation failed for inode 97 at logical offset 0 with max blocks 1 with error 5
[478514.654098] EXT4-fs (sdd): This should not happen!! Data will be lost

[478520.086258] sd 17:0:0:1: [sdd] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.086262] sd 17:0:0:1: [sdd] tag#19 Sense Key : Medium Error [current]
[478520.086264] sd 17:0:0:1: [sdd] tag#19 Add. Sense: Unrecovered read error
[478520.086265] sd 17:0:0:1: [sdd] tag#19 CDB: Write(10) 2a 00 00 10 75 e0 00 00 18 00
[478520.086266] blk_update_request: critical medium error, dev sdd, sector 1078752 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[478520.087016] Aborting journal on device sdd-8.
[478520.123617] sd 17:0:0:1: [sdd] tag#34 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.123620] sd 17:0:0:1: [sdd] tag#34 Sense Key : Medium Error [current]
[478520.123622] sd 17:0:0:1: [sdd] tag#34 Add. Sense: Unrecovered read error
[478520.123624] sd 17:0:0:1: [sdd] tag#34 CDB: Write(10) 2a 08 00 10 00 00 00 00 08 00
[478520.123625] blk_update_request: critical medium error, dev sdd, sector 1048576 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478520.124050] Buffer I/O error on dev sdd, logical block 131072, lost sync page write
[478520.124270] JBD2: Error -5 detected when updating journal superblock for sdd-8.
[478521.695178] sd 19:0:0:1: [sdf] tag#61 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.695182] sd 19:0:0:1: [sdf] tag#61 Sense Key : Medium Error [current]
[478521.695184] sd 19:0:0:1: [sdf] tag#61 Add. Sense: Unrecovered read error
[478521.695185] sd 19:0:0:1: [sdf] tag#61 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478521.695186] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.695977] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478521.698348] EXT4-fs (sdf): previous I/O error to superblock detected
[478521.805999] sd 19:0:0:1: [sdf] tag#62 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.806005] sd 19:0:0:1: [sdf] tag#62 Sense Key : Medium Error [current]
[478521.806007] sd 19:0:0:1: [sdf] tag#62 Add. Sense: Unrecovered read error
[478521.806009] sd 19:0:0:1: [sdf] tag#62 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478521.806010] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478521.806906] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478521.807265] EXT4-fs (sdf): I/O error while writing superblock
[478521.982317] sd 19:0:0:1: [sdf] tag#63 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.982321] sd 19:0:0:1: [sdf] tag#63 Sense Key : Medium Error [current]
[478521.982322] sd 19:0:0:1: [sdf] tag#63 Add. Sense: Unrecovered read error
[478521.982324] sd 19:0:0:1: [sdf] tag#63 CDB: Read(10) 28 00 00 40 00 08 00 00 08 00
[478521.982325] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.983051] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 17, block_bitmap = 524289
[478522.059068] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.059071] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478522.059072] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478522.059073] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.059074] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.059503] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.059687] EXT4-fs (sdf): I/O error while writing superblock
[478522.061139] EXT4-fs (sdf): Delayed block allocation failed for inode 139266 at logical offset 0 with max blocks 1 with error 5
[478522.064481] EXT4-fs (sdf): This should not happen!! Data will be lost

[478522.179755] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.179758] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478522.179759] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478522.179760] sd 19:0:0:1: [sdf] tag#49 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478522.179761] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.180091] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.250674] sd 19:0:0:1: [sdf] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.250676] sd 19:0:0:1: [sdf] tag#50 Sense Key : Medium Error [current]
[478522.250678] sd 19:0:0:1: [sdf] tag#50 Add. Sense: Unrecovered read error
[478522.250679] sd 19:0:0:1: [sdf] tag#50 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.250680] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.251358] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.252521] EXT4-fs (sdf): I/O error while writing superblock
[478522.255525] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.322553] sd 19:0:0:1: [sdf] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.322555] sd 19:0:0:1: [sdf] tag#51 Sense Key : Medium Error [current]
[478522.322556] sd 19:0:0:1: [sdf] tag#51 Add. Sense: Unrecovered read error
[478522.322557] sd 19:0:0:1: [sdf] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.322558] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.324001] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.324902] EXT4-fs (sdf): I/O error while writing superblock
[478522.401437] sd 19:0:0:1: [sdf] tag#52 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.401439] sd 19:0:0:1: [sdf] tag#52 Sense Key : Medium Error [current]
[478522.401440] sd 19:0:0:1: [sdf] tag#52 Add. Sense: Unrecovered read error
[478522.401441] sd 19:0:0:1: [sdf] tag#52 CDB: Read(10) 28 00 00 40 00 10 00 00 08 00
[478522.401442] blk_update_request: critical medium error, dev sdf, sector 4194320 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.401910] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 18, block_bitmap = 524290
[478522.475304] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.475489] EXT4-fs (sdf): I/O error while writing superblock
[478522.542490] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139265 (offset 0 size 4096 starting block 622593)
[478522.542496] Buffer I/O error on device sdf, logical block 622592
[478522.823845] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.908908] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.909231] EXT4-fs (sdf): I/O error while writing superblock
[478522.910286] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.986164] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.986323] EXT4-fs (sdf): I/O error while writing superblock
[478523.094949] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478523.136901] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139268 (offset 0 size 36864 starting block 622602)
[478523.136904] Buffer I/O error on device sdf, logical block 622593
[478523.137062] Buffer I/O error on device sdf, logical block 622594
[478523.137203] Buffer I/O error on device sdf, logical block 622595
[478523.137342] Buffer I/O error on device sdf, logical block 622596
[478523.137485] Buffer I/O error on device sdf, logical block 622597
[478523.137617] Buffer I/O error on device sdf, logical block 622598
[478523.137747] Buffer I/O error on device sdf, logical block 622599
[478523.137874] Buffer I/O error on device sdf, logical block 622600
[478523.137998] Buffer I/O error on device sdf, logical block 622601
[478523.178167] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.178312] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478523.267902] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.354850] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139269 (offset 0 size 4096 starting block 622603)
[478523.354865] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139267 (offset 0 size 4096 starting block 622604)
[478528.163725] JBD2: Detected IO errors while flushing file data on sdf-8
[478528.451874] scsi_io_completion_action: 11 callbacks suppressed
[478528.451881] sd 19:0:0:1: [sdf] tag#64 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.451884] sd 19:0:0:1: [sdf] tag#64 Sense Key : Medium Error [current]
[478528.451886] sd 19:0:0:1: [sdf] tag#64 Add. Sense: Unrecovered read error
[478528.451888] sd 19:0:0:1: [sdf] tag#64 CDB: Write(10) 2a 00 00 85 a8 58 00 00 50 00
[478528.451888] print_req_error: 11 callbacks suppressed
[478528.451889] blk_update_request: critical medium error, dev sdf, sector 8759384 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 0
[478528.452451] Aborting journal on device sdf-8.
[478528.525275] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.525278] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478528.525279] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478528.525280] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 84 00 00 00 00 08 00
[478528.525281] blk_update_request: critical medium error, dev sdf, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478528.525533] Buffer I/O error on dev sdf, logical block 1081344, lost sync page write
[478528.525693] JBD2: Error -5 detected when updating journal superblock for sdf-8.
[478544.415585] sd 17:0:0:1: [sdd] tag#35 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478544.415589] sd 17:0:0:1: [sdd] tag#35 Sense Key : Medium Error [current]
[478544.415591] sd 17:0:0:1: [sdd] tag#35 Add. Sense: Unrecovered read error
[478544.415592] sd 17:0:0:1: [sdd] tag#35 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478544.415593] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478544.416217] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478544.416461] EXT4-fs (sdd): I/O error while writing superblock
[478544.417605] EXT4-fs error (device sdd): ext4_journal_check_start:61: Detected aborted journal
[478544.419662] EXT4-fs (sdd): Remounting filesystem read-only
[478581.772011] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478581.772015] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478581.772017] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478581.772018] sd 19:0:0:1: [sdf] tag#49 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478581.772019] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478581.772648] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478581.772885] EXT4-fs: 3 callbacks suppressed
[478581.772887] EXT4-fs (sdf): I/O error while writing superblock
[478581.773063] EXT4-fs error (device sdf): ext4_journal_check_start:61: Detected aborted journal
[478581.774506] EXT4-fs (sdf): Remounting filesystem read-only

Further questions

What's the NIC bandwidth between worker nodes?
How many volumes/replicas are in one node?
Are the replicas on NVMe SSDs?

@CJCShadowsan Can you provide more details including

support bundle
setup information
dmesg, uname -r of each nodes
how to trigger the issue
for further investigation when you are running into the issue?

I'm not certain when i'm running into the issue, only that this has happened... 5 times in the last 25 days or so. There doesn't seem to be a pattern.
[root@cca8102 ~]# uname -a
Linux cca8102 4.18.0-425.3.1.el8.x86_64 #1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@cca8102 ~]# uname -r
4.18.0-425.3.1.el8.x86_64

dmesg from the node in question:
[29416.542681] perf: interrupt took too long (4038 > 3865), lowering kernel.perf_event_max_sample_rate to 49000
[252372.667269] perf: interrupt took too long (5795 > 5047), lowering kernel.perf_event_max_sample_rate to 34000
[478497.628083] sd 19:0:0:1: [sdf] tag#54 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628087] sd 19:0:0:1: [sdf] tag#54 Sense Key : Medium Error [current]
[478497.628089] sd 19:0:0:1: [sdf] tag#54 Add. Sense: Unrecovered read error
[478497.628090] sd 19:0:0:1: [sdf] tag#54 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478497.628091] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478497.628122] Buffer I/O error on dev sdf, logical block 0, lost async page write
[478497.628140] Buffer I/O error on dev sdf, logical block 1, lost async page write
[478497.628161] sd 19:0:0:1: [sdf] tag#55 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628163] sd 19:0:0:1: [sdf] tag#55 Sense Key : Medium Error [current]
[478497.628164] sd 19:0:0:1: [sdf] tag#55 Add. Sense: Unrecovered read error
[478497.628165] sd 19:0:0:1: [sdf] tag#55 CDB: Write(10) 2a 00 00 40 00 08 00 00 08 00
[478497.628165] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628184] Buffer I/O error on dev sdf, logical block 524289, lost async page write
[478497.628201] sd 19:0:0:1: [sdf] tag#57 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628202] sd 19:0:0:1: [sdf] tag#57 Sense Key : Medium Error [current]
[478497.628203] sd 19:0:0:1: [sdf] tag#57 Add. Sense: Unrecovered read error
[478497.628204] sd 19:0:0:1: [sdf] tag#57 CDB: Write(10) 2a 00 00 40 01 08 00 00 08 00
[478497.628204] blk_update_request: critical medium error, dev sdf, sector 4194568 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628222] Buffer I/O error on dev sdf, logical block 524321, lost async page write
[478497.628237] sd 19:0:0:1: [sdf] tag#56 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628237] sd 19:0:0:1: [sdf] tag#56 Sense Key : Medium Error [current]
[478497.628238] sd 19:0:0:1: [sdf] tag#56 Add. Sense: Unrecovered read error
[478497.628239] sd 19:0:0:1: [sdf] tag#56 CDB: Write(10) 2a 00 00 40 00 80 00 00 08 00
[478497.628239] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628257] Buffer I/O error on dev sdf, logical block 524304, lost async page write
[478497.628272] sd 19:0:0:1: [sdf] tag#58 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628272] sd 19:0:0:1: [sdf] tag#58 Sense Key : Medium Error [current]
[478497.628273] sd 19:0:0:1: [sdf] tag#58 Add. Sense: Unrecovered read error
[478497.628274] sd 19:0:0:1: [sdf] tag#58 CDB: Write(10) 2a 00 00 40 01 a8 00 00 08 00
[478497.628274] blk_update_request: critical medium error, dev sdf, sector 4194728 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628292] Buffer I/O error on dev sdf, logical block 524341, lost async page write
[478497.628307] sd 19:0:0:1: [sdf] tag#59 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628307] sd 19:0:0:1: [sdf] tag#59 Sense Key : Medium Error [current]
[478497.628308] sd 19:0:0:1: [sdf] tag#59 Add. Sense: Unrecovered read error
[478497.628309] sd 19:0:0:1: [sdf] tag#59 CDB: Write(10) 2a 00 00 40 04 c0 00 00 08 00
[478497.628309] blk_update_request: critical medium error, dev sdf, sector 4195520 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628327] Buffer I/O error on dev sdf, logical block 524440, lost async page write
[478497.628343] sd 19:0:0:1: [sdf] tag#60 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Sense Key : Medium Error [current]
[478497.628344] sd 19:0:0:1: [sdf] tag#60 Add. Sense: Unrecovered read error
[478497.628345] sd 19:0:0:1: [sdf] tag#60 CDB: Write(10) 2a 00 00 41 01 98 00 00 08 00
[478497.628345] blk_update_request: critical medium error, dev sdf, sector 4260248 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478497.628363] Buffer I/O error on dev sdf, logical block 532531, lost async page write
[478512.835388] sd 17:0:0:1: [sdd] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.835392] sd 17:0:0:1: [sdd] tag#42 Sense Key : Medium Error [current]
[478512.835394] sd 17:0:0:1: [sdd] tag#42 Add. Sense: Unrecovered read error
[478512.835395] sd 17:0:0:1: [sdd] tag#42 CDB: Write(10) 2a 00 00 00 00 00 00 00 10 00
[478512.835396] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[478512.835424] Buffer I/O error on dev sdd, logical block 0, lost async page write
[478512.835442] Buffer I/O error on dev sdd, logical block 1, lost async page write
[478512.900796] sd 17:0:0:1: [sdd] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.900800] sd 17:0:0:1: [sdd] tag#43 Sense Key : Medium Error [current]
[478512.900802] sd 17:0:0:1: [sdd] tag#43 Add. Sense: Unrecovered read error
[478512.900803] sd 17:0:0:1: [sdd] tag#43 CDB: Write(10) 2a 00 00 00 04 10 00 00 10 00
[478512.900804] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0
[478512.900837] Buffer I/O error on dev sdd, logical block 130, lost async page write
[478512.900854] Buffer I/O error on dev sdd, logical block 131, lost async page write
[478512.936104] sd 17:0:0:1: [sdd] tag#44 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.936107] sd 17:0:0:1: [sdd] tag#44 Sense Key : Medium Error [current]
[478512.936109] sd 17:0:0:1: [sdd] tag#44 Add. Sense: Unrecovered read error
[478512.936110] sd 17:0:0:1: [sdd] tag#44 CDB: Write(10) 2a 00 00 00 04 90 00 00 08 00
[478512.936111] blk_update_request: critical medium error, dev sdd, sector 1168 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.936136] Buffer I/O error on dev sdd, logical block 146, lost async page write
[478512.970808] sd 17:0:0:1: [sdd] tag#45 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478512.970811] sd 17:0:0:1: [sdd] tag#45 Sense Key : Medium Error [current]
[478512.970812] sd 17:0:0:1: [sdd] tag#45 Add. Sense: Unrecovered read error
[478512.970812] sd 17:0:0:1: [sdd] tag#45 CDB: Write(10) 2a 00 00 00 04 b8 00 00 08 00
[478512.970813] blk_update_request: critical medium error, dev sdd, sector 1208 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[478512.970835] Buffer I/O error on dev sdd, logical block 151, lost async page write
[478514.391586] sd 17:0:0:1: [sdd] tag#46 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.391591] sd 17:0:0:1: [sdd] tag#46 Sense Key : Medium Error [current]
[478514.391593] sd 17:0:0:1: [sdd] tag#46 Add. Sense: Unrecovered read error
[478514.391594] sd 17:0:0:1: [sdd] tag#46 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.391595] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.391638] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.391675] EXT4-fs (sdd): previous I/O error to superblock detected
[478514.466483] sd 17:0:0:1: [sdd] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.466488] sd 17:0:0:1: [sdd] tag#51 Sense Key : Medium Error [current]
[478514.466489] sd 17:0:0:1: [sdd] tag#51 Add. Sense: Unrecovered read error
[478514.466492] sd 17:0:0:1: [sdd] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.466493] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.466534] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.467017] EXT4-fs (sdd): I/O error while writing superblock
[478514.467314] EXT4-fs error (device sdd) in ext4_free_blocks:5130: IO failure
[478514.508393] sd 17:0:0:1: [sdd] tag#47 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.508396] sd 17:0:0:1: [sdd] tag#47 Sense Key : Medium Error [current]
[478514.508397] sd 17:0:0:1: [sdd] tag#47 Add. Sense: Unrecovered read error
[478514.508398] sd 17:0:0:1: [sdd] tag#47 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.508399] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.508727] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.508892] EXT4-fs (sdd): I/O error while writing superblock
[478514.577746] sd 17:0:0:1: [sdd] tag#32 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.577749] sd 17:0:0:1: [sdd] tag#32 Sense Key : Medium Error [current]
[478514.577751] sd 17:0:0:1: [sdd] tag#32 Add. Sense: Unrecovered read error
[478514.577752] sd 17:0:0:1: [sdd] tag#32 CDB: Read(10) 28 00 00 00 04 10 00 00 08 00
[478514.577753] blk_update_request: critical medium error, dev sdd, sector 1040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478514.578092] EXT4-fs error (device sdd): ext4_wait_block_bitmap:520: comm vault: Cannot read block bitmap - block_group = 1, block_bitmap = 130
[478514.649212] sd 17:0:0:1: [sdd] tag#33 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478514.649215] sd 17:0:0:1: [sdd] tag#33 Sense Key : Medium Error [current]
[478514.649216] sd 17:0:0:1: [sdd] tag#33 Add. Sense: Unrecovered read error
[478514.649218] sd 17:0:0:1: [sdd] tag#33 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478514.649218] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478514.649915] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478514.650203] EXT4-fs (sdd): I/O error while writing superblock
[478514.650462] EXT4-fs (sdd): Delayed block allocation failed for inode 97 at logical offset 0 with max blocks 1 with error 5
[478514.654098] EXT4-fs (sdd): This should not happen!! Data will be lost

[478520.086258] sd 17:0:0:1: [sdd] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.086262] sd 17:0:0:1: [sdd] tag#19 Sense Key : Medium Error [current]
[478520.086264] sd 17:0:0:1: [sdd] tag#19 Add. Sense: Unrecovered read error
[478520.086265] sd 17:0:0:1: [sdd] tag#19 CDB: Write(10) 2a 00 00 10 75 e0 00 00 18 00
[478520.086266] blk_update_request: critical medium error, dev sdd, sector 1078752 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[478520.087016] Aborting journal on device sdd-8.
[478520.123617] sd 17:0:0:1: [sdd] tag#34 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478520.123620] sd 17:0:0:1: [sdd] tag#34 Sense Key : Medium Error [current]
[478520.123622] sd 17:0:0:1: [sdd] tag#34 Add. Sense: Unrecovered read error
[478520.123624] sd 17:0:0:1: [sdd] tag#34 CDB: Write(10) 2a 08 00 10 00 00 00 00 08 00
[478520.123625] blk_update_request: critical medium error, dev sdd, sector 1048576 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478520.124050] Buffer I/O error on dev sdd, logical block 131072, lost sync page write
[478520.124270] JBD2: Error -5 detected when updating journal superblock for sdd-8.
[478521.695178] sd 19:0:0:1: [sdf] tag#61 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.695182] sd 19:0:0:1: [sdf] tag#61 Sense Key : Medium Error [current]
[478521.695184] sd 19:0:0:1: [sdf] tag#61 Add. Sense: Unrecovered read error
[478521.695185] sd 19:0:0:1: [sdf] tag#61 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478521.695186] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.695977] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478521.698348] EXT4-fs (sdf): previous I/O error to superblock detected
[478521.805999] sd 19:0:0:1: [sdf] tag#62 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.806005] sd 19:0:0:1: [sdf] tag#62 Sense Key : Medium Error [current]
[478521.806007] sd 19:0:0:1: [sdf] tag#62 Add. Sense: Unrecovered read error
[478521.806009] sd 19:0:0:1: [sdf] tag#62 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478521.806010] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478521.806906] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478521.807265] EXT4-fs (sdf): I/O error while writing superblock
[478521.982317] sd 19:0:0:1: [sdf] tag#63 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478521.982321] sd 19:0:0:1: [sdf] tag#63 Sense Key : Medium Error [current]
[478521.982322] sd 19:0:0:1: [sdf] tag#63 Add. Sense: Unrecovered read error
[478521.982324] sd 19:0:0:1: [sdf] tag#63 CDB: Read(10) 28 00 00 40 00 08 00 00 08 00
[478521.982325] blk_update_request: critical medium error, dev sdf, sector 4194312 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478521.983051] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 17, block_bitmap = 524289
[478522.059068] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.059071] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478522.059072] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478522.059073] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.059074] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.059503] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.059687] EXT4-fs (sdf): I/O error while writing superblock
[478522.061139] EXT4-fs (sdf): Delayed block allocation failed for inode 139266 at logical offset 0 with max blocks 1 with error 5
[478522.064481] EXT4-fs (sdf): This should not happen!! Data will be lost

[478522.179755] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.179758] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478522.179759] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478522.179760] sd 19:0:0:1: [sdf] tag#49 CDB: Read(10) 28 00 00 40 00 80 00 00 08 00
[478522.179761] blk_update_request: critical medium error, dev sdf, sector 4194432 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.180091] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.250674] sd 19:0:0:1: [sdf] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.250676] sd 19:0:0:1: [sdf] tag#50 Sense Key : Medium Error [current]
[478522.250678] sd 19:0:0:1: [sdf] tag#50 Add. Sense: Unrecovered read error
[478522.250679] sd 19:0:0:1: [sdf] tag#50 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.250680] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.251358] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.252521] EXT4-fs (sdf): I/O error while writing superblock
[478522.255525] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.322553] sd 19:0:0:1: [sdf] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.322555] sd 19:0:0:1: [sdf] tag#51 Sense Key : Medium Error [current]
[478522.322556] sd 19:0:0:1: [sdf] tag#51 Add. Sense: Unrecovered read error
[478522.322557] sd 19:0:0:1: [sdf] tag#51 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478522.322558] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478522.324001] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.324902] EXT4-fs (sdf): I/O error while writing superblock
[478522.401437] sd 19:0:0:1: [sdf] tag#52 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478522.401439] sd 19:0:0:1: [sdf] tag#52 Sense Key : Medium Error [current]
[478522.401440] sd 19:0:0:1: [sdf] tag#52 Add. Sense: Unrecovered read error
[478522.401441] sd 19:0:0:1: [sdf] tag#52 CDB: Read(10) 28 00 00 40 00 10 00 00 08 00
[478522.401442] blk_update_request: critical medium error, dev sdf, sector 4194320 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[478522.401910] EXT4-fs error (device sdf): ext4_wait_block_bitmap:520: comm postgres: Cannot read block bitmap - block_group = 18, block_bitmap = 524290
[478522.475304] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.475489] EXT4-fs (sdf): I/O error while writing superblock
[478522.542490] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139265 (offset 0 size 4096 starting block 622593)
[478522.542496] Buffer I/O error on device sdf, logical block 622592
[478522.823845] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478522.908908] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.909231] EXT4-fs (sdf): I/O error while writing superblock
[478522.910286] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478522.986164] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478522.986323] EXT4-fs (sdf): I/O error while writing superblock
[478523.094949] EXT4-fs error (device sdf): ext4_read_inode_bitmap:200: comm postgres: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
[478523.136901] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139268 (offset 0 size 36864 starting block 622602)
[478523.136904] Buffer I/O error on device sdf, logical block 622593
[478523.137062] Buffer I/O error on device sdf, logical block 622594
[478523.137203] Buffer I/O error on device sdf, logical block 622595
[478523.137342] Buffer I/O error on device sdf, logical block 622596
[478523.137485] Buffer I/O error on device sdf, logical block 622597
[478523.137617] Buffer I/O error on device sdf, logical block 622598
[478523.137747] Buffer I/O error on device sdf, logical block 622599
[478523.137874] Buffer I/O error on device sdf, logical block 622600
[478523.137998] Buffer I/O error on device sdf, logical block 622601
[478523.178167] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.178312] EXT4-fs error (device sdf) in ext4_free_inode:358: IO failure
[478523.267902] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478523.354850] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139269 (offset 0 size 4096 starting block 622603)
[478523.354865] EXT4-fs warning (device sdf): ext4_end_bio:325: I/O error 7 writing to inode 139267 (offset 0 size 4096 starting block 622604)
[478528.163725] JBD2: Detected IO errors while flushing file data on sdf-8
[478528.451874] scsi_io_completion_action: 11 callbacks suppressed
[478528.451881] sd 19:0:0:1: [sdf] tag#64 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.451884] sd 19:0:0:1: [sdf] tag#64 Sense Key : Medium Error [current]
[478528.451886] sd 19:0:0:1: [sdf] tag#64 Add. Sense: Unrecovered read error
[478528.451888] sd 19:0:0:1: [sdf] tag#64 CDB: Write(10) 2a 00 00 85 a8 58 00 00 50 00
[478528.451888] print_req_error: 11 callbacks suppressed
[478528.451889] blk_update_request: critical medium error, dev sdf, sector 8759384 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 0
[478528.452451] Aborting journal on device sdf-8.
[478528.525275] sd 19:0:0:1: [sdf] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478528.525278] sd 19:0:0:1: [sdf] tag#48 Sense Key : Medium Error [current]
[478528.525279] sd 19:0:0:1: [sdf] tag#48 Add. Sense: Unrecovered read error
[478528.525280] sd 19:0:0:1: [sdf] tag#48 CDB: Write(10) 2a 08 00 84 00 00 00 00 08 00
[478528.525281] blk_update_request: critical medium error, dev sdf, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478528.525533] Buffer I/O error on dev sdf, logical block 1081344, lost sync page write
[478528.525693] JBD2: Error -5 detected when updating journal superblock for sdf-8.
[478544.415585] sd 17:0:0:1: [sdd] tag#35 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478544.415589] sd 17:0:0:1: [sdd] tag#35 Sense Key : Medium Error [current]
[478544.415591] sd 17:0:0:1: [sdd] tag#35 Add. Sense: Unrecovered read error
[478544.415592] sd 17:0:0:1: [sdd] tag#35 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478544.415593] blk_update_request: critical medium error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478544.416217] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[478544.416461] EXT4-fs (sdd): I/O error while writing superblock
[478544.417605] EXT4-fs error (device sdd): ext4_journal_check_start:61: Detected aborted journal
[478544.419662] EXT4-fs (sdd): Remounting filesystem read-only
[478581.772011] sd 19:0:0:1: [sdf] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[478581.772015] sd 19:0:0:1: [sdf] tag#49 Sense Key : Medium Error [current]
[478581.772017] sd 19:0:0:1: [sdf] tag#49 Add. Sense: Unrecovered read error
[478581.772018] sd 19:0:0:1: [sdf] tag#49 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
[478581.772019] blk_update_request: critical medium error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[478581.772648] Buffer I/O error on dev sdf, logical block 0, lost sync page write
[478581.772885] EXT4-fs: 3 callbacks suppressed
[478581.772887] EXT4-fs (sdf): I/O error while writing superblock
[478581.773063] EXT4-fs error (device sdf): ext4_journal_check_start:61: Detected aborted journal
[478581.774506] EXT4-fs (sdf): Remounting filesystem read-only

Further questions

What's the NIC bandwidth between worker nodes?
How many volumes/replicas are in one node?
Are the replicas on NVMe SSDs?

25Gbit between nodes.
There's only 8 volumes, with 3 replicas totalling 646Gi reserved, and only 284Gi used.
The replicas are on the SSDs.

That said, i'm not convinced that this nfs-ganesha issue is the thing that is causing this instability. It could be, hence me reducing the number of recovery backends to 1 to see if it improves the situation.

It's probably a different issue, because issue 5183's caused by the out of sync between etch and the lister.
I would suggest you provide the information as I mentioned for figuring out the root cause.

In this supportbundle I had just rebooted 1 control-plane master that was exhibiting the issue and was listed as NotReady - The other two master nodes have not been rebooted.

Oooh. This looks interesting:
ganesha-2479[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
2023-04-05T15:46:59.273750831+01:00 05/04/2023 14:46:59 : epoch 642d89e3 : share-manager-pvc-e2a556a4-ce12-484d-a0b6-6232b17fe6c8 : nfs-ganesha-2479[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
2023-04-05T15:46:59.273757875+01:00 05/04/2023 14:46:59 : epoch 642d89e3 : share-manager-pvc-e2a556a4-ce12-484d-a0b6-6232b17fe6c8 : nfs-ganesha-2479[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection

Is nfs-ganesha not being given the permissions it needs re: DBUS?


Sentry and LexikJWT are creating fake issues

Hello,
I have Sentry and LexikJWT and each time a user try to login with this payload:
{"email":"","password":""}

It's creates a Sentry issue with this handler:
Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\BadRequestHttpException: "The key "email" must be a string." at /var/www/html/vendor/symfony/security-http/Authenticator/JsonLoginAuthenticator.php line 164

How can i prevent this ?

Looks like you should setup check_path property to any fake page

https://symfony.com/doc/current/reference/configuration/security.html#check-path


Checking if url is with dbFamily when getting active tasks

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Michal Borkowski seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Fix the hard-coded Node Agent TLS option

Fix the hard-coded Node Agent TLS option.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: lei-tang
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: incfly
If they are not already assigned, you can assign the PR to them by writing /assign @incfly in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

security/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Don't know how to stop copying this file.

[write something useful and descriptive here]
I tried to copy file using Mac's Finder. It's stuck in a copying state and Keybase constantly tells me that the write operation has failed, but nothing is working so far to stop it. Do I have to reboot?
my log id: b2e60c79e7846a77888daa1c

It seems that the Finder is stuck though, not Keybase.

Read operation has taken too much time too after the Finder is killed.

I'm still getting these errors.

We'll take a look, thank you....

I suspect the original login problem was never resolved.  Can you:
keybase logout
keybase login

From the command line?

I think this might be related https://github.com/keybase/client/issues/3709

Version 1.0.20 has architectural changes that should greatly improve the large write experience.  If you're still seeing problems, please open a new issue.


Migrate entire project to use SafeArgs in fragments instead of raw bundles

Describe the bug
Large parts of the app currently use raw bundles to access and pass arguments. This is unsafe and does not utilize the Navigation Architecture Component to its fullest extent.
Additional context
Since we already use the Navigation architecture component, it is easier to migrate to using SafeArgs. This will also be benficial if we decide to use Actions in the Nav AAC to switch between different screens.
Would you like to work on the issue?

[x] Yes

@iamareebjamal @liveHarshit Do you think this is worth pursuing?
I want to do this fragment by fragment, but I thought opening a new issue for every fragment is spamming.

Do you think this is worth pursuing?

Go ahead.

Okay, thanks!

I have completed the refactor of the entire project to safe args, and I want to do one more thing that would make things a lot better.
Earlier most of the keys used to retrieve data from fragment arguments were stored as completely capitalized const val fields at the top of each fragment. But now these are obsolete, because we don't need keys to access fragment args anymore. For example, if a fragment used const val EVENT_ID = "EVENT_ID" as a key, i added an argument to the nav graph with the name EVENT_ID.
There is a slight readability problem because of such completely capitalized arg keys, though. Now the SafeArgs gradle plugin generates classes with methods named like setEVENTID(...) etc.
I want to refactor the argument names to be in camel case so that they feel natural when accessed through kotlin code. I also want to put all their names in the strings.xml file (or another xml file called arguments.xml).
Should I go ahead and do this step too? @iamareebjamal @liveHarshit

Another quick question: Will multiple commits for this PR be okay? I currently have one commit for each fragment that I have refactored. Should I squash them?

because we don't need keys to access fragment args anymore.

Then remove these keys, if not used.

Another quick question: Will multiple commits for this PR be okay? I currently have one commit for each fragment that I have refactored. Should I squash them?

You can squash them later when PR is approved.

Another quick question: Will multiple commits for this PR be okay? I currently have one commit for each fragment that I have refactored. Should I squash them?

You can squash them later when PR is approved.

Cool. That works too. Multiple commits will make them easier to review as well.

because we don't need keys to access fragment args anymore.

Then remove these keys, if not used.

I also want to rename the arguments so that they feel natural when used in Kotlin/Java code. For example, change the name of EVENT_ID to eventId. Would that be fine?

And is moving these arg names to strings.xml a good idea?

IMO, not a good idea for getting these from string resources. And use as constant.

String resources should use for layout purposes.

Alright. I'm waiting on @iamareebjamal's review for the EventDetailsFragment safe args PR before I submit one for this issue.