Stable 3227.2.2 randomly causes processes to hang on I/O related operations

Description
We've seen multiple nodes (different regions and environments) stalling completely (unresponsive kubelets and containerds, journald and timesyncd units failing, no login possible, ...) on the 3227.2.2 release. This seems to be happening mostly on the reboot after the update, but we also had this occurring at random.
Impact
This causes Kubernetes nodes to become NotReady before being drained, which involves volumes not being able to be moved and therefore service interruptions.
Environment and steps to reproduce

Set-up: Flatcar Stable 3227.2.2, OpenStack, ESXi hypervisors
Task: on node start, but also during normal operation
Action(s): no clue if anything specific causes this
Error:

we get task blocked for more than 120 seconds errors and related call stacks, see the screenshot

CPU usage is very higher when the issue occurs
the journal gets corrupted when this issue occurs
rebooting once more, brings the node back
once the node is up again, partitions and filesystems appear healthy

Expected behavior
The nodes do not stall completely.
Additional information
We have the feeling that we may hit a kernel bug as we only see this on the 3227.2.2 release were basically only the kernel was changed. Do you have ideas how we can diagnose this further? Thanks.
cc @databus23 @defo89 @jknipper

Thanks for reporting, in https://www.flatcar.org/releases#release-3227.2.2 we have the links to the kernel changelogs for each bugfix release since the last Flatcar stable release. We need to find suspicious changes and try to see if the problem goes away if they are reverted.
Do you have an easy reproducer with https://stable.release.flatcar-linux.net/amd64-usr/3227.2.2/flatcar_production_qemu.sh and the qemu image?

Could not reproduce it using QEMU so far. I tried plenty of reboots and letting the VM ran for while using one of our ignition templates. Also starting with 3227.2.1 and updating did not trigger it. :(

We think we're having this same problem. It seems to manifest most often as sshd hanging. We'll keep a test server running things and hopefully get some idea about what's going on, but probably can't help with a reproduction case if QEMU isn't working.

We are still seeing lock-ups of nodes on a constant basis in our fleet but fail to reproduce the error some what consistently. Affected nodes are hanging, login via ssh or console is just haging. We are always seeing stack traces of "hung task timeouts" that are io/fs related.

The main node we see hangs on has several cephfs mounts. This kernel module has been problematic in the past: does anyone else with this problem use cephfs?

We experienced the issue with ext4 on vmware disks.

my company see's these issues aswell under vmware and 3227.2.2 will be ready to provide more info if necessary.

The servers in our company have the same symptoms after an update to 3227.2.2.
We can reproduce it as soon as we generate a lot of IO on the hard disk.
Unfortunately I don't see any errors in systemd / kernel ring buffer. Only
File /var/log/journal/<some-id>/system.journal corrupted or uncleanly shut down, renaming and replacing.
shortly after that the system stops responding.
As a temporary solution we did a rollback to 3227.2.1 for now.

We can reproduce it as soon as we generate a lot of IO on the hard disk.

Any specific you executed for that? I shuffled around some 100 GB's using dd and could not trigger the issue.

I could reproduce hung task errors on bare metal  by running stress -i 10 -d 5 within a pod's emptyDir, backed by the host's SSD Raid1 after a while. I didn't crosscheck against the previous version so far.
No further negative consequences, esp. after shutting down stress. No more errors AFAICS and load is still usual.
[291596.784816] INFO: task stress:3125869 blocked for more than 123 seconds.
[291596.792511]       Not tainted 5.15.63-flatcar #1
[291596.797871] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[291596.806839] task:stress          state:D stack:    0 pid:3125869 ppid:3125856 flags:0x00004000
[291596.816707] Call Trace:
[291596.819630]  <TASK>
[291596.822164]  __schedule+0x2eb/0x8d0
[291596.826258]  schedule+0x5b/0xd0
[291596.829957]  rwsem_down_write_slowpath+0x220/0x4f0
[291596.835508]  ? wb_wait_for_completion+0x25/0x80
[291596.840776]  sync_inodes_sb+0xba/0x2a0
[291596.845159]  ? __x64_sys_tee+0xd0/0xd0
[291596.849542]  iterate_supers+0x9f/0x100
[291596.853926]  ksys_sync+0x40/0xa0
[291596.857729]  __do_sys_sync+0xa/0x20
[291596.861818]  do_syscall_64+0x3b/0x90
[291596.866008]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[291596.871853] RIP: 0033:0x7f9de9d77b17
[291596.876042] RSP: 002b:00007ffe1b4ffc48 EFLAGS: 00000217 ORIG_RAX:<PHONE_NUMBER>0000a2
[291596.884722] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>00000c RCX: 00007f9de9d77b17
[291596.892903] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>000000
[291596.901101] RBP:<PHONE_NUMBER>000000 R08:<PHONE_NUMBER>000000 R09: 00007f9de9c7f740
[291596.909292] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000217 R12:<PHONE_NUMBER>002328
[291596.917488] R13:<PHONE_NUMBER>000000 R14:<PHONE_NUMBER>000000 R15:<PHONE_NUMBER>000003
[291596.925684]  </TASK>

We tried to reproduce the case again today in our test cluster. We installed one node with flatcar version 3227.2.1 and the other with 3227.2.2.
Both nodes have the following hardware

128 GB DDR4 ECC
2 x 1.92 TB NVME
AMD Ryzen 9 3900 (24 cores)
K8S v1.24.5

We have used dd, stress as well as stress-ng, unsuccessfully! We had a high load, but the 3227.2.2 nodes did not crash.
Then we crafted the following K8S resource with a small Golang tool. And this actually managed to crash the 3227.2.2 node after about 5 minutes. SSH and other services were no longer accessible.
On the 3227.2.1 node the script ran until the filesystem reported full inodes, but it was still reachable.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: io-test
  labels:
    app: io-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: io-test
  template:
    metadata:
      labels:
        app: io-test
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - <select-a-node>
      containers:
        - name: io-test
          image: golang:1.19.1
          command:
          - bash
          - -c
          - |
            apt update && apt install time -y
            echo '
              package main

              import (
                "fmt"
                "log"
                "os"
              )

              func main() {
                suffix := ""
                if len(os.Args) > 1 {
                  suffix = os.Args[1]
                }

                err := run(suffix)
                if err != nil {
                  log.Fatal(err)
                }
              }

              func run(suffix string) error {
                for i := 0; i < 100_000; i++ {
                  if i%10_000 == 0 {
                    log.Printf("file #%d", i)
                  }

                  path := fmt.Sprintf("file_%d_%s", i, suffix)
                  content := path
                  err := os.WriteFile(path, []byte(content), 0644)
                  if err != nil {
                    return err
                  }
                }

                return nil
              }
            ' >> /test/main.go
            cd /test
            go build main.go

            while true
            do
              time ./main $(date +%s)
            done
          volumeMounts:
          - name: io-test
            mountPath: "/test"
      restartPolicy: Always
      volumes:
        - name: io-test
          emptyDir: {}

When the 3227.2.2 node crashed, the kubelet process was marked as a zombie and in the kernel ring buffer we saw the following errors
[2091.827035] INFO: task jbd2/nvme0n1p9-:755 blocked for more than 122 seconds.
[ 2091.834299]       Not tainted 5.15.63-flatcar #1
[ 2091.839081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2091.847099] task:jbd2/nvme0n1p9- state:D stack:    0 pid:  755 ppid:     2 flags:0x00004000
[ 2091.855634] Call Trace:
[ 2091.858241]  <TASK>
[ 2091.860505]  __schedule+0x2eb/0x8d0
[ 2091.864153]  schedule+0x5b/0xd0
[ 2091.867456]  jbd2_journal_commit_transaction+0x301/0x2850 [jbd2]
[ 2091.873628]  ? sugov_update_single_freq+0xb9/0x190
[ 2091.878582]  ? sugov_exit+0xa0/0xa0
[ 2091.882229]  ? wait_woken+0x70/0x70
[ 2091.885878]  ? lock_timer_base+0x61/0x80
[ 2091.889962]  jbd2_journal_check_available_features+0x1ab/0x3f0 [jbd2]
[ 2091.896565]  ? wait_woken+0x70/0x70
[ 2091.900214]  ? jbd2_journal_check_available_features+0x100/0x3f0 [jbd2]
[ 2091.906982]  kthread+0x127/0x150
[ 2091.910379]  ? set_kthread_struct+0x50/0x50
[ 2091.914720]  ret_from_fork+0x22/0x30
[ 2091.918455]  </TASK>
[ 2091.920803] INFO: task systemd-journal:988 blocked for more than 122 seconds.
[ 2091.928104]       Not tainted 5.15.63-flatcar #1
[ 2091.932876] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2091.940891] task:systemd-journal state:D stack:    0 pid:  988 ppid:     1 flags:0x00000224
[ 2091.949428] Call Trace:
[ 2091.952038]  <TASK>
[ 2091.954301]  __schedule+0x2eb/0x8d0
[ 2091.957949]  schedule+0x5b/0xd0
[ 2091.961260]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2091.966815]  ? wait_woken+0x70/0x70
[ 2091.970461]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2091.976104]  ? call_rcu+0xa2/0x330
[ 2091.979667]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2091.985311]  ? step_into+0x47c/0x7b0
[ 2091.989052]  ? __cond_resched+0x16/0x50
[ 2091.993047]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2091.997998]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.003302]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.007902]  __mark_inode_dirty+0x147/0x320
[ 2092.012255]  touch_atime+0x13c/0x150
[ 2092.015991]  filemap_read+0x308/0x320
[ 2092.019810]  ? path_lookupat+0xa2/0x1c0
[ 2092.023815]  new_sync_read+0x119/0x1b0
[ 2092.027735]  ? 0xffffffffb6000000
[ 2092.031207]  vfs_read+0xf6/0x190
[ 2092.034596]  __x64_sys_pread64+0x91/0xc0
[ 2092.038678]  do_syscall_64+0x3b/0x90
[ 2092.042413]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.047622] RIP: 0033:0x7f7b5d090b17
[ 2092.051356] RSP: 002b:00007ffc6dedc4c0 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000011
[ 2092.059113] RAX: ffffffffffffffda RBX: 00007ffc6dedc570 RCX: 00007f7b5d090b17
[ 2092.066401] RDX:<PHONE_NUMBER>000040 RSI: 00007ffc6dedc4f0 RDI:<PHONE_NUMBER>000018
[ 2092.073701] RBP:<PHONE_NUMBER>f25b78 R08:<PHONE_NUMBER>000000 R09: 00007ffc6dedc700
[ 2092.080998] R10:<PHONE_NUMBER>f25b78 R11:<PHONE_NUMBER>000293 R12: 000055d55b175310
[ 2092.088284] R13:<PHONE_NUMBER>000000 R14: 00007ffc6dedc4f0 R15: 000055d55b175310
[ 2092.095576]  </TASK>
[ 2092.097944] INFO: task systemd-timesyn:1140 blocked for more than 123 seconds.
[ 2092.105360]       Not tainted 5.15.63-flatcar #1
[ 2092.110132] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2092.118139] task:systemd-timesyn state:D stack:    0 pid: 1140 ppid:     1 flags:0x00000220
[ 2092.126678] Call Trace:
[ 2092.129287]  <TASK>
[ 2092.131546]  __schedule+0x2eb/0x8d0
[ 2092.135199]  schedule+0x5b/0xd0
[ 2092.138500]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2092.144063]  ? wait_woken+0x70/0x70
[ 2092.147721]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2092.153361]  ? __fget_files+0x79/0xb0
[ 2092.157193]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2092.162846]  ? nd_jump_link+0x4d/0xc0
[ 2092.166681]  ? __cond_resched+0x16/0x50
[ 2092.170677]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2092.175627]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.180921]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.185533]  __mark_inode_dirty+0x147/0x320
[ 2092.189877]  ext4_setattr+0x1fb/0x9a0 [ext4]
[ 2092.194311]  notify_change+0x3c4/0x540
[ 2092.198221]  ? vfs_utimes+0x139/0x220
[ 2092.202043]  vfs_utimes+0x139/0x220
[ 2092.205692]  do_utimes+0xb4/0x120
[ 2092.209165]  __x64_sys_utimensat+0x70/0xb0
[ 2092.213431]  ? syscall_trace_enter.constprop.0+0x143/0x1c0
[ 2092.219073]  do_syscall_64+0x3b/0x90
[ 2092.222815]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.228036] RIP: 0033:0x7f26ccdc801f
[ 2092.231770] RSP: 002b:00007ffc7da41a98 EFLAGS: 00000206 ORIG_RAX:<PHONE_NUMBER>000118
[ 2092.239533] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX: 00007f26ccdc801f
[ 2092.246824] RDX:<PHONE_NUMBER>000000 RSI: 00007ffc7da41aa0 RDI: 00000000ffffff9c
[ 2092.254130] RBP: 00007ffc7da41aa0 R08:<PHONE_NUMBER>000000 R09: 00007ffc7da41930
[ 2092.261427] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000206 R12:<PHONE_NUMBER>000000
[ 2092.268715] R13: 00000000ffffffff R14: ffffffffffffffff R15: 00000000ffffffff
[ 2092.276013]  </TASK>
[ 2092.280018] INFO: task containerd:1257 blocked for more than 123 seconds.
[ 2092.286959]       Not tainted 5.15.63-flatcar #1
[ 2092.291735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2092.299750] task:containerd      state:D stack:    0 pid: 1257 ppid:     1 flags:0x00000000
[ 2092.308287] Call Trace:
[ 2092.310903]  <TASK>
[ 2092.313175]  __schedule+0x2eb/0x8d0
[ 2092.316821]  ? newidle_balance+0x12a/0x400
[ 2092.321086]  schedule+0x5b/0xd0
[ 2092.324399]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2092.329962]  ? wait_woken+0x70/0x70
[ 2092.333617]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2092.339260]  ? select_task_rq_fair+0x130/0xf90
[ 2092.343871]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2092.349531]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2092.354056]  ? __cond_resched+0x16/0x50
[ 2092.358056]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2092.363007]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.368311]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.372920]  __mark_inode_dirty+0x147/0x320
[ 2092.377274]  generic_update_time+0x6c/0xd0
[ 2092.381527]  file_update_time+0x127/0x140
[ 2092.385704]  ? generic_write_checks+0x61/0xc0
[ 2092.390229]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2092.394666]  new_sync_write+0x11c/0x1b0
[ 2092.398660]  ? intel_get_event_constraints+0x300/0x390
[ 2092.403964]  vfs_write+0x1de/0x270
[ 2092.407536]  ksys_write+0x5f/0xe0
[ 2092.411010]  do_syscall_64+0x3b/0x90
[ 2092.414746]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.419954] RIP: 0033:0x55d3b77a2edb
[ 2092.423698] RSP: 002b:000000c0005228d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2092.431454] RAX: ffffffffffffffda RBX: 000000c000054000 RCX: 000055d3b77a2edb
[ 2092.438751] RDX:<PHONE_NUMBER>000117 RSI: 000000c002fe3500 RDI:<PHONE_NUMBER>000255
[ 2092.446053] RBP: 000000c000522920 R08:<PHONE_NUMBER>000201 R09: 000000c003776120
[ 2092.453346] R10:<PHONE_NUMBER>000117 R11:<PHONE_NUMBER>000202 R12: 000000c0005229b0
[ 2092.460644] R13:<PHONE_NUMBER>000000 R14: 000000c002c9cea0 R15:<PHONE_NUMBER>000032
[ 2092.467945]  </TASK>
[ 2092.470307] INFO: task containerd:4089 blocked for more than 123 seconds.
[ 2092.477260]       Not tainted 5.15.63-flatcar #1
[ 2092.482042] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2092.490059] task:containerd      state:D stack:    0 pid: 4089 ppid:     1 flags:0x00000000
[ 2092.498597] Call Trace:
[ 2092.501212]  <TASK>
[ 2092.503475]  __schedule+0x2eb/0x8d0
[ 2092.507131]  schedule+0x5b/0xd0
[ 2092.510437]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2092.515997]  ? wait_woken+0x70/0x70
[ 2092.519647]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2092.525289]  ? select_task_rq_fair+0x130/0xf90
[ 2092.529900]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2092.535550]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2092.540073]  ? __cond_resched+0x16/0x50
[ 2092.544068]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2092.549019]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.554321]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.558932]  __mark_inode_dirty+0x147/0x320
[ 2092.563282]  generic_update_time+0x6c/0xd0
[ 2092.567537]  file_update_time+0x127/0x140
[ 2092.571708]  ? generic_write_checks+0x61/0xc0
[ 2092.576233]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2092.580668]  new_sync_write+0x11c/0x1b0
[ 2092.584672]  ? intel_get_event_constraints+0x300/0x390
[ 2092.589977]  vfs_write+0x1de/0x270
[ 2092.593539]  ksys_write+0x5f/0xe0
[ 2092.597015]  do_syscall_64+0x3b/0x90
[ 2092.600747]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.605957] RIP: 0033:0x55d3b77a2edb
[ 2092.609701] RSP: 002b:000000c0025e18d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2092.617456] RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 000055d3b77a2edb
[ 2092.624747] RDX:<PHONE_NUMBER>0000ac RSI: 000000c000b760c0 RDI:<PHONE_NUMBER>0000d2
[ 2092.632044] RBP: 000000c0025e1920 R08:<PHONE_NUMBER>000001 R09: 000000c001ca3c80
[ 2092.639343] R10:<PHONE_NUMBER>0000ac R11:<PHONE_NUMBER>000202 R12: 000000c0025e19b0
[ 2092.646639] R13:<PHONE_NUMBER>000000 R14: 000000c001504d00 R15:<PHONE_NUMBER>000035
[ 2092.653938]  </TASK>
[ 2092.656293] INFO: task containerd:6560 blocked for more than 123 seconds.
[ 2092.663246]       Not tainted 5.15.63-flatcar #1
[ 2092.668027] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2092.676044] task:containerd      state:D stack:    0 pid: 6560 ppid:     1 flags:0x00000000
[ 2092.684581] Call Trace:
[ 2092.687188]  <TASK>
[ 2092.689450]  __schedule+0x2eb/0x8d0
[ 2092.693100]  ? newidle_balance+0x12a/0x400
[ 2092.697354]  schedule+0x5b/0xd0
[ 2092.700659]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2092.706224]  ? wait_woken+0x70/0x70
[ 2092.709870]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2092.715521]  ? select_task_rq_fair+0x130/0xf90
[ 2092.720124]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2092.725773]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2092.730288]  ? __cond_resched+0x16/0x50
[ 2092.734284]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2092.739231]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.744539]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.749148]  __mark_inode_dirty+0x147/0x320
[ 2092.753500]  generic_update_time+0x6c/0xd0
[ 2092.757754]  file_update_time+0x127/0x140
[ 2092.761931]  ? generic_write_checks+0x61/0xc0
[ 2092.766456]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2092.770895]  new_sync_write+0x11c/0x1b0
[ 2092.774897]  ? intel_get_event_constraints+0x300/0x390
[ 2092.780199]  vfs_write+0x1de/0x270
[ 2092.783763]  ksys_write+0x5f/0xe0
[ 2092.787245]  do_syscall_64+0x3b/0x90
[ 2092.790981]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.796200] RIP: 0033:0x55d3b77a2edb
[ 2092.799942] RSP: 002b:000000c00237a8d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2092.807698] RAX: ffffffffffffffda RBX: 000000c00004f000 RCX: 000055d3b77a2edb
[ 2092.814989] RDX:<PHONE_NUMBER>000097 RSI: 000000c001e1c540 RDI:<PHONE_NUMBER>000227
[ 2092.822287] RBP: 000000c00237a920 R08:<PHONE_NUMBER>000201 R09: 000000c0019f21e0
[ 2092.829581] R10:<PHONE_NUMBER>000097 R11:<PHONE_NUMBER>000202 R12: 000000c00237a9b0
[ 2092.836881] R13:<PHONE_NUMBER>000000 R14: 000000c001cc8d00 R15:<PHONE_NUMBER>000034
[ 2092.844177]  </TASK>
[ 2092.846531] INFO: task containerd:7559 blocked for more than 123 seconds.
[ 2092.853470]       Not tainted 5.15.63-flatcar #1
[ 2092.858243] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2092.866258] task:containerd      state:D stack:    0 pid: 7559 ppid:     1 flags:0x00000000
[ 2092.874794] Call Trace:
[ 2092.877406]  <TASK>
[ 2092.879666]  __schedule+0x2eb/0x8d0
[ 2092.883314]  ? newidle_balance+0x12a/0x400
[ 2092.887569]  schedule+0x5b/0xd0
[ 2092.890873]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2092.896436]  ? wait_woken+0x70/0x70
[ 2092.900085]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2092.905736]  ? select_task_rq_fair+0x130/0xf90
[ 2092.910347]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2092.915997]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2092.920521]  ? __cond_resched+0x16/0x50
[ 2092.924524]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2092.929473]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2092.934777]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2092.939386]  __mark_inode_dirty+0x147/0x320
[ 2092.943741]  generic_update_time+0x6c/0xd0
[ 2092.947995]  file_update_time+0x127/0x140
[ 2092.952170]  ? generic_write_checks+0x61/0xc0
[ 2092.956693]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2092.961124]  new_sync_write+0x11c/0x1b0
[ 2092.965129]  ? intel_get_event_constraints+0x300/0x390
[ 2092.970424]  vfs_write+0x1de/0x270
[ 2092.973984]  ksys_write+0x5f/0xe0
[ 2092.977459]  do_syscall_64+0x3b/0x90
[ 2092.981195]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2092.986412] RIP: 0033:0x55d3b77a2edb
[ 2092.990148] RSP: 002b:000000c00285c8d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2092.997903] RAX: ffffffffffffffda RBX: 000000c000074800 RCX: 000055d3b77a2edb
[ 2093.005203] RDX:<PHONE_NUMBER>000088 RSI: 000000c000a13500 RDI:<PHONE_NUMBER>000236
[ 2093.012502] RBP: 000000c00285c920 R08:<PHONE_NUMBER>000201 R09: 000000c002024f00
[ 2093.019796] R10: 00007ffc79f7e080 R11:<PHONE_NUMBER>000202 R12: 000000c00285c810
[ 2093.027095] R13:<PHONE_NUMBER>000000 R14: 000000c001d89520 R15:<PHONE_NUMBER>00002f
[ 2093.034390]  </TASK>
[ 2093.036749] INFO: task containerd:7563 blocked for more than 124 seconds.
[ 2093.043690]       Not tainted 5.15.63-flatcar #1
[ 2093.048463] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2093.056472] task:containerd      state:D stack:    0 pid: 7563 ppid:     1 flags:0x00000000
[ 2093.065011] Call Trace:
[ 2093.067619]  <TASK>
[ 2093.069881]  __schedule+0x2eb/0x8d0
[ 2093.073537]  ? newidle_balance+0x12a/0x400
[ 2093.077793]  schedule+0x5b/0xd0
[ 2093.081095]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2093.086651]  ? wait_woken+0x70/0x70
[ 2093.090308]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2093.095959]  ? select_task_rq_fair+0x130/0xf90
[ 2093.100569]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2093.106209]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2093.112364]  ? __cond_resched+0x16/0x50
[ 2093.116360]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2093.121309]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2093.126613]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2093.131214]  __mark_inode_dirty+0x147/0x320
[ 2093.135557]  generic_update_time+0x6c/0xd0
[ 2093.139810]  file_update_time+0x127/0x140
[ 2093.143979]  ? generic_write_checks+0x61/0xc0
[ 2093.148494]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2093.152923]  new_sync_write+0x11c/0x1b0
[ 2093.156920]  ? intel_get_event_constraints+0x300/0x390
[ 2093.162222]  vfs_write+0x1de/0x270
[ 2093.165785]  ksys_write+0x5f/0xe0
[ 2093.169267]  do_syscall_64+0x3b/0x90
[ 2093.173004]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2093.178213] RIP: 0033:0x55d3b77a2edb
[ 2093.181947] RSP: 002b:000000c0027fa8d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2093.189704] RAX: ffffffffffffffda RBX: 000000c000074800 RCX: 000055d3b77a2edb
[ 2093.196997] RDX:<PHONE_NUMBER>0001f5 RSI: 000000c0001e22c0 RDI:<PHONE_NUMBER>00013b
[ 2093.204289] RBP: 000000c0027fa920 R08:<PHONE_NUMBER>000101 R09: 000000c003288360
[ 2093.211592] R10:<PHONE_NUMBER>0001f5 R11:<PHONE_NUMBER>000202 R12: 000000c0027fa9b0
[ 2093.218875] R13:<PHONE_NUMBER>000000 R14: 000000c00117c820 R15:<PHONE_NUMBER>00002f
[ 2093.226176]  </TASK>
[ 2093.228524] INFO: task containerd:9227 blocked for more than 124 seconds.
[ 2093.235475]       Not tainted 5.15.63-flatcar #1
[ 2093.240257] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2093.248273] task:containerd      state:D stack:    0 pid: 9227 ppid:     1 flags:0x00000000
[ 2093.256810] Call Trace:
[ 2093.259420]  <TASK>
[ 2093.261682]  __schedule+0x2eb/0x8d0
[ 2093.265330]  ? newidle_balance+0x12a/0x400
[ 2093.269586]  schedule+0x5b/0xd0
[ 2093.272886]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2093.278451]  ? wait_woken+0x70/0x70
[ 2093.282099]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2093.287740]  ? cpumask_next+0x1e/0x30
[ 2093.291562]  ? select_task_rq_fair+0x130/0xf90
[ 2093.296165]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2093.301809]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2093.306331]  ? __cond_resched+0x16/0x50
[ 2093.310327]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2093.315276]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2093.320579]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2093.325192]  __mark_inode_dirty+0x147/0x320
[ 2093.329530]  generic_update_time+0x6c/0xd0
[ 2093.333789]  file_update_time+0x127/0x140
[ 2093.337965]  ? generic_write_checks+0x61/0xc0
[ 2093.342488]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2093.346917]  new_sync_write+0x11c/0x1b0
[ 2093.350921]  ? intel_get_event_constraints+0x300/0x390
[ 2093.356224]  vfs_write+0x1de/0x270
[ 2093.359788]  ksys_write+0x5f/0xe0
[ 2093.363270]  do_syscall_64+0x3b/0x90
[ 2093.367007]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2093.372215] RIP: 0033:0x55d3b77a2edb
[ 2093.375949] RSP: 002b:000000c000fa88d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2093.383696] RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 000055d3b77a2edb
[ 2093.390987] RDX:<PHONE_NUMBER>000153 RSI: 000000c000e96000 RDI:<PHONE_NUMBER>00003f
[ 2093.398276] RBP: 000000c000fa8920 R08:<PHONE_NUMBER>000001 R09: 000000c00084d3e0
[ 2093.405571] R10:<PHONE_NUMBER>000153 R11:<PHONE_NUMBER>000202 R12: 000000c000fa89b0
[ 2093.412860] R13:<PHONE_NUMBER>000000 R14: 000000c001a471e0 R15:<PHONE_NUMBER>000031
[ 2093.420160]  </TASK>
[ 2093.422514] INFO: task containerd:9229 blocked for more than 124 seconds.
[ 2093.429459]       Not tainted 5.15.63-flatcar #1
[ 2093.434232] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2093.442248] task:containerd      state:D stack:    0 pid: 9229 ppid:     1 flags:0x00000000
[ 2093.450785] Call Trace:
[ 2093.453406]  <TASK>
[ 2093.455664]  __schedule+0x2eb/0x8d0
[ 2093.459315]  schedule+0x5b/0xd0
[ 2093.462617]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[ 2093.468171]  ? wait_woken+0x70/0x70
[ 2093.471819]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[ 2093.477469]  ? select_task_rq_fair+0x130/0xf90
[ 2093.482073]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[ 2093.487726]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2093.492247]  ? __cond_resched+0x16/0x50
[ 2093.496244]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 2093.501200]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 2093.506504]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 2093.511114]  __mark_inode_dirty+0x147/0x320
[ 2093.515458]  generic_update_time+0x6c/0xd0
[ 2093.519714]  file_update_time+0x127/0x140
[ 2093.523879]  ? generic_write_checks+0x61/0xc0
[ 2093.528403]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[ 2093.532832]  new_sync_write+0x11c/0x1b0
[ 2093.536836]  ? intel_get_event_constraints+0x300/0x390
[ 2093.542140]  vfs_write+0x1de/0x270
[ 2093.545702]  ksys_write+0x5f/0xe0
[ 2093.549176]  do_syscall_64+0x3b/0x90
[ 2093.552913]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 2093.558133] RIP: 0033:0x55d3b77a2edb
[ 2093.561864] RSP: 002b:000000c00245f8d0 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000001
[ 2093.569624] RAX: ffffffffffffffda RBX: 000000c000054000 RCX: 000055d3b77a2edb
[ 2093.576918] RDX:<PHONE_NUMBER>001158 RSI: 000000c0013a9900 RDI:<PHONE_NUMBER>0002e3
[ 2093.584221] RBP: 000000c00245f920 R08:<PHONE_NUMBER>000201 R09: 000000c002d243c0
[ 2093.591506] R10:<PHONE_NUMBER>001158 R11:<PHONE_NUMBER>000202 R12: 000000c00245f9b0
[ 2093.598793] R13:<PHONE_NUMBER>000000 R14: 000000c004509860 R15:<PHONE_NUMBER>000033
[ 2093.606086]  </TASK>

cc (@damoon)

We are also experiencing this problem but on 3227.2.1 that we have updated some packages of. On 3227.2.1 the only updates we made were containerd 1.6.8 and kernel 5.15.62.
When this issues occurs, our CPU metrics show that containerd jumps to 90% utilisation of the node's CPU and some services such as SSHD stop responding. As stated in previous replies the node becomes entirely unresponsive and kubernetes reports the node as unreachable. The node doesnt recover unless force restarted or replaced by a cloud provider.
I have tried downgrading the containerd version to 1.6.6 (which worked fine for us on a previous flatcar version), and I have also tried upgrading the kernel to 5.15.68, both of which did not resolve the problem. Comparing the kernel versions of upstream 3227.2.1 and our current stable flatcar version (3139.2.3) I see 5.15.58 and 5.15.54 respectively. So potentially some kernel version after this is causing the issue?

@Champ-Goblem since you have decent testing capabilities, would you mind also reproducing this with 5.15.70 kernel? Looks that it will be in the next stable release https://github.com/flatcar/coreos-overlay/commit/ea82f1873da310245cbb174ac9fa6f9908c493c6

I was not able to repro with the reproducer in my own testing, but an infrastructure VM that we have on the stable channel has also hit this (14 days of uptime, no particular io load):

log:
```
[1282119.153912] INFO: task jbd2/sda9-8:544 blocked for more than 122 seconds.
[1282119.157088]       Not tainted 5.15.63-flatcar #1
[1282119.159281] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.162723] task:jbd2/sda9-8     state:D stack:    0 pid:  544 ppid:     2 flags:0x00004000
[1282119.166441] Call Trace:
[1282119.167640]
[1282119.168675]  __schedule+0x2eb/0x8d0
[1282119.170341]  schedule+0x5b/0xd0
[1282119.171806]  jbd2_journal_commit_transaction+0x301/0x2850 [jbd2]
[1282119.175448]  ? wait_woken+0x70/0x70
[1282119.177174]  ? lock_timer_base+0x61/0x80
[1282119.179015]  jbd2_journal_check_available_features+0x1ab/0x3f0 [jbd2]
[1282119.181922]  ? wait_woken+0x70/0x70
[1282119.183533]  ? jbd2_journal_check_available_features+0x100/0x3f0 [jbd2]
[1282119.186566]  kthread+0x127/0x150
[1282119.188087]  ? set_kthread_struct+0x50/0x50
[1282119.190346]  ret_from_fork+0x22/0x30
[1282119.192027]
[1282119.193081] INFO: task systemd-journal:748 blocked for more than 122 seconds.
[1282119.196255]       Not tainted 5.15.63-flatcar #1
[1282119.198321] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.201776] task:systemd-journal state:D stack:    0 pid:  748 ppid:     1 flags:0x00000004
[1282119.205604] Call Trace:
[1282119.206773]
[1282119.207794]  __schedule+0x2eb/0x8d0
[1282119.209410]  schedule+0x5b/0xd0
[1282119.210887]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.213342]  ? wait_woken+0x70/0x70
[1282119.214946]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.217482]  ? call_rcu+0xa2/0x330
[1282119.219070]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.221672]  ? step_into+0x47c/0x7b0
[1282119.223372]  ? __cond_resched+0x16/0x50
[1282119.225175]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.227342]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.229688]  ext4_dirty_inode+0x35/0x80 [ext4]
[1282119.231735]  __mark_inode_dirty+0x147/0x320
[1282119.233712]  touch_atime+0x13c/0x150
[1282119.235429]  filemap_read+0x308/0x320
[1282119.238370]  ? may_delete+0x2a0/0x2f0
[1282119.240286]  ? do_filp_open+0xa9/0x150
[1282119.242071]  new_sync_read+0x119/0x1b0
[1282119.244052]  ? 0xffffffffad000000
[1282119.245736]  vfs_read+0xf6/0x190
[1282119.247362]  __x64_sys_pread64+0x91/0xc0
[1282119.249365]  do_syscall_64+0x3b/0x90
[1282119.251173]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[1282119.253772] RIP: 0033:0x7f59ebb32b17
[1282119.255634] RSP: 002b:00007ffefe2b6520 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000011
[1282119.259426] RAX: ffffffffffffffda RBX: 00007ffefe2b65d0 RCX: 00007f59ebb32b17
[1282119.262876] RDX:<PHONE_NUMBER>000040 RSI: 00007ffefe2b6550 RDI:<PHONE_NUMBER>000020
[1282119.266269] RBP:<PHONE_NUMBER>10fdb0 R08:<PHONE_NUMBER>000000 R09: 00007ffefe2b6760
[1282119.270866] R10:<PHONE_NUMBER>10fdb0 R11:<PHONE_NUMBER>000293 R12: 000055705e025030
[1282119.274357] R13:<PHONE_NUMBER>000000 R14: 00007ffefe2b6550 R15: 000055705e025030
[1282119.277808]
[1282119.279102] INFO: task python:1117 blocked for more than 123 seconds.
[1282119.282317]       Not tainted 5.15.63-flatcar #1
[1282119.284531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.288819] task:python          state:D stack:    0 pid: 1117 ppid:  1010 flags:0x00000000
[1282119.292823] Call Trace:
[1282119.294282]
[1282119.295429]  __schedule+0x2eb/0x8d0
[1282119.297126]  schedule+0x5b/0xd0
[1282119.298734]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.301713]  ? wait_woken+0x70/0x70
[1282119.303438]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.306313]  ? hrtimer_try_to_cancel.part.0+0x50/0xd0
[1282119.308836]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.311496]  ? mlx5e_select_queue+0x3c/0x2d0 [mlx5_core]
[1282119.314170]  ? __cond_resched+0x16/0x50
[1282119.315956]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.318816]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.321540]  ext4_setattr+0x3a8/0x9a0 [ext4]
[1282119.323842]  notify_change+0x3c4/0x540
[1282119.325750]  ? ext4_es_delayed_clu+0x170/0x430 [ext4]
[1282119.328375]  ? ext4_es_delayed_clu+0x1ef/0x430 [ext4]
[1282119.330894]  ? do_truncate+0x7d/0xd0
[1282119.332698]  do_truncate+0x7d/0xd0
[1282119.334571]  path_openat+0x24d/0x1280
[1282119.336461]  do_filp_open+0xa9/0x150
[1282119.338190]  ? __check_object_size+0x146/0x160
[1282119.340409]  do_sys_openat2+0x9b/0x160
[1282119.342289]  __x64_sys_openat+0x54/0xa0
[1282119.344180]  do_syscall_64+0x3b/0x90
[1282119.346050]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[1282119.348496] RIP: 0033:0x7f47adf699e4
[1282119.352234] RSP: 002b:00007ffd95f911e0 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000101
[1282119.357348] RAX: ffffffffffffffda RBX: 00007f47ac34a558 RCX: 00007f47adf699e4
[1282119.360865] RDX:<PHONE_NUMBER>080241 RSI: 00007f47acb90b48 RDI: 00000000ffffff9c
[1282119.364381] RBP: 00007f47acb90b48 R08:<PHONE_NUMBER>000000 R09: 0000561f71986760
[1282119.369496] R10:<PHONE_NUMBER>0001b6 R11:<PHONE_NUMBER>000293 R12:<PHONE_NUMBER>080241
[1282119.374035] R13: 00007f47acce1a80 R14: 00007f47adc43248 R15:<PHONE_NUMBER>000000
[1282119.377852]
[1282119.379101] INFO: task VM Periodic Tas:11249 blocked for more than 123 seconds.
[1282119.383214]       Not tainted 5.15.63-flatcar #1
[1282119.385459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.389306] task:VM Periodic Tas state:D stack:    0 pid:11249 ppid:  8007 flags:0x00000000
[1282119.393418] Call Trace:
[1282119.394722]
[1282119.395852]  __schedule+0x2eb/0x8d0
[1282119.397882]  schedule+0x5b/0xd0
[1282119.399608]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.402324]  ? wait_woken+0x70/0x70
[1282119.404155]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.406854]  ? update_load_avg+0x7a/0x5e0
[1282119.408822]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.411679]  ? psi_task_switch+0x1e0/0x200
[1282119.414387]  ? __cond_resched+0x16/0x50
[1282119.416760]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.419236]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.421878]  ext4_dirty_inode+0x35/0x80 [ext4]
[1282119.424391]  __mark_inode_dirty+0x147/0x320
[1282119.426430]  generic_update_time+0x6c/0xd0
[1282119.428570]  file_update_time+0x127/0x140
[1282119.430744]  ext4_page_mkwrite+0x86/0xc50 [ext4]
[1282119.433058]  do_page_mkwrite+0x55/0xc0
[1282119.434861]  do_wp_page+0x20a/0x300
[1282119.436643]  ? task_tick_fair+0x7c/0x370
[1282119.438526]  __handle_mm_fault+0xb7a/0x1470
[1282119.440623]  handle_mm_fault+0xcf/0x2b0
[1282119.442491]  do_user_addr_fault+0x1be/0x670
[1282119.444588]  exc_page_fault+0x68/0x140
[1282119.446694]  asm_exc_page_fault+0x21/0x30
[1282119.448720] RIP: 0033:0x7f08a7c0b29e
[1282119.450453] RSP: 002b:00007f0879ffed30 EFLAGS: 00010206
[1282119.453058] RAX: 00048c1340635bb2 RBX: 00007f08a66851b8 RCX:<PHONE_NUMBER>000018
[1282119.456630] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>138faa RDI:<PHONE_NUMBER>000001
[1282119.460183] RBP: 00007f0879ffed40 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000032
[1282119.467273] R10: 00007ffe86f54080 R11: 00007ffe86f54090 R12:<PHONE_NUMBER>000008
[1282119.470823] R13: 00007f08a0303b40 R14:<PHONE_NUMBER>000002 R15:<PHONE_NUMBER>000001
[1282119.474328]
[1282119.475620] INFO: task jenkins.util.Ti:11283 blocked for more than 123 seconds.
[1282119.479241]       Not tainted 5.15.63-flatcar #1
[1282119.481472] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.485306] task:jenkins.util.Ti state:D stack:    0 pid:11283 ppid:  8007 flags:0x00000000
[1282119.489061] Call Trace:
[1282119.490271]
[1282119.491341]  __schedule+0x2eb/0x8d0
[1282119.492969]  schedule+0x5b/0xd0
[1282119.494583]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.497032]  ? wait_woken+0x70/0x70
[1282119.498657]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.501246]  ? pagecache_get_page+0x28b/0x470
[1282119.503242]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.505761]  ? __cond_resched+0x16/0x50
[1282119.507555]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.509870]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.512310]  __ext4_new_inode+0x73f/0x1710 [ext4]
[1282119.514499]  ext4_insert_dentry+0x1c75/0x1d30 [ext4]
[1282119.517080]  path_openat+0xf4b/0x1280
[1282119.518780]  do_filp_open+0xa9/0x150
[1282119.520453]  ? vfs_statx+0x74/0x130
[1282119.522096]  ? __check_object_size+0x146/0x160
[1282119.524139]  do_sys_openat2+0x9b/0x160
[1282119.525976]  __x64_sys_openat+0x54/0xa0
[1282119.527774]  do_syscall_64+0x3b/0x90
[1282119.529493]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[1282119.531871] RIP: 0033:0x7f08a89067e4
[1282119.533542] RSP: 002b:00007f08079fa410 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000101
[1282119.536954] RAX: ffffffffffffffda RBX: 00007f08079fa588 RCX: 00007f08a89067e4
[1282119.540170] RDX:<PHONE_NUMBER>0000c2 RSI: 00007f084c02c500 RDI: 00000000ffffff9c
[1282119.543579] RBP: 00007f084c02c500 R08:<PHONE_NUMBER>000000 R09: 00000007acb8f77e
[1282119.546883] R10:<PHONE_NUMBER>0001b6 R11:<PHONE_NUMBER>000293 R12:<PHONE_NUMBER>0000c2
[1282119.550079] R13:<PHONE_NUMBER>0000c2 R14: 00007f084c02c500 R15: 00007f08240f4000
[1282119.553255]
[1282119.554327] INFO: task SCM polling for:13607 blocked for more than 123 seconds.
[1282119.557600]       Not tainted 5.15.63-flatcar #1
[1282119.559902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.563425] task:SCM polling for state:D stack:    0 pid:13607 ppid:  8007 flags:0x00000000
[1282119.567152] Call Trace:
[1282119.568335]
[1282119.569377]  __schedule+0x2eb/0x8d0
[1282119.571001]  ? recalibrate_cpu_khz+0x10/0x10
[1282119.572959]  schedule+0x5b/0xd0
[1282119.575701]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.578165]  ? wait_woken+0x70/0x70
[1282119.579791]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.582297]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.584770]  ? __update_load_avg_cfs_rq+0x28c/0x310
[1282119.587014]  ? __cond_resched+0x16/0x50
[1282119.588776]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.591102]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.593532]  ext4_setattr+0x3a8/0x9a0 [ext4]
[1282119.595540]  notify_change+0x3c4/0x540
[1282119.597319]  ? ext4_es_delayed_clu+0x170/0x430 [ext4]
[1282119.599641]  ? ext4_es_delayed_clu+0x1ef/0x430 [ext4]
[1282119.601964]  ? do_truncate+0x7d/0xd0
[1282119.603622]  do_truncate+0x7d/0xd0
[1282119.605236]  path_openat+0x24d/0x1280
[1282119.606999]  do_filp_open+0xa9/0x150
[1282119.608658]  ? __fput+0xff/0x250
[1282119.610171]  ? __check_object_size+0x146/0x160
[1282119.612351]  do_sys_openat2+0x9b/0x160
[1282119.614152]  __x64_sys_openat+0x54/0xa0
[1282119.615983]  do_syscall_64+0x3b/0x90
[1282119.617649]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[1282119.619993] RIP: 0033:0x7f08a89067e4
[1282119.621775] RSP: 002b:00007f07f7faa2e0 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000101
[1282119.625933] RAX: ffffffffffffffda RBX: 000000071bca3a90 RCX: 00007f08a89067e4
[1282119.629183] RDX:<PHONE_NUMBER>000241 RSI: 00007f0840094c40 RDI: 00000000ffffff9c
[1282119.632397] RBP: 00007f0840094c40 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000051
[1282119.635645] R10:<PHONE_NUMBER>0001b6 R11:<PHONE_NUMBER>000293 R12:<PHONE_NUMBER>000241
[1282119.638964] R13:<PHONE_NUMBER>000241 R14: 00007f0840094c40 R15: 00007f081402cb48
[1282119.642228]
[1282119.643335] INFO: task Handling GET /l:830503 blocked for more than 123 seconds.
[1282119.646702]       Not tainted 5.15.63-flatcar #1
[1282119.648846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.652383] task:Handling GET /l state:D stack:    0 pid:830503 ppid:  8007 flags:0x00000000
[1282119.656380] Call Trace:
[1282119.657559]
[1282119.658594]  __schedule+0x2eb/0x8d0
[1282119.660229]  ? trigger_load_balance+0x60/0x300
[1282119.662271]  schedule+0x5b/0xd0
[1282119.663751]  rwsem_down_read_slowpath+0x32c/0x380
[1282119.665950]  ? hw_breakpoint_exceptions_notify+0x30/0xf0
[1282119.668366]  ? ktime_get+0x38/0xa0
[1282119.670018]  do_user_addr_fault+0x42f/0x670
[1282119.672059]  ? read_hv_sched_clock_tsc+0x5/0x40
[1282119.674158]  exc_page_fault+0x68/0x140
[1282119.675939]  asm_exc_page_fault+0x21/0x30
[1282119.677791] RIP: 0033:0x7f08a77a4d3c
[1282119.679642] RSP: 002b:00007f07f521bff0 EFLAGS: 00010202
[1282119.682089] RAX: 00007f08a6683d28 RBX: 00007f08a00402d0 RCX: 000000009c700000
[1282119.685259] RDX: 000000007ce00000 RSI: 000000007ce00000 RDI: 00007f08a0195bb0
[1282119.691672] RBP: 00007f07f521bff0 R08: 00007f08a850987c R09: 00007f0835091f20
[1282119.694867] R10: ffffffffffffffff R11:<PHONE_NUMBER>000000 R12: 00007f083c9b61d0
[1282119.698137] R13:<PHONE_NUMBER>000000 R14: 00007f08a76e57b0 R15:<PHONE_NUMBER>000800
[1282119.701408]
[1282119.702576] INFO: task org.jenkinsci.p:875804 blocked for more than 123 seconds.
[1282119.705988]       Not tainted 5.15.63-flatcar #1
[1282119.708152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.711696] task:org.jenkinsci.p state:D stack:    0 pid:875804 ppid:  8007 flags:0x00000000
[1282119.715495] Call Trace:
[1282119.716676]
[1282119.717757]  __schedule+0x2eb/0x8d0
[1282119.719495]  ? update_load_avg+0x7a/0x5e0
[1282119.721348]  schedule+0x5b/0xd0
[1282119.722885]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.725377]  ? wait_woken+0x70/0x70
[1282119.727048]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.729522]  ? timerqueue_del+0x2a/0x50
[1282119.731304]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.733797]  ? hrtimer_cancel+0x1d/0x40
[1282119.735798]  ? futex_wait+0x21a/0x240
[1282119.737584]  ? __cond_resched+0x16/0x50
[1282119.739363]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.741573]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.743920]  ext4_dirty_inode+0x35/0x80 [ext4]
[1282119.745987]  __mark_inode_dirty+0x147/0x320
[1282119.748024]  generic_update_time+0x6c/0xd0
[1282119.749950]  file_update_time+0x127/0x140
[1282119.751906]  ? generic_write_checks+0x61/0xc0
[1282119.753905]  ext4_llseek+0x3fa/0x1cb0 [ext4]
[1282119.755906]  new_sync_write+0x11c/0x1b0
[1282119.757693]  ? intel_get_event_constraints+0x300/0x390
[1282119.760046]  vfs_write+0x1de/0x270
[1282119.761659]  ksys_write+0x5f/0xe0
[1282119.763218]  do_syscall_64+0x3b/0x90
[1282119.764911]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[1282119.767352] RIP: 0033:0x7f08a8905fef
[1282119.769060] RSP: 002b:00007f07f1ade4c0 EFLAGS: 00000293 ORIG_RAX:<PHONE_NUMBER>000001
[1282119.772473] RAX: ffffffffffffffda RBX: 00007f0814407348 RCX: 00007f08a8905fef
[1282119.775715] RDX:<PHONE_NUMBER>00005e RSI: 00007f07f1ade540 RDI:<PHONE_NUMBER>000338
[1282119.779100] RBP: 00007f07f1ade510 R08:<PHONE_NUMBER>000000 R09: 000000076ae08648
[1282119.782445] R10:<PHONE_NUMBER>178e34 R11:<PHONE_NUMBER>000293 R12:<PHONE_NUMBER>00005e
[1282119.785816] R13: 00007f07f1ade540 R14:<PHONE_NUMBER>000338 R15:<PHONE_NUMBER>00005e
[1282119.789069]
[1282119.790140] INFO: task kworker/u8:2:874634 blocked for more than 123 seconds.
[1282119.793381]       Not tainted 5.15.63-flatcar #1
[1282119.795512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1282119.800485] task:kworker/u8:2    state:D stack:    0 pid:874634 ppid:     2 flags:0x00004000
[1282119.804354] Workqueue: writeback wb_workfn (flush-8:0)
[1282119.806726] Call Trace:
[1282119.807936]
[1282119.809026]  __schedule+0x2eb/0x8d0
[1282119.810692]  schedule+0x5b/0xd0
[1282119.812181]  jbd2_journal_free_reserved+0xca/0x9d0 [jbd2]
[1282119.814641]  ? wait_woken+0x70/0x70
[1282119.816909]  jbd2_journal_free_reserved+0x2d9/0x9d0 [jbd2]
[1282119.819409]  ? find_get_pages_range+0x197/0x200
[1282119.821507]  jbd2_journal_free_reserved+0x5ab/0x9d0 [jbd2]
[1282119.823988]  ? ext4_convert_inline_data+0xb07/0x2020 [ext4]
[1282119.826555]  ? __cond_resched+0x16/0x50
[1282119.828352]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[1282119.830542]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[1282119.833049]  __ext4_mark_inode_dirty+0x502/0x1880 [ext4]
[1282119.835474]  ? __cond_resched+0x16/0x50
[1282119.837281]  ? __getblk_gfp+0x27/0x60
[1282119.838986]  ? __ext4_handle_dirty_metadata+0x56/0x19b0 [ext4]
[1282119.841629]  ? ext4_mark_iloc_dirty+0x56a/0xaf0 [ext4]
[1282119.843979]  do_writepages+0xd1/0x200
[1282119.845682]  __writeback_single_inode+0x39/0x290
[1282119.847884]  writeback_sb_inodes+0x20d/0x490
[1282119.849861]  __writeback_inodes_wb+0x4c/0xe0
[1282119.851844]  wb_writeback+0x1c0/0x280
[1282119.853561]  wb_workfn+0x29f/0x4d0
[1282119.855195]  ? psi_task_switch+0x1e0/0x200
[1282119.857128]  process_one_work+0x226/0x3c0
[1282119.859031]  worker_thread+0x50/0x410
[1282119.860747]  ? process_one_work+0x3c0/0x3c0
[1282119.862674]  kthread+0x127/0x150
[1282119.864307]  ? set_kthread_struct+0x50/0x50
[1282119.867863]  ret_from_fork+0x22/0x30
[1282119.869538]
```

State of our research so far:

syzbot found a potential deadlock and a kernel dev noted there's a likeliness of this being a real issue: https://lore.kernel.org/lkml/20220824100652.227m7eq4zqq7luir@quack3/#t
the commit causing the above issue, 51ae846cff5, was included in 5.15.61, so 3227.2.2 would have it (5.15.63) while 3227.2.1 would not (5.15.58)

The suspicious commit has not been reverted yet, and is still in main (therefore, also in the 5.15 LTS kernel series). We will continue to closely monitor the situation.

@Champ-Goblem Since we still have trouble reproducing this on our end, would it be possible for you to apply below patch to 5.15.63 (i.e. 3227.2.2) and check if you can still reproduce the issue? The patch reverts the suspicious commit 51ae846cff5.
We have just started a discussion on linux-ext4 (replying to the syzbot thread) to raise awareness of potential issues with 51ae846cff5. A clear bisect with the patch reverted (and no issues caused) would help a lot in this discussion.
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 98f381f6fc18..072a9bc931fc 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3139,15 +3139,13 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)
 {
 	struct inode *inode = mapping->host;
 	journal_t *journal;
-	sector_t ret = 0;
 	int err;

-	inode_lock_shared(inode);
 	/*
 	 * We can get here for an inline file via the FIBMAP ioctl
 	 */
 	if (ext4_has_inline_data(inode))
-		goto out;
+		return 0;

 	if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY) &&
 			test_opt(inode->i_sb, DELALLOC)) {
@@ -3186,14 +3184,10 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)
 		jbd2_journal_unlock_updates(journal);

 		if (err)
-			goto out;
+			return 0;
 	}

-	ret = iomap_bmap(mapping, block, &ext4_iomap_ops);
-
-out:
-	inode_unlock_shared(inode);
-	return ret;
+	return iomap_bmap(mapping, block, &ext4_iomap_ops);
 }

 static int ext4_readpage(struct file *file, struct page *page)

@t-lo Yep I can give this a go, although I may not be able to provide results till some time next week

Should we help by building the patched image?

I have already started a build on our infra for the patched image, so that should be fine, it will probably take an hour and a bit for this to build. Ill look at getting it rolled out and start testing by the afternoon, so hopefully ill have some results either later today or tomorrow some time.
Ill be off from Friday till Monday so it may also be worth you guys doing some testing on your end in case I am unable to get any results by then.

Okay I think the patch provided above fixes the problem. Ive deployed two nodes, one running stock 5.15.63 and a second running with the patch.
On the regular node without the patch, I could recreate this reliably by spawning 20-30 pods each pulling different images. Every test run would result in SSH connections dying and the node would be become completely unresponsive with most of the images stuck in a ContainerCreating state.
On the node with the patch I ran the pull a couple of times, pruning the images between each run. No problems were encountered and the images all entered a Running state successfully, SSH connections were also responsive during this time.

Awesome, thank you very much for testing! We're discussing this issue with Jan Kara, an Ext4 maintainer, upstream. (https://www.spinics.net/lists/linux-ext4/msg85417.html ff., messages from today will become available in the archive tomorrow).
Unfortunately, there seems to be an issue with the stack traces we provided (from this very issue); we're currently looking into possible causes and will follow up with the maintainer.

@Champ-Goblem would you be able to rebuild and test and image with the 5.15.63 kernel, without the revert but with this PR applied: https://github.com/flatcar/coreos-overlay/pull/2196? This should fix the issue with the stacktraces being slightly off, and help upstream come up with a proper fix.

I have rebuilt flatcar with the above PR, below should be the logs from the system during the failure.
[ 1599.005306] INFO: task jbd2/sda9-8:702 blocked for more than 122 seconds.
[ 1599.012290]       Not tainted 5.15.63-flatcar #1
[ 1599.017128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.025100] task:jbd2/sda9-8     state:D stack:    0 pid:  702 ppid:     2 flags:0x00004000
[ 1599.033579] Call Trace:
[ 1599.036144]  <TASK>
[ 1599.038354]  __schedule+0x2eb/0x8d0
[ 1599.042109]  schedule+0x5b/0xd0
[ 1599.045372]  jbd2_journal_commit_transaction+0x301/0x18e0 [jbd2]
[ 1599.051518]  ? wait_woken+0x70/0x70
[ 1599.055127]  ? lock_timer_base+0x61/0x80
[ 1599.059181]  kjournald2+0xab/0x270 [jbd2]
[ 1599.063317]  ? wait_woken+0x70/0x70
[ 1599.066923]  ? load_superblock.part.0+0xb0/0xb0 [jbd2]
[ 1599.072200]  kthread+0x124/0x150
[ 1599.075543]  ? set_kthread_struct+0x50/0x50
[ 1599.079849]  ret_from_fork+0x1f/0x30
[ 1599.083538]  </TASK>
[ 1599.085835] INFO: task kworker/u32:13:732 blocked for more than 122 seconds.
[ 1599.093010]       Not tainted 5.15.63-flatcar #1
[ 1599.097739] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.105688] task:kworker/u32:13  state:D stack:    0 pid:  732 ppid:     2 flags:0x00004000
[ 1599.114160] Workqueue: writeback wb_workfn (flush-8:0)
[ 1599.119418] Call Trace:
[ 1599.121976]  <TASK>
[ 1599.124192]  __schedule+0x2eb/0x8d0
[ 1599.127797]  schedule+0x5b/0xd0
[ 1599.131051]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1599.136227]  ? wait_woken+0x70/0x70
[ 1599.139829]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1599.145091]  ? find_get_pages_range+0x197/0x200
[ 1599.149832]  start_this_handle+0xfb/0x520 [jbd2]
[ 1599.154591]  ? mpage_release_unused_pages+0x1c7/0x1e0 [ext4]
[ 1599.160524]  ? __cond_resched+0x16/0x50
[ 1599.164478]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1599.169393]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1599.174669]  ext4_writepages+0x302/0xfd0 [ext4]
[ 1599.179328]  ? __find_get_block+0xb3/0x2c0
[ 1599.183539]  ? __cond_resched+0x16/0x50
[ 1599.187518]  ? __getblk_gfp+0x27/0x60
[ 1599.191307]  ? cpumask_next_and+0x1f/0x30
[ 1599.195433]  ? update_sd_lb_stats.constprop.0+0xff/0x8a0
[ 1599.200888]  do_writepages+0xce/0x200
[ 1599.204788]  ? _raw_spin_unlock_irqrestore+0xa/0x30
[ 1599.209796]  ? percpu_counter_add_batch+0x5b/0x70
[ 1599.214627]  ? fprop_reflect_period_percpu.isra.0+0x7b/0xc0
[ 1599.220439]  __writeback_single_inode+0x39/0x290
[ 1599.225183]  writeback_sb_inodes+0x20d/0x490
[ 1599.229569]  __writeback_inodes_wb+0x4c/0xe0
[ 1599.233951]  wb_writeback+0x1c0/0x280
[ 1599.237727]  wb_workfn+0x29f/0x4d0
[ 1599.241244]  process_one_work+0x223/0x3c0
[ 1599.245371]  worker_thread+0x50/0x410
[ 1599.249146]  ? process_one_work+0x3c0/0x3c0
[ 1599.253460]  kthread+0x124/0x150
[ 1599.256811]  ? set_kthread_struct+0x50/0x50
[ 1599.261110]  ret_from_fork+0x1f/0x30
[ 1599.264820]  </TASK>
[ 1599.267171] INFO: task systemd-journal:1098 blocked for more than 123 seconds.
[ 1599.274538]       Not tainted 5.15.63-flatcar #1
[ 1599.279282] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.287256] task:systemd-journal state:D stack:    0 pid: 1098 ppid:     1 flags:0x00000224
[ 1599.295753] Call Trace:
[ 1599.298343]  <TASK>
[ 1599.300570]  __schedule+0x2eb/0x8d0
[ 1599.304178]  schedule+0x5b/0xd0
[ 1599.307434]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1599.312814]  ? wait_woken+0x70/0x70
[ 1599.316431]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1599.321722]  start_this_handle+0xfb/0x520 [jbd2]
[ 1599.326460]  ? __cond_resched+0x16/0x50
[ 1599.330414]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1599.335331]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1599.340615]  ext4_truncate+0x167/0x480 [ext4]
[ 1599.345134]  ext4_setattr+0x59a/0x9a0 [ext4]
[ 1599.349567]  ? virtnet_poll+0x31b/0x45b [virtio_net]
[ 1599.354663]  ? common_interrupt+0xf/0xa0
[ 1599.358705]  notify_change+0x3c1/0x540
[ 1599.362591]  ? do_truncate+0x7d/0xd0
[ 1599.366292]  do_truncate+0x7d/0xd0
[ 1599.369836]  do_sys_ftruncate+0xc9/0x150
[ 1599.373882]  do_syscall_64+0x38/0x90
[ 1599.377581]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1599.382757] RIP: 0033:0x7fc405986757
[ 1599.386449] RSP: 002b:00007ffdf6776af8 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>00004d
[ 1599.394126] RAX: ffffffffffffffda RBX: 00007ffdf6776b40 RCX: 00007fc405986757
[ 1599.401371] RDX: 0000557b78aa8f40 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>000015
[ 1599.408615] RBP: 0000557b78aac520 R08:<PHONE_NUMBER>000001 R09: 0000557b78aac5ac
[ 1599.415860] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12: 0000557b78a9c600
[ 1599.423119] R13: 00007ffdf6776b38 R14:<PHONE_NUMBER>000003 R15:<PHONE_NUMBER>000000
[ 1599.430406]  </TASK>
[ 1599.432754] INFO: task systemd-timesyn:1277 blocked for more than 123 seconds.
[ 1599.440097]       Not tainted 5.15.63-flatcar #1
[ 1599.444842] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.452799] task:systemd-timesyn state:D stack:    0 pid: 1277 ppid:     1 flags:0x00000224
[ 1599.461268] Call Trace:
[ 1599.463836]  <TASK>
[ 1599.466046]  __schedule+0x2eb/0x8d0
[ 1599.469684]  schedule+0x5b/0xd0
[ 1599.472946]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1599.478133]  ? wait_woken+0x70/0x70
[ 1599.481839]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1599.487387]  ? __fget_files+0x79/0xb0
[ 1599.491185]  start_this_handle+0xfb/0x520 [jbd2]
[ 1599.495963]  ? nd_jump_link+0x4d/0xc0
[ 1599.499752]  ? __cond_resched+0x16/0x50
[ 1599.503708]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1599.508620]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1599.513897]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 1599.518477]  __mark_inode_dirty+0x144/0x320
[ 1599.522795]  ext4_setattr+0x1fb/0x9a0 [ext4]
[ 1599.527200]  notify_change+0x3c1/0x540
[ 1599.531071]  ? vfs_utimes+0x139/0x220
[ 1599.534845]  vfs_utimes+0x139/0x220
[ 1599.538446]  do_utimes+0xb4/0x120
[ 1599.541874]  __x64_sys_utimensat+0x70/0xb0
[ 1599.546132]  ? syscall_trace_enter.constprop.0+0x143/0x1c0
[ 1599.551751]  do_syscall_64+0x38/0x90
[ 1599.555443]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1599.560628] RIP: 0033:0x7fce21af901f
[ 1599.564858] RSP: 002b:00007ffedd36c8c8 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>000118
[ 1599.572559] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX: 00007fce21af901f
[ 1599.579806] RDX:<PHONE_NUMBER>000000 RSI: 00007ffedd36c8d0 RDI: 00000000ffffff9c
[ 1599.587062] RBP: 00007ffedd36c8d0 R08:<PHONE_NUMBER>000000 R09: 00007ffedd36c760
[ 1599.594313] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12:<PHONE_NUMBER>000000
[ 1599.601558] R13: 00000000ffffffff R14: ffffffffffffffff R15: 00000000ffffffff
[ 1599.608806]  </TASK>
[ 1599.611124] INFO: task bash:1925 blocked for more than 123 seconds.
[ 1599.617618]       Not tainted 5.15.63-flatcar #1
[ 1599.622347] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.630283] task:bash            state:D stack:    0 pid: 1925 ppid:  1924 flags:0x00000004
[ 1599.638750] Call Trace:
[ 1599.641309]  <TASK>
[ 1599.643528]  __schedule+0x2eb/0x8d0
[ 1599.647140]  schedule+0x5b/0xd0
[ 1599.650399]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1599.655571]  ? wait_woken+0x70/0x70
[ 1599.659188]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1599.664444]  ? pagecache_get_page+0x28b/0x470
[ 1599.668914]  start_this_handle+0xfb/0x520 [jbd2]
[ 1599.673664]  ? __cond_resched+0x16/0x50
[ 1599.677615]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1599.682525]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1599.687801]  __ext4_new_inode+0x73f/0x1710 [ext4]
[ 1599.692639]  ext4_create+0x115/0x1d0 [ext4]
[ 1599.696968]  path_openat+0xf48/0x1280
[ 1599.700751]  ? _raw_spin_unlock_irqrestore+0xa/0x30
[ 1599.705742]  ? __wake_up_common_lock+0x8a/0xc0
[ 1599.710299]  do_filp_open+0xa9/0x150
[ 1599.713990]  ? vfs_statx+0x74/0x130
[ 1599.717615]  ? __check_object_size+0x146/0x160
[ 1599.722179]  do_sys_openat2+0x9b/0x160
[ 1599.726057]  __x64_sys_openat+0x54/0xa0
[ 1599.730003]  do_syscall_64+0x38/0x90
[ 1599.733691]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1599.738870] RIP: 0033:0x7f138fdb5337
[ 1599.742558] RSP: 002b:00007ffdc49dcaa0 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000101
[ 1599.750249] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX: 00007f138fdb5337
[ 1599.757497] RDX:<PHONE_NUMBER>000241 RSI: 00005583f2829d70 RDI: 00000000ffffff9c
[ 1599.764747] RBP: 00005583f2829d70 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000000
[ 1599.771998] R10:<PHONE_NUMBER>000180 R11:<PHONE_NUMBER>000246 R12:<PHONE_NUMBER>000241
[ 1599.779258] R13:<PHONE_NUMBER>000000 R14: 00005583f1f45534 R15:<PHONE_NUMBER>000000
[ 1599.786510]  </TASK>
[ 1599.788906] INFO: task MVStore backgro:8970 blocked for more than 123 seconds.
[ 1599.796262]       Not tainted 5.15.63-flatcar #1
[ 1599.800992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.808931] task:MVStore backgro state:D stack:    0 pid: 8970 ppid:  8062 flags:0x00000000
[ 1599.817398] Call Trace:
[ 1599.819968]  <TASK>
[ 1599.822205]  __schedule+0x2eb/0x8d0
[ 1599.825816]  schedule+0x5b/0xd0
[ 1599.829073]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1599.834250]  ? wait_woken+0x70/0x70
[ 1599.837852]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1599.843108]  start_this_handle+0xfb/0x520 [jbd2]
[ 1599.847859]  ? __cond_resched+0x16/0x50
[ 1599.851810]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1599.856715]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1599.861992]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 1599.866585]  __mark_inode_dirty+0x144/0x320
[ 1599.870892]  generic_update_time+0x6c/0xd0
[ 1599.875118]  file_update_time+0x127/0x140
[ 1599.879242]  ? generic_write_checks+0x61/0xc0
[ 1599.883721]  ext4_buffered_write_iter+0x5a/0x180 [ext4]
[ 1599.889103]  do_iter_readv_writev+0x14f/0x1b0
[ 1599.893582]  do_iter_write+0x80/0x1c0
[ 1599.897370]  ovl_write_iter+0x2d3/0x450 [overlay]
[ 1599.902195]  new_sync_write+0x119/0x1b0
[ 1599.906151]  ? intel_get_event_constraints+0x300/0x390
[ 1599.911403]  vfs_write+0x1de/0x270
[ 1599.914917]  __x64_sys_pwrite64+0x91/0xc0
[ 1599.919039]  do_syscall_64+0x38/0x90
[ 1599.922727]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1599.927968] RIP: 0033:0x7f2f532424a3
[ 1599.931659] RSP: 002b:00007f2f2cc8aa78 EFLAGS: 00000246 ORIG_RAX:<PHONE_NUMBER>000012
[ 1599.939352] RAX: ffffffffffffffda RBX: 00007f2f2cc8bb38 RCX: 00007f2f532424a3
[ 1599.946606] RDX:<PHONE_NUMBER>001000 RSI: 00007f2f2c769d90 RDI:<PHONE_NUMBER>000014
[ 1599.953875] RBP: 00007f2f2cc8aaf0 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000000
[ 1599.961127] R10:<PHONE_NUMBER>0a3000 R11:<PHONE_NUMBER>000246 R12:<PHONE_NUMBER>000012
[ 1599.968527] R13: 00007f2f2c769d90 R14:<PHONE_NUMBER>0a3000 R15: 00007f2f2f94b800
[ 1599.975775]  </TASK>
[ 1599.978135] INFO: task k3s-server:12051 blocked for more than 123 seconds.
[ 1599.985142]       Not tainted 5.15.63-flatcar #1
[ 1599.989873] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1599.997816] task:k3s-server      state:D stack:    0 pid:12051 ppid:     1 flags:0x00000000
[ 1600.006283] Call Trace:
[ 1600.008862]  <TASK>
[ 1600.011083]  __schedule+0x2eb/0x8d0
[ 1600.014779]  schedule+0x5b/0xd0
[ 1600.018034]  rwsem_down_write_slowpath+0x220/0x4f0
[ 1600.022947]  chown_common+0x152/0x250
[ 1600.026740]  ? __do_sys_newfstat+0x57/0x60
[ 1600.030950]  ? __fget_files+0x79/0xb0
[ 1600.034749]  ksys_fchown+0x74/0xb0
[ 1600.038283]  __x64_sys_fchown+0x16/0x20
[ 1600.042230]  do_syscall_64+0x38/0x90
[ 1600.045922]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1600.051091] RIP: 0033:0x3f8045f
[ 1600.054346] RSP: 002b:00007f749fe0a670 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>00005d
[ 1600.062050] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX:<PHONE_NUMBER>f8045f
[ 1600.069297] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>000188
[ 1600.076557] RBP: 00007f749dfd2498 R08:<PHONE_NUMBER>000188 R09:<PHONE_NUMBER>72a6fb
[ 1600.083808] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12:<PHONE_NUMBER>000188
[ 1600.091051] R13: 00007f749f5416cd R14:<PHONE_NUMBER>080006 R15:<PHONE_NUMBER>0001a4
[ 1600.098312]  </TASK>
[ 1600.100611] INFO: task k3s-server:12052 blocked for more than 123 seconds.
[ 1600.107601]       Not tainted 5.15.63-flatcar #1
[ 1600.112344] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1600.120281] task:k3s-server      state:D stack:    0 pid:12052 ppid:     1 flags:0x00000000
[ 1600.128758] Call Trace:
[ 1600.131315]  <TASK>
[ 1600.133536]  __schedule+0x2eb/0x8d0
[ 1600.137136]  schedule+0x5b/0xd0
[ 1600.140408]  wait_transaction_locked+0x8a/0xd0 [jbd2]
[ 1600.145574]  ? wait_woken+0x70/0x70
[ 1600.149190]  add_transaction_credits+0xd9/0x2b0 [jbd2]
[ 1600.154442]  ? __cond_resched+0x16/0x50
[ 1600.158395]  ? dput+0x32/0x310
[ 1600.161559]  start_this_handle+0xfb/0x520 [jbd2]
[ 1600.166290]  ? asm_sysvec_apic_timer_interrupt+0x15/0x20
[ 1600.171713]  ? __cond_resched+0x16/0x50
[ 1600.175660]  jbd2__journal_start+0xfb/0x1e0 [jbd2]
[ 1600.180568]  __ext4_journal_start_sb+0xf8/0x110 [ext4]
[ 1600.185846]  ext4_dirty_inode+0x35/0x80 [ext4]
[ 1600.190441]  __mark_inode_dirty+0x144/0x320
[ 1600.194737]  ext4_setattr+0x1fb/0x9a0 [ext4]
[ 1600.199134]  notify_change+0x3c1/0x540
[ 1600.203002]  ? chown_common+0x168/0x250
[ 1600.206957]  chown_common+0x168/0x250
[ 1600.210731]  ? __fget_files+0x79/0xb0
[ 1600.214502]  ksys_fchown+0x74/0xb0
[ 1600.218017]  __x64_sys_fchown+0x16/0x20
[ 1600.221965]  do_syscall_64+0x38/0x90
[ 1600.225674]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1600.230851] RIP: 0033:0x3f8045f
[ 1600.234114] RSP: 002b:00007f749fde7670 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>00005d
[ 1600.241794] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX:<PHONE_NUMBER>f8045f
[ 1600.249050] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>000145
[ 1600.256388] RBP: 00007f749e2213c8 R08:<PHONE_NUMBER>000145 R09:<PHONE_NUMBER>72a6fb
[ 1600.263636] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12:<PHONE_NUMBER>000145
[ 1600.270890] R13: 00007f749dfbef0d R14:<PHONE_NUMBER>080006 R15:<PHONE_NUMBER>0001a4
[ 1600.278138]  </TASK>
[ 1600.280436] INFO: task k3s-server:12055 blocked for more than 124 seconds.
[ 1600.287449]       Not tainted 5.15.63-flatcar #1
[ 1600.292184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1600.300127] task:k3s-server      state:D stack:    0 pid:12055 ppid:     1 flags:0x00000000
[ 1600.308590] Call Trace:
[ 1600.311151]  <TASK>
[ 1600.313358]  __schedule+0x2eb/0x8d0
[ 1600.316957]  schedule+0x5b/0xd0
[ 1600.320208]  rwsem_down_write_slowpath+0x220/0x4f0
[ 1600.325113]  chown_common+0x152/0x250
[ 1600.328890]  ? __do_sys_newfstat+0x57/0x60
[ 1600.333107]  ? __fget_files+0x79/0xb0
[ 1600.336885]  ksys_fchown+0x74/0xb0
[ 1600.340400]  __x64_sys_fchown+0x16/0x20
[ 1600.344346]  do_syscall_64+0x38/0x90
[ 1600.348035]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1600.353218] RIP: 0033:0x3f8045f
[ 1600.356471] RSP: 002b:00007f749fcae670 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>00005d
[ 1600.364149] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX:<PHONE_NUMBER>f8045f
[ 1600.371397] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>00019c
[ 1600.378648] RBP: 00007f749f539888 R08:<PHONE_NUMBER>00019c R09:<PHONE_NUMBER>72a6fb
[ 1600.385895] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12:<PHONE_NUMBER>00019c
[ 1600.393155] R13: 00007f749e2e5a3d R14:<PHONE_NUMBER>080006 R15:<PHONE_NUMBER>0001a4
[ 1600.400407]  </TASK>
[ 1600.402761] INFO: task k3s-server:12057 blocked for more than 124 seconds.
[ 1600.409779]       Not tainted 5.15.63-flatcar #1
[ 1600.414512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1600.422470] task:k3s-server      state:D stack:    0 pid:12057 ppid:     1 flags:0x00000000
[ 1600.430934] Call Trace:
[ 1600.433510]  <TASK>
[ 1600.435722]  __schedule+0x2eb/0x8d0
[ 1600.439328]  schedule+0x5b/0xd0
[ 1600.442583]  rwsem_down_write_slowpath+0x220/0x4f0
[ 1600.447495]  chown_common+0x152/0x250
[ 1600.451274]  ? __do_sys_newfstat+0x57/0x60
[ 1600.455483]  ? __fget_files+0x79/0xb0
[ 1600.459258]  ksys_fchown+0x74/0xb0
[ 1600.462773]  __x64_sys_fchown+0x16/0x20
[ 1600.466724]  do_syscall_64+0x38/0x90
[ 1600.470416]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 1600.475604] RIP: 0033:0x3f8045f
[ 1600.478857] RSP: 002b:00007f749fc28670 EFLAGS: 00000202 ORIG_RAX:<PHONE_NUMBER>00005d
[ 1600.486539] RAX: ffffffffffffffda RBX:<PHONE_NUMBER>000000 RCX:<PHONE_NUMBER>f8045f
[ 1600.493788] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000000 RDI:<PHONE_NUMBER>000196
[ 1600.501050] RBP: 00007f749f64dac8 R08:<PHONE_NUMBER>000196 R09:<PHONE_NUMBER>72a6fb
[ 1600.508318] R10:<PHONE_NUMBER>000000 R11:<PHONE_NUMBER>000202 R12:<PHONE_NUMBER>000196
[ 1600.515569] R13: 00007f749eadef0d R14:<PHONE_NUMBER>080006 R15:<PHONE_NUMBER>0001a4
[ 1600.522817]  </TASK>
[ 1629.084561] systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.

Thanks, @Champ-Goblem, I'll forward that to the mailing list, any chance you can also provide the following:

So probably first try find out why stacktraces are not working right on
your kernel and fix them. And then, if the hang happens, please trigger
sysrq-w (or do echo w >/proc/sysrq-trigger if you can still get to the
machine) and send here the output. It will dump all blocked tasks and from
that we should be able to better understand what is happening.

@jepio I think I have managed to get some of the list of hung tasks, I cant be sure how complete the list is, but I think it should be a good starting point nonetheless
hung-tasks.txt

We also experience this. Any ideas if and when this will be solved? Is this certain that's vanilla kernel issue? If so, other distos should have the same problem as well.
We're already making decision to come back to LTS due this... but what if this comes to LTS too?
This is pretty serious. :/

This is definitely a vanilla kernel issue, as Flatcar runs a vanilla kernel (the 6 patches we carry are arm64 specific or build system tweaks).
We agree this is pretty serious, but since none of us are filesystem developers, we don't just want to go around reverting filesystem code patches, and risking the integrity of users' data. We're waiting on upstream kernel developers. In the mean time I suggest you freeze the Flatcar release that you're using to one that is unaffected (an earlier stable or lts).

@Champ-Goblem Thanks again for getting the list of tasks! This is helpful but we don't have a smoking gun yet. Upstream suspects that additional patches to 6.x mitigate the issue there, so only 5.15.x is affected currently.
Upstream linux-ext4 folks produced a debug helper patch (attached) and kindly request to reproduce the issue with a kernel that has this patch. We once again need the output of echo w > /proc/sysrq-trigger as well as the kernel log (dmesg). Could you please help out?
0001-Debug-buffer_head-lock.patch.txt

This also happened on one of our clusters today.
Environment: OpenStack
(Not a flatcar problem, but) I think rebooting the nodes made k0s/etcd barf somehow and it generated a lot of load while trying to start up and join the cluster. i/o is through the roof for some reason and eventually overwhelmed the node and even crashes the k0s service.
SSH somewhat unresponsive (it seems to be vary between hosts). Some login is possible, others, nothing.
In the end, the node is becoming totally unresponsive. I managed to revive one so far by downgrading to the LTS and it seems to have settled.
sudo flatcar-update --to-version 3033.3.6 --disable-afterwards

Hi @Champ-Goblem, are you still able to help with debugging? Could you try it with the patch and steps that @t-lo posted. We're getting close to finding the bug

@jepio @t-lo I will try and have a look at this over the next 2 days

@jepio @t-lo Sorry for the wait, I have the output from dmesg including the sysrq-trigger output, but unfortunately I dont see any logs from the patch I was asked to apply. I am confident that the patch was applied correctly to the kernel (I added another patch after it that added helloworld to the Command line: log line because I was not sure if it was being applied at first)
5-15-63-dmesg-with-patch.txt

That's too bad. On a positive note - i'm now able to recreate this with the reproducer you provided :)
Turns out only looking at the hungs tasks only reveals half the picture because there are also some processes that are stuck running somewhere in ext4 code, i'll post an update as soon as i understand better.

So the update is that the issue is also present in upstream 6.0 kernels, it appears to be a logic bug in this commit: https://github.com/torvalds/linux/commit/65f8b80053a1b2fd602daa6814e62d6fa90e5e9b, which results in processes getting stuck trying to reuse a block that has too many references.
I've provided more debug output to the kernel mailing list, now that the problem has been identified I hope this will lead to a fix soon.

There's a fix available now (https://lore.kernel.org/linux-ext4/20221122115715.kxqhsk2xs4nrofyb@quack3/T/#ma65f0113b4b2f9259b8f7d8af16b8cb351728d20) and I've tested it on 5.15 with the reproducer in this thread, I can no longer reproduce the hang.
This should now get merged upstream and subsequently backported. Then it'll be included in a Flatcar stable release.

Thanks for all the work that went into fixing this.
Any update on the upstream merging? Is this progressing? I'm not familiar with kernel development and so far it seems to me nobody has responded to the patch by the ext4 developer and nothing seems to be happening.
Any advise were I can track if the patch has landed upstream?
We have stopped flatcar OS upgrades for 3 month now and our PCI compliance is not looking too good... 😬

@databus23 We have worked around this by changing the root FS from Ext4 to XFS. Our tests look good so far but haven't rolled it out to production yet. In case you need to upgrade flatcar for some reason, this might be an alternative idea for you until the ext4 fix has made it's way up to a flatcar stable release.

We have been testing the new patch [1] which we have applied against 5.15.80 in a custom build and it is currently looking stable again on our development clusters with Flatcar 3227.2.2
[1]<EMAIL_ADDRESS>

Any update on the upstream merging? Is this progressing? I'm not familiar with kernel development and so far it seems to me nobody has responded to the patch by the ext4 developer and nothing seems to be happening.

Unfortunately there hasn't been any response from the ext4 maintainer, I've poked them this week.
In the meantime we'll likely take the backport into the next alpha/beta Flatcar release next week.
@Champ-Goblem thanks all the help with testing and reproducing, it's good to have a datapoint confirming that no new issues pop-up with the patch.

When is it in stable?

Didn't mean to close this. It will land in stable in 1-2 release cycles (next time beta promotes), or when the bugfix lands in the stable trees.
If you're hitting this issue and have been holding off on updates - do update to the beta release that will come this week.

is this still not in stable? I've tought it already is and updated to 3374.2.1 and still get processes to hang.

Hi @schmitch
I think the kernel patch is available in the latest beta release 3432.1.0 but not promoted to stable yet.

is this still not in stable?

No, it's added to beta in 3432.1.0. So as long as stable is on a smaller version it hasn't trickled down to stable.

No, it's added to beta in 3432.1.0. So as long as stable is on a smaller version it hasn't trickled down to stable.

Correct. The fix is not in Stable yet, but only in Beta & Alpha.
Good news is, the ext4 deadlock fix was recently merged to the mainline Kernel. Though that is still not included in any Kernel release. Looks like it needs a little more time.
Anyway most of Flatcar maintainers already left for holidays, so the next release should happen in Jan. in the next year.

is this still not in stable? I've tought it already is and updated to 3374.2.1 and still get processes to hang.

I can confirm that this is still an issue on the stable track, as it is still on version 3374.2.1. This appeared immediately as I rolled the upgrades out manually to 4 of my nodes. The issue appears to be present on only 1 of the 4 nodes, although the nodes have exactly the same hardware, software, and Kubernetes installed with Kubespray. I'm now updating to Beta channel.

Looking forward to the next stable channel release to solve this problem.

No more tears with beta 3432.1.0.

The fix is now released with Kernel 6.1.4

And i'm working on getting it into 5.15.x.
<EMAIL_ADDRESS><EMAIL_ADDRESS>

the fix is queued up for 5.15.87<EMAIL_ADDRESS>

Yesterdays release did not include the bugfix for this! was that intentional.
as i am really waiting for this! If that will take a longer time, i would need to temporary switch to the beta channel, but that is a lot of work to switch everything :-(

5.15.87 got released with the patch included!
Could you please release a new version of flatcar-linux including the new kernel release!?

We'll make sure the fix (either 5.15.87 or the backport) is part of the next stable release scheduled for 2022-01-23.

Flatcar Stable 3374.2.3 was released with the bugfix, Kernel 5.15.86 with the backport.
Please try it out, and see if the bug was gone.

Looks good now

We are still facing the issue with the current stable release 3374.2.3:

Beta 3432.1.0 is working fine for us.

@mduerr3cgi that does not look like the same issue. Can you open a new issue and paste full logs (and test the 2 more recent beta releases).

I assume that this issue was resolved in Stable 3374.2.3.
Please create new issues for other bugs.
Thanks.


No tests are generated for String inputs

Description
utbot-cli does not generate some tests with strings.
(run without fuzzing)
To Reproduce
package org.example.checks;

public class StringCheck {
    public boolean example(String s, String s2, String s3, String s4) {
        if ((s + s2).equals(s3 + "." + s4)) {
            return true;
        }
        return false;
    }
}

run utbot-cli in idea (fe0b89c):
program arguments: generate --classpath "/home/andrew/ex/build/classes/java/main" --source /home/andrew/ex/src/main/java/org/example/checks/StringCheck.java --output Test.java org.example.checks.StringCheck

Expected behavior
Generated tests for true and for false
Actual behavior
test only for false
package org.example.checks;

import org.junit.Test;

import static org.junit.Assert.assertFalse;

public final class Test {
    ///region Test suites for executable org.example.checks.StringCheck.example

    ///region OTHER: SUCCESSFUL EXECUTIONS for method example(java.lang.String, java.lang.String, java.lang.String, java.lang.String)

    @Test
    public void testExample1() {
        StringCheck stringCheck = new StringCheck();
        String string = "";
        String string1 = "";

        boolean actual = stringCheck.example(string, string1, string, string);

        assertFalse(actual);
    }
    ///endregion

    ///endregion
}

Environment
Ubuntu 21.04, jdk 17.

@pogrebnoijak
As discussed with the team such cases are not always covered by UnitTestBot. As they contain several String concatenations with StringBuilder append operations inside. Fuzzing improves coverage of similar cases. But it's not 100% also. There are plans to improve how UnitTestBot works with Strings.


Update Compute SDK to latest (including ACS)

Description
This finishes @brendandburns's efforts to get the Compute SDK regenerated. (Prior pull request: https://github.com/Azure/azure-sdk-for-net/pull/2614)
I took his commit, verbatim, and simply fixed what needed to be fixed to get the tests passing.
Commits are self explanatory.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] I have read the contribution guidelines.
[ ] The pull request does not introduce breaking changes.

Per the last discussion, it does introduce a breaking change, but the version is bumped accordingly.
General Guidelines

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[ ] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[ ] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as a link to the swagger spec, used to generate the code.
[x] The project.json and AssemblyInfo.cs files have been updated with the new version of the SDK.

Can one of the admins verify this patch?

Hi @colemickens, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (colemick). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@azuresdkci add to whitelist

@colemickens How does this PR relate to this PR in compute:  https://github.com/Azure/azure-sdk-for-net/pull/2751
If your changes are necessary, please work with Hyonho on getting them incorproarted into his PR.

It would appear that that PR does basically the same things as mine, and then a few more. I'll comment on that issue to see if my PR can simply be dropped... Thanks for the heads up.

(though the AZURE_TEST_MODE commit will be preserved - I wasted a large amount of time on that)

cc: @markcowl @hyonholee
I feel strongly that this PR should be preferred:

It includes additional fixes to make the ContainerServices tests really pass
Separate commits for the various bumping of things
Includes a fix for AZURE_TEST_MODE problem which cost a lot of time

More concerningly, the other PR is created by manually updating the contents of the test recordings and by my measure could be (and likely is) masking failure cases.
Case in point, my latest push has changes to the test record files, but only a subset. There are a set of Compute tests that are reliably failing - including failures that look like potential regressions or indicate problems with the Compute tests.
I have updated this PR with the latest. The test record files included in the most recent commit are the ones that work. The ones that remain, and need investigation by Compute are:

Compute.Tests.ExtensionTests.TestVMExtensionOperations
Compute.Tests.ListVMTests.TestListVMInSubscription
Compute.Tests.OSProfileTests.TestVMWithLinuxOSProfile
Compute.Tests.UsageTests.TestListUsages
Compute.Tests.VMDataDiskTests.TestVMDataDiskScenario
Compute.Tests.VMDiagnosticsTests.TestVMBootDiagnostics
Compute.Tests.VMDiskSizeTests.TestVMDiskSizeScenario
Compute.Tests.VMMarketplaceTest.TestVMBYOL
Compute.Tests.VMMarketplaceTest.TestVMMarketplace
Compute.Tests.VMNetworkInterfaceTests.TestEffectiveRouteAndAcls
Compute.Tests.VMNetworkInterfaceTests.TestMultiIpConfigForMultiNICVM
Compute.Tests.VMNetworkInterfaceTests.TestMultiNicVirtualMachineReference
Compute.Tests.VMNetworkInterfaceTests.TestNicVirtualMachineReference
Compute.Tests.VMOperationalTests.TestVMOperations
Compute.Tests.VMOperationalTests.TestVMOperations_Redeploy
Compute.Tests.VMScenarioTests.TestVMScenarioOperations

(and note, I've fixed all of the things requested by @shahabhijeet)

There is one thing I'm unclear about. I'm unsure whether I actually pulled the Compute swagger up to the latest in any of these commits. How is that done? I couldn't easily discern this from the other PR as the commits are too big to review easily on GitHub.

Offline discussion with @shahabhijeet has quelled my concerns. I have indeed generated for the latest available compositeComputeSwagger, since the Compute generate.cmd just pulls from master and the composite compute swagger is... unversioned...

(BTW: @hyonholee the cred scanner was correctly complaining. The ssh public key in the code now is actually a private key...)


eos: implement virtual shared folder

Implement virtual share folder behaviour by relying on the read amplification model.
The virtual overlay ETAG for the root folder (/home) and the shared folder (/home/MyShares) is done by the gateway to avoid client-side developments for the time being.
On an empty path:
EOS Console [root://localhost] |/eos/test/reva/> find .
/eos/test/reva/

The user logs in:
[gonzalhu@iota reva]$ reva login basic
username: gonzalhu
password: OK

The user real and shadow homedirectories are created. The shadow homedir contains a MyShares folder that will contain the references for all the shares of the user.
EOS Console [root://localhost] |/eos/test/reva/> find .
/eos/test/reva/
/eos/test/reva/users/
/eos/test/reva/users/.shadow/
/eos/test/reva/users/.shadow/g/
/eos/test/reva/users/.shadow/g/gonzalhu/
/eos/test/reva/users/.shadow/g/gonzalhu/MyShares/
/eos/test/reva/users/g/
/eos/test/reva/users/g/gonzalhu/

The user cannot fiddle around with the MyShares virtual folder:
[gonzalhu@iota reva]$ reva mkdir /home/MyShares
error: code=CODE_INTERNAL msg="error creating container: path:\"/home/MyShares\" " support_trace="b5c8092eb5a8f80db30f7f5b30d25fda"

7:24PM ERR home/gonzalhu/dev/reva/internal/grpc/services/storageprovider/storageprovider.go:342 > error creating container: path:"/home/MyShares"  error="error: permission denied: eos: cannot create folder under the share folder" pid=8931 pkg=rgrpc traceid=b5c8092eb5a8f80db30f7f5b30d25fda

The shadow cannot be accessed by users.

AFAICT this is issue is about how to implement references in the eos storage driver. IIUC you plan to use a shadow tree where you place all shares references in a <username>/MyShares folder. Since this is inaccessible for normal users they cannot break anything on the cli. More importantly, you can use the Myshares folder to store the calculated ETag for all shares. This will allow you to calculate the ETag for the root of the namespace that is visible to users by combining the ETag of the users home and the MyShared node.
Correct me if I am wrong until this point.
Now, I think you are semantically doing exactly the same thing I am proposing in https://github.com/owncloud/ocis/pull/127 but using the shadow tree in eos to cache and persist the share entry points. I like the ide of using EOS or the filesystem to store a flat list of share references per user and calculate an etag for them.
How about implementing references using .ref files, similar to windows .lnk, linux .desktop files and Mac OS aliases? Instead of a /MyShares folder that is visible to clients users can decide to mount a share anywhere they want. Technically the shadow tree is still there and is used to store the ETag of the shares, overlaying the users home. Overlay, because if a .ref file is in a sub sub sub folder all nodes from the users root to the mount point need to calculate the ETag by combining the users home and the shadow tree. The challenge is keeping the shadow tree in sync when moving .ref files around, which is possible, but would require additional work. IMO it would be too fragile ... depending on the filesystem capabilites and how we are notfied. Well, we could fallback to the share marnager and query it for the mount points.
Or better yet, implement a share manager that uses your shadow tree to not only persist the shares, but also the ETag and calculated ETag in the filesystem. The data can only be changed by the apis, users have no direct access. Moving .ref files around in the filesystem will propagate an etag change up in the users home tree. Whenever the gateway encounters a reference in the listing of a folder it can check and update the path in the share manager.
Both types of changes are covered:

changes to shared files can be recursively discovered by the shadow tree and combining its ETags with the user home tree.
moving share mount points can be detected by the normal etag propagation in a users home tree AND the gateway can use references to update the share manager about changed mount points

@butonic the idea is of storing the reference files in the user home directory is apeeling however it has at least two drawbacks:

Users accessing the filesystem outside Reva can manipulate those files which are useless for them as they won't give you access to the shared resources, they are only files.
The cost of having such references spread across arbitrary places in the homedirectory is orders of magnitude bigger for two reasons: ETAG propagation across various arbitrary tree dephts compared to two and inefficient response inspections (with locked folder only paths starting with /home/MyShares/ are inspected rather than all responses)

On the workshop we have a good understanding to start with this model as it should work and will give us a very good baseline on performance, then we can explore the other model and compare.

Users accessing the filesystem outside Reva can manipulate those files which are useless for them as they won't give you access to the shared resources, they are only files.

We can store a JWT inside it to prevent users from messing with the data. Yo can consider the .ref file a feature, because it lets users see share mount points when doing an ls. There are ways to improve the user experience. The cli commands could be wrapped to understand .ref files or they could be transparently replaced with symbolic links an a fuse filesystem. In the first step I would only use them to persist them.

The cost of having such references spread across arbitrary places in the homedirectory is orders of magnitude bigger for two reasons:

ETAG propagation across various arbitrary tree dephts compared to two and

I agree, but you don't need to write ETags for intermediate nodes (between the root and the mount point). The share manager knows the full path to all mount points. We can combine the etag of the nodes in the home folder with the etag of mounted shares to calculate an ETag that changes when a shared file in another tree changes.

inefficient response inspections (with locked folder only paths starting with /home/MyShares/ are inspected rather than all responses)

We could cache the calculated ETag and only invalidate it when the storage containing the share changes. With your MyShares folder you still need to propagate the etag to all share recipients, which might be hundreds or even thousands if a group share is large enough. And you still need to calculate the etag for the root by combining the home storage root with the MyShares node. So the normal PROPFINDS have to calculate the ETag AFAIU.
IMO we need to further decouple share references from efficient polling.
The storage providers and storage drivers are responsible for persisting the shares.
The Share manager is responsible for making expensive operations like listing shares efficient.
The gateway should use the share manager to enrich the users home with ETags from shared nodes to allow efficient polling.


Option to only show bottom or top of a correlation matrix plot

Correlation matrix with option to hide half of it
Would be cool to have a really nice pre-packaged set of analysis tools in python, previously I do a lot of this EDA in R, but will fit far better into workflows if I can just reproduce the same EDA in python...(I have a few other ideas but this is a quick add I would think)
I propose adding an option to hide either the top or bottom of the plot that is outputted on top
What is the current outcome?
Not sure what to say here...
I think a quick implementation of this searborn example somewhere around here in nominal.py is a possible solution.
Is it backward-compatible?
Should be backward compatible, just changes the look of the plot to remove clutter.

Hey @jayceslesar ! The idea of plotting only specific rows or columns is awesome, and is actually something that's been on my mind for quite some time now, but didn't have the option to actually implement. If you're up to the challenge to take this PR the extra mile, that will be awesome, as I believe it will be mush more used and helpful than only selecting the top half, which I think might be a little arbitrary in some cases. What do you think?


The Chinese translation can not displayed as we want.

My laptop installed Windows 7(Japanese language), when I start to debug through visual studio 2015, switch language to Chinese, the menu can not be displayed as we want. Do you have any idea?
By the way, I downloaded the vault client from https://earth.vault.euclideon.com/ , the Chinese translation work well!

The process of loading the font takes significantly longer in debug, so we decided to reduce the available characters to just the minimum required for the English translation. You can temporarily change this code to view it in debug, however we won't accept PRs that modify this:
https://github.com/Euclideon/vaultclient/blob/7b2c929bf5d77f98e9414765520e84cdd9ac8dc8/src/main.cpp#L676-L709

Thanks @samsinsane , I closed it!


fmt: off blocks ignored, reformatted

Consider this code:
# fmt: off
leave_this_alone =       42
# fmt: on

It gets reformatted to this when running BlackConect with ctrl+alt+L:
# fmt: off
leave_this_alone = 42

# fmt: on

If I run black simply from the terminal to the same file, that works fine.
What am I missing?
(black: 22.10.0, python: 3.9.7, Widows10)

Not sure, probably blackd ignores some restrictions that you can put in the comments. Let me check and get back to you!

Strange, but I cannot reproduce it with the version of black you provided (22.10.0).
I pasted provided fragment into a Python file and called "Reformat Whole File" - nothing has been changed.
I tried reformatting it using "Reformat Selected Fragment" - still no change.
Maybe I am missing something? Could you please provide more context about this issue? E.g. can you reproduce it on non-Windows machine?

I've played around, and the problem only occurs if I select the "Trigger on code reformat" option.

If I use the "Trigger when saving changed files" option, everything works fine.

Thank you, this may help!

If I disable the BlackConnect plugin, stop blackd.exe, and perform a Code->Reformat File in PyCharm, the same happens. Based on this I'd say, that maybe the "Trigger on code reformat" option does not even invoke blackd.

It is actually a bit different: call to blackd does happen and for that specific example it returns "nothing to be done here" response. This is exactly what is expected in that situation.
However, the way integration with "Code Reformat" feature is written right now is a bit incorrect: in case call can not be made or no changes are needed it will trigger the default PyCharm coffeee reformatter.
I will see what can be done about this shortly!

Oh, I see. Thanks a lot!

After looking into this further, I see no way to prevent this from the plugin's side. Due to the integration being written as a "post-processor", the default PyCharm code formatter will always be called no matter what. :(
But there is a workaround:

In the Settings/Preferences dialog , go to Editor | Code Style.
Switch to the Formatter tab and enable the Turn formatter on/off with markers in code comments option.
Change default markers to fmt: on and fmt: off respectively.

In this way, you will get the same behaviour from both PyCharm and black.
In one of the following releases, the current method of integration in IDE's code reformatted will be replaced with a version that uses another mechanism entirely and will not have this issue.
Hope that helps!

Thanks, that'll do the trick.


FR: Add support for kotlin writte create/update views in blaze DGS support

Currently we only support java based Create/Update views in the com.blazebit.persistence.integration.graphql.dgs.mapper.EntityViewInputObjectMappersince mapToKotlinObject does just call the super implementation without handling entity views.
We should add support for kotlin based create/update views too.
It might be that we extract the logic with com.blazebit.persistence.integration.graphql.dgs.mapper.EntityViewInputObjectMapper#mapToJavaObject and can call this from both, just the default implementation of the primitive types must then either go to super.mapToJavaObject or super.mapToKotlinObject respectively

We will try to implement this @beikov - i would suggest doing so after the sb 3.0 effort so we do not open to many unfinished efforts. What are your thoughts?

Sounds good to me


Fix mutation behavior of position

This PR makes position(pos, PH) with PH::AbstractProjTTN copy safe. A test example verifying copy safety is provided.

Any idea why the test was canceled?

Any idea why the test was canceled?

No idea, could just be an issue on Github's side. We can try restarting them.

Seems like we could change all uses of ::TTN to ::AbstractTTN to make it more general.

I think overall the new code is definitely an improvement, but needs a cleanup and style improvements based on my comments.

Codecov Report
Attention: 22 lines in your changes are missing coverage. Please review.

Comparison is base (56fcb93) 72.90% compared to head (8b0ca9f) 72.43%.
Report is 3 commits behind head on main.

Files
Patch %
Lines

src/treetensornetworks/projttns/projttn_apply.jl
71.42%
10 Missing :warning:

src/treetensornetworks/projttns/abstractprojttn.jl
80.55%
7 Missing :warning:

src/treetensornetworks/projttns/projttn.jl
84.84%
5 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #131      +/-   ##
==========================================
- Coverage   72.90%   72.43%   -0.47%
==========================================
  Files          71       68       -3
  Lines        4097     4020      -77
==========================================
- Hits         2987     2912      -75
+ Misses       1110     1108       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Looks good, thanks @b-kloss. This kind of cleanup/refactor will help when generalizing to loopy graphs with BP (@JoeyT1994).