[4.11] Kubelet consumes a lot of CPU and hangs after running a lot of cronjobs

[vrutkovs] See below for thread summary
Hi team,
After upgrading to 4.11 we faced to new issue:
We use cronjobs with '* * * * *'. After upgrading some pods from cronjobs stuck in "ContainerCreating" or "Init 0/1" status.
In pod describe we can see:
`Events:
Type     Reason                    Age        From               Message

Normal   Scheduled                 51s        default-scheduler  Successfully assigned 0xbet-prod/podname7673773-q6bq5 to ip-10-0-216-195.eu-central-1.compute.internal by ip-10-0-201-118
Warning  FailedCreatePodContainer  9s         kubelet            unable to ensure pod container exists: failed to create container for [kubepods burstable pod3847723b-b7c8-4adc-a9d7-f3cdb83ae03f] : Timeout waiting for systemd to create kubepods-burstable-pod3847723b_b7c8_4adc_a9d7_f3cdb83ae03f.slice
Normal   AddedInterface              multus             Add eth0 [<IP_ADDRESS>/23] from ovn-kubernetes
Normal   Pulled                      kubelet            Container image "ghcr.io/banzaicloud/vault-env:1.13.0" already present on machine
Normal   Created                     kubelet            Created container copy-vault-env
Normal   Started                     kubelet            Started container copy-vault-env
.... `
The symptom is pod scheduling slower and slower and after some time it stuck on
  Normal   Scheduled                 51s        default-scheduler  Successfully assigned 0xbet-prod/podname7673773-q6bq5 to ip-10-0-216-195.eu-central-1.compute.internal by ip-10-0-201-118
While login to node , I can see this in journalctl:
`Aug 13 21:38:49 ip-10-0-216-195 hyperkube[1546]: I0813 21:38:49.823654    1546 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable podbcc994a0-6720-48fc-889b

Aug 13 21:38:49 ip-10-0-216-195 hyperkube[1546]: I0813 21:38:49.834117    1546 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod36584156-6776-410f-861b

Aug 13 21:38:49 ip-10-0-216-195 hyperkube[1546]: I0813 21:38:49.835958    1546 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod24ca3b44-6fb3-47be-b111

Aug 13 21:38:49 ip-10-0-216-195 hyperkube[1546]: I0813 21:38:49.837512    1546 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod23e81c45-5fd2-4e7a-975c`
Reboot helps, but not for a long time.
As result pods can't schedule and stuck on whole cluster.

Short thread summary.
What we know so far:

bug is happening on any new container starts, cronjob is an easy way to trigger it, but other ways to create new pods would also show the same behaviour
kubelet seems to use a lot of CPU and RAM
switching to cgroupsv2 seems to stop RAM bleeding, but doesn't help with CPU load

Note that fresh OKD 4.9 installs are using cgroupsv2 by default, upgraded cluster need to be switched manually

Probable cause:

latest kernel is known to leak blkios, this is mitigated by the workaround (see below)

Workaround: https://github.com/okd-project/okd/issues/1310#issuecomment-1312848841 - thanks to @framelnl there's a DaemonSet which can clean up the extra cgroups
Upstream issue refs:

https://github.com/kubernetes/kubernetes/issues/106957#issuecomment-1147441007
https://github.com/kubernetes/kubernetes/pull/113050 (1.25 cherrypick of crio fix)

https://dropmefiles.com/6nn0s must-gather

Also noticed this on kubelet 1.24 and the situation gets worse the more pods are running.
Its not OKD specific, seems to be a dupe of https://issues.redhat.com/browse/OCPBUGSM-39381

This is most likely a runc regression. Could you check if switching to runc helps. Example MachineConfig
Seems we'd need runc 1.1.3, which has two systemd/cgroups fixes

This is most likely a runc regression. Could you check if switching to runc helps (make sure you apply it to all MachineConfigPools). Example MachineConfig
Seems we'd need runc 1.1.3, which has two systemd/cgroups fixes

@vrutkovs
Hi, Vadim!
I have applied your machineconfig and recreate cron nodes.
Will report about results

@vrutkovs
Hi Vadim,
Seems it hasn't helped. Worked fine for 3h, but now one node returned to failed state:

Normal   Scheduled                 4m55s  default-scheduler  Successfully assigned bank-prod/bank-cronjob-27674577-7wjld to ip-10-0-217-73.eu-central-1.compute.internal by ip-10-0-133-210
Warning  FailedCreatePodContainer  3m16s  kubelet            unable to ensure pod container exists: failed to create container for [kubepods burstable pod103107db-62e6-4f67-b2fd-50a066f4af36] : Timeout waiting for systemd to create kubepods-burstable-pod103107db_62e6_4f67_b2fd_50a066f4af36.slice

Seems that kube-rbac-proxy began to eat memory before the case.

Okay, it looks like kubelet leaking memory, not runc

The same issue here. Process:
root        2009       1 73 Aug12 ?        3-00:33:07 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=fedora --node-ip=<IP_ADDRESS> --minimum-container-ttl-duration=6m0s --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --cloud-provider= --hostname-override= --provider-id= --pod-infra-container-image=quay.io/openshift/okd-content@sha256:c4e32171a302b1a0d21f936b795b9505b992404b6335bb7e63d3b1bddc0b91ab --system-reserved=cpu=500m,memory=1Gi --v=2

Hi,
Also experiencing the same issue in multiple test 4.11 clusters, we have ~30 production 4.10 clusters and have not seen this issue there at all.

root        1458 74.9 72.8 22674508 11924036 ?   Ssl  Aug02 14736:44 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=fedora --node-ip=<IP_ADDRESS> --minimum-container-ttl-duration=6m0s --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --cloud-provider= --hostname-override= --provider-id= --pod-infra-container-image=quay.io/openshift/okd-content@sha256:c4e32171a302b1a0d21f936b795b9505b992404b6335bb7e63d3b1bddc0b91ab --system-reserved=cpu=500m,memory=1Gi --v=2

Switched to crun and not seeing staggering the memory growth here. Its probably visible on nodes with many containers running?

Switched to crun and not seeing staggering the memory growth here. Its probably visible on nodes with many containers running?

As for me, I have switched to crun when you wrote and haven't rolled it back
I rewrote the application for removing k8s cronjobs (packed them into container with crontab) and cluster started to be stable. So it seems the reason not in the number of running pods, but in 'starting' pods.
Also I noticed that when I had a lot of cronjobs the node degradation went for this way (from k8s side):

Normal state: container scheduled to the node and started in 1-3s
Degradation started: container scheduled to the node and started in  20-60s.
Degraded state: container scheduled to the node and stuck.
After some time the affected node changes a status to NotReady. Pods eviction started. All pods changed their state to Terminating. This state can be fixed only if you hard reboot the node or remove machine.

on steps 1, 2 and 3 all running containers work fine on this node.

It seems there are fixes that seems related to this problem in the kubernetes v1.24.4 releases notes
Specificaly:

Fix JobTrackingWithFinalizers when a pod succeeds after the job is considered failed, which led to API conflicts that blocked finishing the job. (https://github.com/kubernetes/kubernetes/pull/111664, @alculquicondor) [SIG Apps and Testing]
Fix memory leak in the job controller related to JobTrackingWithFinalizers (https://github.com/kubernetes/kubernetes/pull/111722, @alculquicondor) [SIG Apps]
Fixed potential scheduler crash when scheduling with unsatisfied nodes in PodTopologySpread. (https://github.com/kubernetes/kubernetes/pull/111511, @kerthcet) [SIG Scheduling]

Excellent, thanks. There's a PR open for 1.24.3 - https://github.com/openshift/kubernetes/pull/1326 - hopefully it would be soon superceded with .4 and merged soon. Once it happens we'll pick it up in okd-machine-os build and release a new stable

https://amd64.origin.releases.ci.openshift.org/releasestream/4-stable/release/4.11.0-0.okd-2022-08-20-022919 includes runc 1.1.3, which should have some kubelet-related fixes.

cronjob pinned to one node to scale an alpine deployment between 50 and 0 replicas and reverse, every minute, this test shows the problem within a few hours on 4.11 clusters. no problem found on 4.10 clusters over multiple days.
Screenshot above is from 4.11 cluster, crun machineconfig was added to workers 2022-08-21 (confirmed with crio-status config), Updated cluster to 4.11.0-0.okd-2022-08-20-022919 at ~2022-08-21 12:00, crun machineconfig was removed from workers, rendered config shows creation 21 Aug 2022, 20:43, neither crun, nor runc version 1.1.3 seem to have had effect on kubelet problem.
Kubelet version in 4.11.0-0.okd-2022-08-20-022919 now v1.24.0+4f0dd4d
Below screenshot from 4.10 cluster running same cronjob since 2022-08-19

It also seem to happen with AWX starting Ansible playbooks as pods. I see this every night as a lot of jobs start after midnight. A node reboot fixed the memory usage. Also, no problems on 4.10 clusters.

1.24.4 bump - https://github.com/openshift/kubernetes/pull/1352

By the way, Kubelet consumes also more and more CPU usage in the long run on OKD Version 4.10.0-0.okd-2022-07-09-073606 (vSphere IPI). You can easily reproduce if you are using for example "Red Hat OpenShift Logging" Operator, because it continuously executes cron tasks. But of course, the CPU load of Kubelet is nowhere near as severe as with the version OKD Version 4.11.0-0.okd-2022-08-20-022919.
I think the CPU usage of Kubelet also very depended of how much pods are deployed within the cluster/node. (Maybe the overhead of vSphere IPI certainly plays a part in this as well. Compared to the bare metal installation.)
After restarting Kubelet on a node, the overall node CPU usage drops from about 3 to 1 core. This is at least for us, besides the node restart, a workaround for this unwanted behavior.

Known issue: kubelet with CRI-O not cleaning up cgroups etc when pod gets deleted.
https://github.com/kubernetes/kubernetes/issues/106957
Fix: https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f

@vrutkovs is it possible to include this fix?

It is rather a workaround so probably not fit to include in release. My bad, wrong wording.
It also should be solved in next release I hope. That is wat the remark '1.24.4 bump' is about, I guess. The issue is still open.
Just wanted to point out that this bug is around for many months. Not sure it has always been the same bug but it can be worked around with that nice work from Andrei.

We won't be including a workaround officially - its a kubelet problem and we're waiting for fixes to land, no need to digress from upstream too far.
I don't mind including this workaround as "official" mitigation recipe, if it has been verified to work

I can't say for sure it still works in 4.11 but it stabilizes our 'cron job hungry' 4.9 and 4.10 clusters. I will test it in 4.11 and let it know here.

It most definitely still cleans up a lot of left over cgroups. Tail of the daemon running on a 4.11 node that runs 10 short lived pods every minute:
2022-09-29T07:29:19+00:00 Removing CGROUP crio-f330b2f8f55951bb5f5445bcd6fa811bdca11447cdfd91b044647e3b846ba4ff.scope and its parent...
2022-09-29T07:29:19+00:00 Scope crio-f6d78fac4951a3296c7e6b118f9e67a0dcb56766005b5bef6776f4ea48652173.scope found under running pod, skipping...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-f86a48b76ea5a734428e11aa18e3a4f3fc9dbfc3894008dcde1d539ac5b50438.scope and its parent...
2022-09-29T07:29:19+00:00 Scope crio-f8f49053255d505d7d6bd2f018dc1985ebf3c34aee673abdd640404db16ef070.scope found under running pod, skipping...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-f8feb93e6aad0dc6b0d7c3dca881a3c5bdd601eff3a60a24a54e8c622abbfd57.scope and its parent...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-fab2dc0351b831ca6b1395e2721b60d49e10fe043c2d75cbbaa26c9c70ee2ef1.scope and its parent...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-fbc1d4ddc0707b6e8e06d4f87dbb6e8adf7c922e07275ad61d5934396a9d41fb.scope and its parent...
2022-09-29T07:29:19+00:00 Scope crio-fbcc50052c55de371d2927d4deb8f5bba8a7b9129e69e220cbadba4e33a93c10.scope found under running pod, skipping...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-fde14b0c95048cdd7c7fa91b9cef3bd3fe8e8120f2c7ae83c330a6a423a2971f.scope and its parent...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-ff49445e58458cb366245cbf1b85c72ba7e8a0868561505eb3355b7ce91f9154.scope and its parent...
2022-09-29T07:29:19+00:00 Removing CGROUP crio-ff6a76f198151084777c816edaea1ad6f2ec1d0acd10ca7e04fb451d89ec841c.scope and its parent...
2022-09-29T07:29:19+00:00 Sleeping for 600 seconds...

System memory spikes and keeps increasing after disabling the workaround, while running the test:

When re-enabling the workaround I see this pattern:

The workaround sleeps 10 minutes between runs, which explains the sawtooth pattern I guess.
Credits to @aneagoe

I put the workaround on a test cluster on tuesday, and the issue has not gone away there, and actually it seems to increase the log spam. i did not use the daemon set, instead i tossed it into a script and am running it on the nodes directly at the moment.
It is going through and appears to be doing what its supposed to, as an example:
Sep 29 07:13:36 MYSERVER.com check.sh[1119]: 2022-09-29T12:13:36+00:00 Starting k8s garbage collector run...
Sep 29 07:13:37 MYSERVER.com check.sh[1119]: 2022-09-29T12:13:37+00:00 Found POD MYSERVERcom-debug unknown to k8s control plane and without any PIDs, will delete it...
Sep 29 07:13:37 MYSERVER.com check.sh[3239730]: Stopped sandbox e62be3eacc7e2286012ff67f2aa969a6131fb4b87ac9d24f7eb98ae08cd346b3
Sep 29 07:13:37 MYSERVER.com check.sh[3239737]: Removed sandbox e62be3eacc7e2286012ff67f2aa969a6131fb4b87ac9d24f7eb98ae08cd346b3

But the memory leaks are still there - I lost two nodes (one control plane, one worker) last night to low memory on this cluster. the CPU usage is still high (3-4 cores normally on these nodes, which are all 4 core) on nodes that are building up.
And while i still get the
pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort p ........
logs, as well as the
kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/e9da40e0-99ac-40d6-802a- .........
logs, i now also get new errors from systemd about the cgroups that the workaround removed:
systemd[1]: kubepods-burstable-pod4f247343_7b97_45fe_899b_9b023d5316cf.slice: Failed to open /run/systemd/transient/kubepods-burstable-pod4f247343_7b97_45fe_899b_9b023d5316cf.slice: No such file or directory
This specific cluster i am testing the workaround on has barely any workloads left on it, no jobs (besides the couple built-in ones). but i have some automation that is launching numerous debug pods with regularity, which seems to cause the same symptoms as the cronjobs do (numerous temporary containers)
For me, restarting kubelet helps with the memory right away, but doesnt help the CPU, and gives unpredictable results (unable to launch pods, cannot get logs, cant connect to terminal of pods etc). so i have just been nicely draining and rebooting nodes automatically when the RAM usage gets too high. but before the oom reaper kicks in, as he wont touch kubelet and it just makes the situation worse as the cluster doesnt realize apps have been killed.

Exact same version here in my test cluster. To me this different behaviour does not make sense. I monitored the workaround for a long time some months ago to make sure it worked and didn't cause problems and it seemed to do so...
I will continue my test for longer time and watch for what you are describing and the cluster stability.

Exact same version here in my test cluster. To me this different behaviour does not make sense. I monitored the workaround for a long time some months ago to make sure it worked and didn't cause problems and it seemed to do so...
I will continue my test for longer time and watch for what you are describing and the cluster stability.

I'm dropping it on a second cluster right now, to see if I see different results there. one of those nodes is actually in it's slow downward spiral as i type this (spawned the SystemMemeoryExceedsReservation alert 30 minutes ago, which is a good forebearer of the memory leak taking over). but it has a lot of ram to burn through yet, so would take it awhile

I'm dropping it on a second cluster right now, to see if I see different results there. one of those nodes is actually in it's slow downward spiral as i type this (spawned the SystemMemeoryExceedsReservation alert 30 minutes ago, which is a good forebearer of the memory leak taking over). but it has a lot of ram to burn through yet, so would take it awhile

Indeed, I was also seeing that for months before I found some help (the workaround) in the openshift-users slack channel: https://kubernetes.slack.com/archives/C6AD6JM17/p1653030671315219

You can clearly see the problem and the garbage collecting every 10 minutes using a simple find
# pwd
/sys/fs/cgroup
# find . -type d | wc -l
463

The cgroups (and other things) don't get cleaned up by kubelet so the folder count keeps increasing until the garbage collectors drops by... The workaround does clean that up, at least in my 4.11 test cluster.
I will let my test run for a few days because it seems like I have a little bit of memory leaking going on in 4.11, even with the workaround in place but not sure yet.

I'm still watching it on this second cluster i added it to as well. I had some results about 10 or 15 minutes after starting the garbage collector, but the SystemMemoryExceedsReservation alert never went away, and now two hours later the memory usage is starting to climb back up again. The g/c is still running, and still cleaning things up once in awhile.

Looks like memory is filling up, even when running the workaroud :-(

Definitely show stopper for us to move to 4.11 if the node crashes eventually, which it probably will do. I will try to find out what is staying behind this time.
This memory leak is going on since we started using OKD 4.
Maybe OKD should swap crio with containerd, which upstream clearly supports better..

Edit: here it is 24 hours later. it seems the garbage collector must help, but it still spikes for awhile before dropping suddenly. So while this is indeed prolonging the uptime, it doesnt seem to fix it for good, and after another spike or two it will still OOM I would suspect. It's on it's third upward climb as we speak. the SystemMemoryExceedsReservation alert has never gone away since it started climbing 24 hours ago either.

On bigger clusters it is not abnormal the default 'SystemMemory' limit is too low. There is a RH issue explaining how to adjust the limit to your needs. We have also 1 production host that is always in 'SysMem' alert but that is not causing problems.

I am still not sure about my test cluster. The RSS of the system slice keeps on going up very slowly but the total memory usage is not... I am still hoping it is memory that gets flushed eventually by crio or kubelet. Not sure where the RSS is building up yet.
Time will tell.

My test kept on running over the weekend and I think we can conclude the memory usage stabilises. And thus the workaround is still doing its thing for us. So I believe this are the same bugs that are around since many versions.
For us no problem to move to 4.11.

The graph might be misleading: I also added workload in our testing cluster so gues some of the increase in 'system' slice memory can be attributed to that. Most important is that it seems to flatline at some point.

My test kept on running over the weekend and I think we can conclude the memory usage stabilises. And thus the workaround is still doing its thing for us. So I believe this are the same bugs that are around since many versions.
For us no problem to move to 4.11.
The graph might be misleading: I also added workload in our testing cluster so gues some of the increase in 'system' slice memory can be attributed to that. Most important is that it seems to flatline at some point.

Testing the patch on OKD  4.11.0-0.okd-2022-08-20-022919, nodes crash due to memory exhaust after three days.
Hope k8s 1.24.6 will coming soon ?
Thank-you very much for the work.

Hope k8s 1.24.6 will coming soon ?

It does not look like it's getting backported https://github.com/kubernetes/kubernetes/commits/v1.24.7-rc.0

It looks that k8s 1.24.6 has been merged to openshift/kubernetes:release-4.11: https://github.com/openshift/kubernetes/pull/1381
When can we expect new release of OKD?

We're waiting for it to be promoted in OCP nightlies and bumped up in machine-os-content. Hopefully that'd be this or the following weekend

It doesn't look like 9its fixing the leak. I've used a reproducer in https://github.com/okd-project/okd/issues/1310#issuecomment-1221965431 and I still see used memory climbing.

My prom-foo is not good enough to see if it climbs slower though

Also I upgraded to 4.12 (w/ workaround) and applied https://github.com/kubernetes/kubernetes/pull/108855 (using https://github.com/vrutkovs/custom-okd-os/pkgs/container/custom-okd-os/45544090?tag=custom-kubelet) - that seems to have stopped the memory growth and the log spam

@vrutkovs @tyronewilsonfh can you share the reproducer used from https://github.com/kubernetes/kubernetes/pull/108855?
I've been trying to repro this as part of https://github.com/kubernetes/kubernetes/issues/112151 but haven't had too much luck yet.

I used https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#creating-a-cron-job to trigger memory leak

Will this help to create a 4.11 without this bug (backports of the fixed components)? Or what is the way forward?

Will this help to create a 4.11 without this bug (backports of the fixed components)? Or what is the way forward?
+1 for this. It would be great to understand the correct procedure to get our stable 4.10 cluster upgraded safely to 4.11.

We're going to need to backport https://github.com/kubernetes/kubernetes/pull/108855 all the way to 1.24 to have it fixed.
Meanwhile seems the workaround is to switch back to cgroups1 (add systemd.unified_cgroup_hierarchy=0 to kernel params) - could anyone confirm that?

I have what seems to be this issue (in the form of slowly increasing CPU usage requiring rebooting nodes) with kubelet and often also olm.  As the cluster dates from cgroups1 days, and has been upgraded I am running cgroups1 (mount | grep group confirms.
cgroups2 is present, but not being used.

I am running cgroups1 (mount | grep group confirms. cgroups2 is present, but not being used.

same here

sorry, if the question is irrelevant, but does latest stabel 4.11.0-0.okd-2022-10-15 release fix the issue in any way @vrutkovs ?
Over the weekend 2/3 masters in my 4.11.0-0.okd-2022-08-20-022919 production cluster were crushed due to memory leakage. Hopefully I checked the alerts Saturday between the machines crushes :).
I also checked that without any modifications system seems to use both, cgroups version 1 and 2. Should I switch to cgroups v1 as you proposed? For reasons I would like not to mention, I can't use my test cluster for that, so I would have to the direct changes in production, what I would prefer not to do without knowing that it would help...

does latest stabel 4.11.0-0.okd-2022-10-15 release fix the issue in any way

Most likely - not. It includes a memory leak in job controller, but seems more leaks are present:

/var/run issue - https://github.com/kubernetes/kubernetes/issues/106957#issuecomment-1147441007
backport of https://github.com/kubernetes/kubernetes/pull/108855 to 1.24

sorry, if the question is irrelevant, but does latest stabel 4.11.0-0.okd-2022-10-15 release fix the issue in any way @vrutkovs ? Over the weekend 2/3 masters in my 4.11.0-0.okd-2022-08-20-022919 production cluster were crushed due to memory leakage. Hopefully I checked the alerts Saturday between the machines crushes :).
I also checked that without any modifications system seems to use both, cgroups version 1 and 2. Should I switch to cgroups v1 as you proposed? For reasons I would like not to mention, I can't use my test cluster for that, so I would have to the direct changes in production, what I would prefer not to do without knowing that it would help...

It does not seem to fix it.  I upgraded last night and awoke to 2 worker nodes unresponsive.
@vrutkovs care to share your upgrade directions for OKD 4.12?

So we'll have to wait a little longer ...
I have noticed BTW that when I see that kubelet memory climbs up a lot and I then only restart kubelet, that will cure the memory problem but the node will crash later anyway. Probably due to cgroups not being cleaned up?
So I keep rebooting the nodes when the problem occurs. That then usually helps for that node for about 3 days or so.

We see same or similar problems in a 4.9 production cluster: the most busy (= constantly creating containers due to crons) bare metal node crashes hard after a few months of operation. We can postpone this using the workaround that cleans up cgroups (and other things) but eventually the node still crashes..
So guess the workaround is draining an rebooting the most busy node every few months until we can upgrade to a stable OKD release, hopefully soon.

So we'll have to wait a little longer ...
I have noticed BTW that when I see that kubelet memory climbs up a lot and I then only restart kubelet, that will cure the memory problem but the node will crash later anyway. Probably due to cgroups not being cleaned up?
So I keep rebooting the nodes when the problem occurs. That then usually helps for that node for about 3 days or so.

BTW, does anyone see this problem also on OCP 4.11?

Hi,
We're running 4.11.0-0.okd-2022-07-29-154152 and can confirm that this issue is happening for us.

(I was asking for OCP = OpenShift 4.11.)

Experiencing accidental memory consumption growth after updating to 4.11.0-0.okd-2022-10-15-073651 and switching on "systemd.unified_cgroup_hierarchy=0". It happens randomly on different nodes

Right, that seems to be cgroupsv2 related then

BTW, does anyone see this problem also on OCP 4.11?

Nobody knows or wants to share it? I will spend some time in my weekends to evaluate OCP.. Curious about the differences of core components. I know about the OS.

BTW, does anyone see this problem also on OCP 4.11?

Let's see for ourself ;-)
"You have 60 days left to try
Red Hat® OpenShift® Container Platform."

After removing "systemd.unified_cgroup_hierarchy=0" from MachineConfigPool (switching back to default cgroup v2) no memory issues so far. @vrutkovs is there a chance it's fixed in 4.11.0-0.okd-2022-10-28-153352?

Its unlikely, no kubelet/systemd bump:

Upgraded packages
```
Upgraded:
  NetworkManager 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  NetworkManager-cloud-setup 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  NetworkManager-libnm 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  NetworkManager-ovs 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  NetworkManager-team 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  NetworkManager-tui 1:1.38.4-1.fc36 -> 1:1.38.6-1.fc36
  amd-gpu-firmware 20220913-140.fc36 -> 20221012-141.fc36
  bash 5.2.2-1.fc36 -> 5.2.2-2.fc36
  btrfs-progs 5.18-1.fc36 -> 6.0-1.fc36
  catatonit 0.1.7-5.fc36 -> 0.1.7-10.fc36
  chrony 4.2-5.fc36 -> 4.3-1.fc36
  conmon 2:2.1.4-2.fc36 -> 2:2.1.4-3.fc36
  coreos-installer 0.16.0-1.fc36 -> 0.16.1-2.fc36
  coreos-installer-bootinfra 0.16.0-1.fc36 -> 0.16.1-2.fc36
  dbus 1:1.14.0-1.fc36 -> 1:1.14.4-1.fc36
  dbus-common 1:1.14.0-1.fc36 -> 1:1.14.4-1.fc36
  dbus-libs 1:1.14.0-1.fc36 -> 1:1.14.4-1.fc36
  ethtool 2:5.19-1.fc36 -> 2:6.0-1.fc36
  fedora-release-common 36-18 -> 36-20
  fedora-release-coreos 36-18 -> 36-20
  fedora-release-identity-coreos 36-18 -> 36-20
  fuse-overlayfs 1.9-1.fc36 -> 1.9-6.fc36
  git-core 2.37.3-1.fc36 -> 2.38.1-1.fc36
  glibc 2.35-17.fc36 -> 2.35-20.fc36
  glibc-common 2.35-17.fc36 -> 2.35-20.fc36
  glibc-minimal-langpack 2.35-17.fc36 -> 2.35-20.fc36
  gnutls 3.7.7-1.fc36 -> 3.7.8-2.fc36
  grub2-common 1:2.06-53.fc36 -> 1:2.06-54.fc36
  grub2-efi-x64 1:2.06-53.fc36 -> 1:2.06-54.fc36
  grub2-pc 1:2.06-53.fc36 -> 1:2.06-54.fc36
  grub2-pc-modules 1:2.06-53.fc36 -> 1:2.06-54.fc36
  grub2-tools 1:2.06-53.fc36 -> 1:2.06-54.fc36
  grub2-tools-minimal 1:2.06-53.fc36 -> 1:2.06-54.fc36
  intel-gpu-firmware 20220913-140.fc36 -> 20221012-141.fc36
  kernel 5.19.14-200.fc36 -> 5.19.16-200.fc36
  kernel-core 5.19.14-200.fc36 -> 5.19.16-200.fc36
  kernel-modules 5.19.14-200.fc36 -> 5.19.16-200.fc36
  libidn2 2.3.3-1.fc36 -> 2.3.4-1.fc36
  libksba 1.6.0-3.fc36 -> 1.6.2-1.fc36
  libmaxminddb 1.6.0-2.fc36 -> 1.7.1-1.fc36
  libsmbclient 2:4.16.5-0.fc36 -> 2:4.16.6-0.fc36
  libwbclient 2:4.16.5-0.fc36 -> 2:4.16.6-0.fc36
  libxml2 2.9.14-1.fc36 -> 2.10.3-1.fc36
  linux-firmware 20220913-140.fc36 -> 20221012-141.fc36
  linux-firmware-whence 20220913-140.fc36 -> 20221012-141.fc36
  nvidia-gpu-firmware 20220913-140.fc36 -> 20221012-141.fc36
  procps-ng 3.3.17-4.fc36 -> 3.3.17-4.fc36.1
  qemu-guest-agent 2:6.2.0-15.fc36 -> 2:6.2.0-16.fc36
  rpm-ostree 2022.13-1.fc36 -> 2022.14-1.fc36
  rpm-ostree-libs 2022.13-1.fc36 -> 2022.14-1.fc36
  rsync 3.2.6-1.fc36 -> 3.2.7-1.fc36
  runc 2:1.1.3-1.fc36 -> 2:1.1.4-1.fc36
  samba-client-libs 2:4.16.5-0.fc36 -> 2:4.16.6-0.fc36
  samba-common 2:4.16.5-0.fc36 -> 2:4.16.6-0.fc36
  samba-common-libs 2:4.16.5-0.fc36 -> 2:4.16.6-0.fc36
  skopeo 1:1.9.2-1.fc36 -> 1:1.10.0-3.fc36
  ssh-key-dir 0.1.3-2.fc36 -> 0.1.4-1.fc36
  tzdata 2022d-1.fc36 -> 2022e-1.fc36
  unbound-libs 1.16.3-1.fc36 -> 1.16.3-2.fc36
  vim-data 2:9.0.475-1.fc36 -> 2:9.0.803-1.fc36
  vim-minimal 2:9.0.475-1.fc36 -> 2:9.0.803-1.fc36
  xmlsec1 1.2.33-2.fc36 -> 1.2.33-3.fc36
  xmlsec1-openssl 1.2.33-2.fc36 -> 1.2.33-3.fc36
```

My home cluster has been fairly stable last release too (with /var/run -> /run workaround), so lets wait for more reports

I am testing a lot of 'hello' cronjobs in my home Openshift 4.11.9 cluster. Time will tell if my little VMs will remain stable.
But the first thing that stands out for me comparing latest Openshift versus OKD is the crio version: '1.24.3-4.rhaos4.11.git0e72422.el8' versus 1.24.0.
Any reason why OKD lags behind? If I check other components (kernel, runc) OKD is usually ahead.
Should study the changelogs but it seems obvious to me that a .3 will have fixed some bugs compared to the .0.
[core@master1 ~]$ crio version && runc -v
INFO[2022-11-01 04:51:22.893623628Z] Starting CRI-O, version: 1.24.3-4.rhaos4.11.git0e72422.el8, git: ()
Version:          1.24.3-4.rhaos4.11.git0e72422.el8
GoVersion:        go1.18.4
Compiler:         gc
Platform:         linux/amd64
Linkmode:         dynamic
BuildTags:        exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false
runc version 1.1.2
spec: 1.0.2-dev
go: go1.18
libseccomp: 2.5.2
[core@mec-okdtest-master-01 ~]$ crio version && runc -v
INFO[2022-11-01 04:51:37.165262983Z] Starting CRI-O, version: 1.24.0, git: ()
Version:          1.24.0
GoVersion:        go1.18
Compiler:         gc
Platform:         linux/amd64
Linkmode:         dynamic
BuildTags:        seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false
runc version 1.1.4
spec: 1.0.2-dev
go: go1.18.7
libseccomp: 2.5.3

OKD uses CRI-O module from Fedora repos. There are newer builds in Koji, which were never submitted to Bodhi and never got to actual repos (@LorbusChris, should we revert this? This clearly makes maintainers do more work and now OKD lags behind)

It seems Openshift 4.11.9 is more stable than the latest OKD. Openshift has a smaller load but should be similar enough.
I run in both the cleanup DaemonSet, also seems to remove CRIO stuff in Openshift, but it does not find left behind cgroups, which is a clear difference with OKD in my clusters.
Running 3 'hello' cron jobs I get these 'system slice' memory usage graphs.
Openshift vs OKD system slice memory usage:

I have a test/demo cluster with OCP 4.10 available and thought about upgrading it to 4.11 to see what happens.
However, Red Hat also seems to not recommend upgrading to 4.11?

Possibly also related https://github.com/coreos/fedora-coreos-tracker/issues/1330

Latest 4.11 includes CRIO 1.24.3 but that does not seem to make a difference for my test cluster: cgroups still seem to be left behind (maybe because it is using cgroups v2?) and memory slowly but surely fills up running 3 'hello' cron jobs

I was now able to updated the cluster mentioned above (update block vanished away) two days ago to OCP 4.11.9.
I have not seen the problem on this cluster yet, but it's only been 2 days so far.

The next OKD release of 4.11, 2022-11-05 is there.
Same Kubernetes version, so it's not supposed to fix this problem yet, either?

The next OKD release of 4.11, 2022-11-05 is there. Same Kubernetes version, so it's not supposed to fix this problem yet, either?

Nope, my previous reply was talking about latest version..

OKD version: 4.11.0-0.okd-2022-11-05-030711
Have been testing effect on memory usage with cgroups v1 and cgroups v2
Below steps done so far:
cgroups v2 machineconfig applied at ~12:00 (https://docs.okd.io/4.11/post_installation_configuration/machine-configuration-tasks.html#nodes-nodes-cgroups-2_post-install-machine-configuration-tasks)
cronjob started at ~14:00 which scales an alpine deployment between 100 and 0 replicas every minute (targets a single node)
cgroups v2 machine config deleted at ~17:00, rebooted node anytime it was becoming unresponsive
cgroups v2 machine config re-added ~22:00
Restarted node this morning and halved cronjob test values to see how this is after a longer period of time
From all the suggestions here, switching to cgroups v2 seems to have had most impact so far for me with this specific test, will keep this test going for a week and check if I still have to restart other nodes in this cluster now they are all using cgroups v2

Excellent, thanks. Could you check if a workaround from https://github.com/coreos/fedora-coreos-tracker/issues/1330#issuecomment-1299330024 (disabling blkio controller)?

Excellent, thanks. Could you check if a workaround from coreos/fedora-coreos-tracker#1330 (comment) (disabling blkio controller)?

Should be something like that right?
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-disable-blkio
spec:
  config:
    ignition:
      version: 3.3.0
    shouldExist:
      - systemd.unified_cgroup_hierarchy=0
    systemd:
      units:
        - contents: |
            [Slice]
            DisableControllers=blkio
          name: "-.slice"

The problem affects master nodes too, so the MC would need to be applied to these as well.

All my OKD clusters (4.9, 4.10 and 4.11) have cgroups v2. Can't seem to enable v1 on latest 4.11 test cluster with kernel 6... How is it possible others are running v1? Older machineconfigs that get dragged along with upgrades. Can't remember I did something custom that might affect this.
mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)

All my OKD clusters (4.9, 4.10 and 4.11) have cgroups v2. Can't seem to enable v1 on latest 4.11 test cluster with kernel 6... How is it possible others are running v1? Older machineconfigs that get dragged along with upgrades. Can't remember I did something custom that might affect this.
mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)

If you have java applications running on an old version. It does not support cgroup v2.
You have to force the use of cgroup v1 using this MC:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-openshift-machineconfig-worker-kargs
spec:
  kernelArguments:
    - systemd.unified_cgroup_hierarchy=0
EOF

@msteenhu We managed to keep cgroupsv1 with the solution from #1002

With cgroupsv1 and that 'no blkio' '-.slice' file my 4.11 test cluster also leaks memory. Steeper curve comparing with cgroups v2 indeed but leaking it does, unfortunately.

Excellent, thanks. Could you check if a workaround from coreos/fedora-coreos-tracker#1330 (comment) (disabling blkio controller)?

Have applied the below and tested this afternoon for few hours
kind: MachineConfig
metadata:
 labels:
   machineconfiguration.openshift.io/role: worker
 name: 99-worker-disable-blkio
spec:
 config:
   ignition:
     version: 3.2.0
   shouldExist:
     - systemd.unified_cgroup_hierarchy=0
   systemd:
     units:
       - contents: |
           [Slice]
           DisableControllers=blkio
         name: "-.slice"```
Still see ```unable to destroy cgroup paths for cgroup [kubepods besteffort podd0de
061d-728e-4305-bb31-5f009ad1fe77] : Timed out while waiting for systemd to remove kubepods-besteffort-podd0de061d_728e_4305_bb31_5f009ad1fe77.slice"```

I made some tests, I would like to share. Good news is that I confirm what said @tyronewilsonfh, that cgroups v2 are (at least partial) solution. I will try to be brief.
Test case: I run 10 parallel "hello" cronjobs every minute on a single node and observe what happens.
Nevertheless of version (including the latest 4.11.0-0.okd-2022-11-05), something crashes within the node, and after 1.5-2.5hours the pods stop creating (they are in container creating state) and finally kubelet eats all the memory and node is unusable. By default (without any changes) I have mounted cgroups v1 and v2.
What I did to try to overcome this problem (read from this issue):

I deployed daemonset with garbage collector, but cgroups are left behind and node crashes
I changed kubelet runtime argument /var/run -> /run, stil crashes in the same manner (including the newest okd version)
finally I switched to cgroups v2 and the node is more stable; after 20h, the cronjobs are still running, but creationg take a lot of time (up to 1m in comparison to few seconds before).

Few observations of the solution with cgroup v2:

kubelet has plenty of errors still (after this 20h)

sudo journalctl -u kubelet --since -10m | grep -i error | wc -l
100833
sudo journalctl -u kubelet --since -10m | grep -i error | tail -1
Nov 10 10:59:30 dcw1-xyz-infra-0-jnkk8.novalocal hyperkube[366101]: W1110 10:59:30.696142  366101 container.go:589] Failed to update stats for container "/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001b9cfe_5a2e_4d0e_9405_a208719380b4.slice/crio-c6e03c8c2faa6e96b873919e55c2089474a703d1f081b9ce457664747616edfd.scope": unable to determine device info for dir: /var/lib/containers/storage/overlay/b8f6d7526de2280ee997c5b826ce6e08888d5e72176712b4caffb8f00298cc53/diff: stat failed on /var/lib/containers/storage/overlay/b8f6d7526de2280ee997c5b826ce6e08888d5e72176712b4caffb8f00298cc53/diff with error: no such file or directory, continuing to push stats
$ find /sys/fs/cgroup/ -name kubepods-burstable-pod001b9cfe_5a2e_4d0e_9405_a208719380b4.slice
/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001b9cfe_5a2e_4d0e_9405_a208719380b4.slice

cgroups are left behind ind "pids" directory:

$ ls -la /sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/ | wc -l
11971

the memroy utilised is constatly rising, but much less steep and doesn't crash the node

If I have time (after the weekend), I will try to improve the garbage collector to remove left over cgroups
PS. in comparison cgroup files left over, when using cgroups v1:
journalctl -u kubelet --since -10m | grep error | tail -1
Nov 10 10:44:58 dcw1-xyz-worker-1-zxy.novalocal hyperkube[2139]: W1110 10:44:58.392021    2139 container.go:589] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice/crio-483470c57fd83305e8850d78f250d21cab761a2b3e3b88fa243d3b8a06341d8f.scope": unable to determine device info for dir: /var/lib/containers/storage/overlay/6cada6086eec3951850fbf978ad8b72f848d4158450c58344723a4d95d65a779/diff: stat failed on /var/lib/containers/storage/overlay/6cada6086eec3951850fbf978ad8b72f848d4158450c58344723a4d95d65a779/diff with error: no such file or directory, continuing to push stats
$ find /sys/fs/cgroup/ -name kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/blkio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/misc/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/systemd/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice
/sys/fs/cgroup/unified/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae74a3e_be8f_4340_ac5e_16823909ab56.slice

Thanks for the detailed report and - basically - the confirmation that we do not really have any reliable workaround or even solution. So we still wait for a solution of the problem at its root (in the Kubernetes code?).
Regarding the cgroup garbage collector script/daemonset - I briefly tried it on an affected cluster here but the daemonset pods do not even start, due to a rights issue. But did not have time to dig further. Do the pods start for you or perhaps this is also for you the cause why it does not help (and you did not notice yet)?

Thanks for the detailed report and - basically - the confirmation that we do not really have any reliable workaround or even solution. So we still wait for a solution of the problem at its root (in the Kubernetes code?).
Regarding the cgroup garbage collector script/daemonset - I briefly tried it on an affected cluster here but the daemonset pods do not even start, due to a rights issue. But did not have time to dig further. Do the pods start for you or perhaps this is also for you the cause why it does not help (and you did not notice yet)?

They start. I don't remember what I did to make it so. The pods needs to run as priviledged. Maybe something is blocking it? Probably SA, or namespace restrictions?
BTW. I tried quickly to run GC, by my script, but I cannot delete cgroups. My experience is limited on this, so probably I have to read more:
journalctl -u kubelet --since -10m  | grep "Failed to update stats for container" | grep -oE "kubepods-burstable-pod[^/]{10,}slice" | sed "s/.slice//" | xargs -i rm -rf "/sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/{}.slice"

Thanks for the detailed report and - basically - the confirmation that we do not really have any reliable workaround or even solution. So we still wait for a solution of the problem at its root (in the Kubernetes code?). Regarding the cgroup garbage collector script/daemonset - I briefly tried it on an affected cluster here but the daemonset pods do not even start, due to a rights issue. But did not have time to dig further. Do the pods start for you or perhaps this is also for you the cause why it does not help (and you did not notice yet)?

@kai-uwe-rommel did you run the entire manifest as described under https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f? There are required rolebindings and SAs there, under https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f#file-permissions-yaml.
@marqsbla I'd suggest reviewing the script in the above gist... I was able to run it manually on the nodes as well but after creating the daemonset I haven't bothered re-checking the manual runs. What errors are you getting? It would be interesting to improve the daemonset; I've only tested it on OKD 4.9 and 4.10.

Yes, I did run all four.

Hey, i have been silently following this thread for a bit now as i have also been having issues with nodes going down with the same type of memory signature.

My cluster(okd4.11) does not run a whole lot of cronjobs (except for the logging stack). so nodes running out of cpu/memory do not happen (that) often.
What i did notice on my cluste ris a lot of "kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/${POD}/volumes". logs. From what i could gather from the issue referred to here it is a fairly harmless log. (https://github.com/kubernetes/kubernetes/issues/112124)
After doing some analysis of my own this weekend i found out that is not the case. I think particular log is hiding the origin of the nodes going down (at least for me).
I think what happens is the following:

A pod is terminated (because of a normal cause, like a job that has run its course to competion).
After the pod is shut down its cleaned up by the housekeeping loop. Among other things, the housekeeping loop cleans up the directories and cgroups related to the pod. Cleanup up the cgroups is done through the dbus (systemd).
For some reason (i haven't found out why yet) it fails to cleanup all cgroups (this was also already reported in this issue). (note, on loglevel 2 this fails silently).
the next housekeeping loop, it tries to clean the directies (again). because the directory is (already) gone it wil log the message: "kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/${POD}/volumes"
it tries to delete the cgroups again, and again it silently fails.

my hypothesis is this eventually leads to systemd getting hammered with requests to stop cgroups which leads to systemd being overloaded and taking more time to respond to requests (which would align with the fact that it takes progressively more time for containers to start and stop). Eventually this leads to systemd not responding within 30 seconds. i believe this to be the point where both the number of cpu cycles and memory rapidly increase up to the point where the oomkiller starts killing off processes.
I have created a simply function to cleanup these particular cgroups and added them alongside the garbage-collector that was listed earlier. I now see a whole lot less "path does not exist errors".
i've forked the garbage-collector and added my function to it:
https://gist.github.com/framelnl/63387ac66893cb8e856ad8a94bfccea0
Also, i also use the oc debug node container in the daemonset as registry.redhat.io requires a login.

@framelnl, thanks for the detailed analysis. For the two clusters where I have the problem I do also see high CPU usage not only for kubelet but also for systemd on affected nodes, which aligns with your observations. What kills the nodes is then the memory "explosion" of the kubelet.
See my comment of 2 days ago, I did then also install the garbage-collector from @aneagoe but since then I did already still have one node with such a kubelet memory runaway, so it did not really help yet. I will try your fork to see if your change makes a difference.

@framelnl, regarding the image used for the daemonset ... to not depend on any external image at all, I usually use one that is always already present in the cluster's internal registry: "image-registry.openshift-image-registry.svc:5000/openshift/cli". As the name says, it contains the oc CLI, otherwise seems to be the standard RHEL 8 UBI.

With cgroup v2 (systemd.unified_cgroup_hierarchy=1) no more kubelet oom_kills, cluster (okd 4.12) is stable for week now.

For me the stock option cgroups v2 also seems to be "stable", I am running 3 hello cron jobs for 1,5 weeks now in the latest OKD 4.11. I also have to run the cleanup daemonset to achieve this. But I do not find "Path does not exist" in my logs. So the addition from @framelnl is not needed in my setup, weirdly.
I quoted stable because the memory usage still seems to increase on the hosts, in the system slice. Can anyone confirm the following: 'systemd-journald' seems to have a memory leak. I restarted the service which decreased memory usage of the corresponding cgroup from +2G to <100M, after which it builds up again. The system slice still reports using the memory but host memory usage builld up seems to slow down or stop until journald is back to its previous level.
What's puzzling me the most is the differences in behaviour people are reporting. How can this ever be fixed? There are definitely multiple bugs at play in crio(?), kubelet(?), systemd(?). I believe it is important to mention as much details as possible: OKD version, if you use cgroups 2 (stock) or 1 (maybe you changed it for legacy Java versions), ...

For me the stock option cgroups v2 also seems to be "stable", I am running 3 hello cron jobs for 1,5 weeks now in the latest OKD 4.11. I also have to run the cleanup daemonset to achieve this. But I do not find "Path does not exist" in my logs. So the addition from @framelnl is not needed in my setup, weirdly.
I quoted stable because the memory usage still seems to increase on the hosts, in the system slice. Can anyone confirm the following: 'systemd-journald' seems to have a memory leak. I restarted the service which decreased memory usage of the corresponding cgroup from +2G to <100M, after which it builds up again. The system slice still reports using the memory but host memory usage builld up seems to slow down or stop until journald is back to its previous level.
What's puzzling me the most is the differences in behaviour people are reporting. How can this ever be fixed? There are definitely multiple bugs at play in crio(?), kubelet(?), systemd(?). I believe it is important to mention as much details as possible: OKD version, if you use cgroups 2 (stock) or 1 (maybe you changed it for legacy Java versions), ...

@msteenhu thanks for info. Did you face any problem when switching to cgroup v2, for instance with masters? Logging stack? I restrained myself from switching to cgroup v2 on the whole cluster before confirming that everything works :).
Did you make any modifications to garbage collector, or you use this one? I didn't have time to debug if it could be improved. I'm wondering if it works smoothly with cgroup v2...

@msteenhu thanks for info. Did you face any problem when switching to cgroup v2, for instance with masters? Logging stack? I restrained myself from switching to cgroup v2 on the whole cluster before confirming that everything works :).
Did you make any modifications to garbage collector, or you use this one? I didn't have time to debug if it could be improved. I'm wondering if it works smoothly with cgroup v2...

Well, AFAIK cgroups 2 is the norm with OKD since 4.9:
[root@mec-okdtest-worker-01 ~]# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)

So you installed a MachineConfig to switch it back to cgroups v1 then?
I use the mentioned garbage collector already since 4.9. I just did a small change to kill the log spamming (in our production OKD 4.9) from the DaemonSet. @aneagoe helped me out after a question in #openshift-users on the kubernetes Slack by putting his solution on Github. All that time my clusters were running cgroups v2 so my guess is that the garbage collector works best with cgroups v2(?).

@msteenhu to clarify, i run my cluster with cgroups v1 as i upgraded it from 4.8.

@msteenhu I think I originally installed 4.7 and updated since then. I also have cgroups v1 in OKD 4.11 as a default.
Ok, good to know! Thanks

Will the default be different if a cluster was upgraded from previous OKD versions?
On my test cluster:
[root@master-03 ~]# mount|fgrep cgroup                                                                           tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)  cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)

Will the default be different if a cluster was upgraded from previous OKD versions? On my test cluster:
So upgraded and fresh clusters are really different? What is your advice how to "convert" upgraded clusters?

Depending on your history there might be MachineConfig(s) in place influencing the Cgroups mount.
You could verify for yourself with a command like this:
oc get mc -o yaml | grep kernelArguments -A10 | grep cgroup

Thanks. No such MC found, so the cluster is on defaults. And this is apparently kind of "mixed".

@kai-uwe-rommel It looks for me that you have cgroup v2.
In my cluster it looks like this:
mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,misc)

It is really a bit unclear ... my test cluster seems to have both.
But if you says it's using cgroup v2 - it still has the kubelet problem.
While msteenhu reports that with cgroup v2 his cluster is running fine.

It is really a bit unclear ... my test cluster seems to have both. But if you says it's using cgroup v2 - it still has the kubelet problem. While msteenhu reports that with cgroup v2 his cluster is running fine.

The mount output looks v2 to me too, which is "stock" for current kernels AFAIK, and that is also consistent with you not finding MCs setting 'cgroup' kernel arguments.
It is running fine if the garbage collector is running, cleaning up up cgroups and crio sandboxes (not sure what the latter means, just reporting what the GC logs). Kubelet is behaving well, no excess cpu or memory usage. Just some log messages that are related to cgroups/sandboxes that the GC could not clean up, I assume.

Ah, now I understand. So the garbage-collector daemonset is still required. I thought you meant the cluster would not even have the problem at all with cgroups v2 for you and thus not need garbage-collector.
The original garbage-collector from @aneagoe did not help for my test cluster but the extended version from @framelnl seems to help. I'm running with it for three days now without any problem yet. I should wait for a couple more days if this is really the case but it looks better so far.

The mount output looks v2 to me too

I do not this that's right, e.g. see: https://kubernetes.io/docs/concepts/architecture/cgroups/#check-cgroup-version it says tmpfs which mean v1.

The mount output looks v2 to me too

I do not this that's right, e.g. see: https://kubernetes.io/docs/concepts/architecture/cgroups/#check-cgroup-version it says tmpfs which mean v1.

I stand corrected. Bit unclear when v1 or v2 is used then for me but that seems to explain the difference in behaviour of the GC I guess: v1 needs the addition, v2 not.

My test cluster now runs for a week with the garbage-collector extended version from @framelnl and I had no more problems with any node. So at least I have a real workaround for now.
I assume the new 2022-11-19 release of OKD 4.11 does not have any specific fixes for this problem yet? Still the same K8s version.

Will the default be different if a cluster was upgraded from previous OKD versions? On my test cluster:
[root@master-03 ~]# mount|fgrep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)

So upgraded and fresh clusters are really different? What is your advice how to "convert" upgraded clusters?

The "official" documentation of switching over to cgroups v2: https://docs.openshift.com/container-platform/4.10/post_installation_configuration/machine-configuration-tasks.html#nodes-nodes-cgroups-2_post-install-machine-configuration-tasks

My test cluster now runs for a week with the garbage-collector extended version from @framelnl and I had no more problems with any node. So at least I have a real workaround for now.
I assume the new 2022-11-19 release of OKD 4.11 does not have any specific fixes for this problem yet? Still the same K8s version.

Thanks for testing it. Did you also change to cgroup2 or just the extended garbage collector helped to fix it ?

Something to note with the cgroup change is I've had nodes end up with kernel command lines like
systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all systemd.unified_cgroup_hierarchy=0
and nothing starts at all..
I had to edit MachineConfig/99-okd-master-disable-mitigations (and worker) to change the
ADD systemd.unified_cgroup_hierarchy=0
line from /etc/pivot/kernel-args to
DELETE systemd.unified_cgroup_hierarchy=0
otherwise crun throws errors and containers do not start

Is the issue now resolved within the latest 4.11.0-0.okd-2022-12-02-145640 release?

@danielchristianschroeter nodes are still crashing on     4.11.0-0.okd-2022-12-02-145640

The workaround suggested several messages above, with the garbage-collector and switching to cgroups v2, though not ideal, does prevent memory usage from skyrocketing and nodes from dying. We confirmed that on two OKD clusters, now running 4.11.0-0.okd-2022-12-02-145640.

Short thread summary.
What we know so far:

bug is happening on any new container starts, cronjob is an easy way to trigger it, but other ways to create new pods would also show the same behaviour
kubelet seems to use a lot of CPU and RAM
switching to cgroupsv2 seems to stop RAM bleeding, but doesn't help with CPU load

Note that fresh OKD 4.9 installs are using cgroupsv2 by default, upgraded cluster need to be switched manually

Probable cause:

latest kernel is known to leak blkios, this is mitigated by the workaround (see below)

Workaround: https://github.com/okd-project/okd/issues/1310#issuecomment-1312848841 - thanks to @framelnl there's a DaemonSet which can clean up the extra cgroups
Upstream issue refs:

https://github.com/kubernetes/kubernetes/issues/106957#issuecomment-1147441007
https://github.com/kubernetes/kubernetes/pull/113050 (1.25 cherrypick of crio fix)

Thanks for the summary. But what is the plan how to finally solve the problem?
Just wait for upstream fixes?
OCP does not seem to have this issue...

But what is the plan how to finally solve the problem? Just wait for upstream fixes?

Well, yes, seems its crio+kernel problems, which we take from upstream we should first land fixes there.

OCP does not seem to have this issue...

That probably means that kernel leaking blkios is the main contributor to the issue

Thanks for the conclusion but it does not match entirely with mine. Where am I wrong?

What we know so far:

bug is happening on any new container starts, cronjob is an easy way to trigger it, but other ways to create new pods would also show the same behaviour
kubelet seems to use a lot of CPU and RAM
switching to cgroupsv2 seems to stop RAM bleeding, but doesn't help with CPU load

Stops RAM bleeding? The cleanup DaemonSet (DS) is deleting cgroups at each run. In my eyes this is a memory leak? Not sure what you mean.
CPU load is in my opinion normal when using the DS.

Probable cause:

latest kernel is known to leak blkios, this is mitigated by the workaround (see below)

This I could not detect on our systems. I will look into it again.

Workaround: https://github.com/okd-project/okd/issues/1310#issuecomment-1312848841 - thanks to @framelnl there's a DaemonSet which can clean up the extra cgroups

@aneagoe is the original creator of the DS ( May 20th: https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f/revisions).
We run the exact same original DS workaround in clusters ranging versions from 4.9 till 4.11.

Is there anything new about this issue?

Is there anything new about this issue?

swtiching to cgroup v2 and running the GC (https://github.com/okd-project/okd/issues/1310#issuecomment-1312848841) in the cluster seems to overcome the issue. I didn't have the problem with master/workers since 2 weeks now.

Is there anything new about this issue?

swtiching to cgroup v2 and running the GC (#1310 (comment)) in the cluster seems to overcome the issue. I didn't have the problem with master/workers since 2 weeks now.

I will try. What is the proper way to switch to cgroup2?

Is there anything new about this issue?

swtiching to cgroup v2 and running the GC (#1310 (comment)) in the cluster seems to overcome the issue. I didn't have the problem with master/workers since 2 weeks now.

I will try. What is the proper way to switch to cgroup2?

Take a look at the documentation. This option disables cgroupv1 and enables cgroupv2.

Is there anything new about this issue?

swtiching to cgroup v2 and running the GC (#1310 (comment)) in the cluster seems to overcome the issue. I didn't have the problem with master/workers since 2 weeks now.

Found one issue or bug with this solution. We have oVirt infra in background of OKD 4.11. We decided to extend our MachineSet and deploy new worker VM, and it just stuck in forever "Provisioned" status. After digging in google found other bug with cgroupv2 and podman https://bugzilla.redhat.com/show_bug.cgi?id=1897579.
machine-config-daemon-pull.service crashing and not finishing deploy and provisioning of new worker VM.
So, after playing around with cgroup, for us solution was:

Connect through SSH to new VM in provisioned state and copy /etc/ignition-machine-config-encapsulated.json.bak to /etc/ignition-machine-config-encapsulated.json then edit file (remove KernelArgument bla bla cgroupv2)
/usr/bin/rpm-ostree kargs --delete systemd.unified_cgroup_hierarchy=0
systemctl reboot
Check systemctl status machine-config-daemon-pull.service it should be active w/o errors.
After that VM will be provisioned fully and appear in Nodes as ready

ing to cgroup v2 and running t

Do we have to switch to cgroup2 on master nodes only or worker nodes too?

Do we have to switch to cgroup2 on master nodes only or worker nodes too?

I'm running masters on cgroup v2 and some nodes on cgroup v1. When running the corrected GC, everything seems stable. I have cronjobs on both masters and cgroup v1/v2 nodes.

Do we have to switch to cgroup2 on master nodes only or worker nodes too?

I'm running masters on cgroup v2 and some nodes on cgroup v1. When running the corrected GC, everything seems stable. I have cronjobs on both masters and cgroup v1/v2 nodes.

There are some errors in all DS pods

I believe the problem is not (completely) solved using cgroups V2 and the garbage collection DaemonSet.
The number of cgroups keep increasing on my 4.11 and 4.9 hosts hosts. Whe are 1 update behind on our 4.11 cluster.
$ cat /proc/cgroups
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	19500	1
cpu	0	19500	1
cpuacct	0	19500	1
blkio	0	19500	1
memory	0	19500	1
devices	0	19500	1
freezer	0	19500	1
net_cls	0	19500	1
perf_event	0	19500	1
net_prio	0	19500	1
hugetlb	0	19500	1
pids	0	19500	1
misc	0	19500	1

Anyone else seeing those counts (3th column) slowly but surely building up?
The behaviour we notice is that a host might lock up suddenly or hard reboot after running fine for several months. Host lock ups are a real problem to us since this locks 'ReadWriteOnce' storage until manual action is taken (or I should invest time in a fencing solution).
This bug(s) are going on for many months now and I really would like a free choice of OS/kernel to get a way out of this stability problems.

No. I see MUCH lower counts:
[root@master-01 ~]# cat /proc/cgroups #subsys_name    hierarchy       num_cgroups     enabled cpuset  2       260     1 cpu     4       376     1 cpuacct 4       376     1 blkio   3       376     1 memory  9       470     1 devices 10      370     1 freezer 6       260     1 net_cls 5       260     1 perf_event      8       260     1 net_prio        5       260     1 hugetlb 12      260     1 pids    11      377     1 misc    7       168     1
Do you use the newer/updated garbage collector or the original one?
I noticed that only the updated from @framelnl solves the problem on our clusters:
https://gist.github.com/framelnl/63387ac66893cb8e856ad8a94bfccea0

Also please note that the GC workaround has a serious drawback when the api is not available for whatever reason (network problems, ...): it starts destroying all pods in a very sneaky way, networking breaks subtly (DNS requests, ...) because the GC interprets API unavailability as unexisting pods, it tries to delete pods against CRIO. This fails eventually but still breaks networking for many pods. Even worse is pods do not auto-recover when api is available again. You need to kill all problem pods manually so CRIO can build up networking again.
This is the patch we applied to prevent this, if I find time I will share a fork on github:
diff --git a/infra_kustomize/base/garbagecollection/cm-garbagecollector.yaml b/infra_kustomize/base/garbagecollection/cm-garbagecollector.yaml
index e97289e..4f09723 100644
--- a/infra_kustomize/base/garbagecollection/cm-garbagecollector.yaml
+++ b/infra_kustomize/base/garbagecollection/cm-garbagecollector.yaml
@@ -74,6 +74,11 @@ data:
         KUBECONFIG=/var/lib/kubelet/kubeconfig kubectl get pod ${POD_NAME} -n ${POD_NAMESPACE} &>/dev/null
         if [[ $? -ne 0 ]]; then
           # additional check for safety, making sure that if there's a problem with apiserver we don't blindly remove pods with running processes
+          KUBECONFIG=/var/lib/kubelet/kubeconfig kubectl get --raw='/readyz' &>/dev/null
+          if [[ $? -ne 0 ]]; then
+            logger "Kubernetes API unavailable. Could be false positive so skipping deletion of POD ${POD_NAME}"
+            continue
+          fi
           if [[ ${#POD_PIDS[@]} -eq 0 ]]; then
             logger "Found POD ${POD_NAME} unknown to k8s control plane and without any PIDs, will delete it..."

-      gc_pods
-      gc_cgroups
-      cleanup_misc_cgroups
+      KUBECONFIG=/var/lib/kubelet/kubeconfig kubectl get --raw='/readyz' &>/dev/null
+      if [[ $? -ne 0 ]]; then
+        logger "Kubernetes API unavailable. Cancelling run."
+      else
+        gc_pods
+        gc_cgroups
+        cleanup_misc_cgroups
+      fi
       logger "Sleeping for ${SLEEP_INTERVAL} seconds..."
       sleep ${SLEEP_INTERVAL}
     done

No. I see MUCH lower counts: [root@master-01 ~]# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 2 260 1 cpu 4 376 1 cpuacct 4 376 1 blkio 3 376 1 memory 9 470 1 devices 10 370 1 freezer 6 260 1 net_cls 5 260 1 perf_event 8 260 1 net_prio 5 260 1 hugetlb 12 260 1 pids 11 377 1 misc 7 168 1
Do you use the newer/updated garbage collector or the original one? I noticed that only the updated from @framelnl solves the problem on our clusters: https://gist.github.com/framelnl/63387ac66893cb8e856ad8a94bfccea0

It makes sense your masters are more stable: they will typically not create short lived pods (f.e. cron jobs) all the time. IMy
numbers are from a bare metal server with 80 cpu cores and 256G of RAM. I believe I am using the latest GC version (although I believe it only contained fixes relevant for cgroups v1) but I will double check.

I've updated the original gist with the information above (though cleanup_misc_cgroups  function is still missing there). I've also checked these metrics on one of my nodes:
[root@worker-01 ~]# cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt1)/ostree/fedora-coreos-4def74ccdefacdfcbe6a46101f0ddaa48f9f8e80216a0c7c9aa9fda6d5df075b/vmlinuz-5.18.5-100.fc35.x86_64 console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal ostree=/ostree/boot.1/fedora-coreos/4def74ccdefacdfcbe6a46101f0ddaa48f9f8e80216a0c7c9aa9fda6d5df075b/0 rootflags=prjquota mitigations=off
[root@worker-01 ~]# uptime
 12:23:10 up 158 days, 19:01,  1 user,  load average: 1.27, 0.57, 0.41

[root@worker-01 ~]# mount|fgrep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)

[root@worker-01 ~]# cat /proc/cgroups | column -t
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        0          92679        1
cpu           0          92679        1
cpuacct       0          92679        1
blkio         0          92679        1
memory        0          92679        1
devices       0          92679        1
freezer       0          92679        1
net_cls       0          92679        1
perf_event    0          92679        1
net_prio      0          92679        1
hugetlb       0          92679        1
pids          0          92679        1
misc          0          92679        1

the num_cgroups number is very high indeed. But I'm not entirely sure that number denotes value since inception or current value. The latter would be strange, as I don't even have that many processes in total or directories under cgroup:
[root@worker-01 ~]# find /sys/fs/cgroup -type d | wc -l
172
[root@worker-01 ~]# tail -n 1 /proc/*/cgroup | grep ^0:: | wc -l
747

That being said, I have not observed any issue on this node, as well as other nodes in my cluster, for quite some time (see uptime on this one). Running version 4.10.0-0.okd-2022-07-09-073606 together with the GC DS. All nodes are using cgroupsv2 while masters are still on v1 (just didn't get a chance to migrate them away).

I believe something is left behind in the kernel and that is really the current number of cgroups the kernel is keeping track of. I will try to further investigate. Could be wrong of course.
I am pretty sure our downtime is related to this problem since it is always the same 'type' of node that crashes. Kubernetes always tends to put short lived pods on the same node, namely the least active one (or whatever the scheduler does by default) and it is always that one that crashes after a few months.
But indeed a quite impressive uptime already. I'd say that is pretty close to our maximum uptime for the most active node.

Ok, mystery solved (I guess): https://docs.kernel.org/admin-guide/cgroup-v2.html#deprecated-v1-core-features

This confuses me even more because cgroup.controllers does not tell me anything?
And why do those number keep increasing (and also go down sometimes) on the node that I am quite sure will crash in a few week/months?
Anyways I will not pollute this issue any further and try to find the time to dive deeper first.

Ok, mystery solved (I guess): https://docs.kernel.org/admin-guide/cgroup-v2.html#deprecated-v1-core-features

This confuses me even more because cgroup.controllers does not tell me anything? And why do those number keep increasing (and also go down sometimes) on the node that I am quite sure will crash in a few week/months?
Anyways I will not pollute this issue any further and try to find the time to dive deeper first.

Can you please provide steps how to avoid this issue (with simple explanation 🙂).

Can you please provide steps how to avoid this issue (with simple explanation 🙂). Which DS workaround is latest and best version to use?

Seems like consensus is use cgroupv2 or make sure your 'mount | grep cgroup' output looks like this:
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)
I'd go for the original: https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f
@framelnl added an extra cleanup function 'cleanup_misc_cgroups' which seems to help for some people:
https://gist.github.com/framelnl/63387ac66893cb8e856ad8a94bfccea0/revisions
The changes added on Nov 13, 2022 are the relevant ones.

Can you please provide steps how to avoid this issue (with simple explanation 🙂). Which DS workaround is latest and best version to use?

Seems like consensus is use cgroupv2 or make sure your 'mount | grep cgroup' output looks like this: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)
I'd go for the original: https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f @framelnl added an extra cleanup function 'cleanup_misc_cgroups' which seems to help for some people: https://gist.github.com/framelnl/63387ac66893cb8e856ad8a94bfccea0/revisions The changes added on Nov 13, 2022 are the relevant ones.

Is switching to cgroup2 even necessary?

Is switching to cgroup2 even necessary?
Depending on your history you need to switch, yes. Your 'mount' output will tell. If you have a relatively new cluster you should be good to go, since OKD seems to use cgroupsv2 now.

To me it looks like our clusters run in "mixed mode" cgroups v1+v2. But the garbage collector still works fine and keeps the cluster working.

To me it looks like our clusters run in "mixed mode" cgroups v1+v2. But the garbage collector still works fine and keeps the cluster working.

I get some errors in DS pods. Is that ok? Master nodes use cgroup2, worker nodes use cgroup1.

I get some errors in DS pods. Is that ok? Master nodes use cgroup2, worker nodes use cgroup1.

There seem to be too many if you see these at each run (I only find 1 such line in our logging the last week). If so, better to look like for help elsewhere. I'd recommend the #openshift-users channel in the Kubernetes slack.

I do also see frequent messages like these ("Error fetching pod PIDs ..."). It does not seem to hurt, though. The "garbage collection" works and the cluster stays healthy.

To me it looks like our clusters run in "mixed mode" cgroups v1+v2. But the garbage collector still works fine and keeps the cluster working.

To me it looks like our clusters run in "mixed mode" cgroups v1+v2. But the garbage collector still works fine and keeps the cluster working.

I last week had a 4.11 where CRIO was using 20 CPU cores and PODs ovviously hardly could get sheduled anymore. It was running in mixed mode (probably due to fiddling in the console in the past).
Not sure if it was the mixed mode but I put in cgroupsv2 only using this command (and reboot):
rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy=0

But the 99-okd-worker-disable-mitigations MC (and the one for masters as well) does contain "ADD systemd.unified_cgroup_hierarchy=0". So wouldn't it re-add this parameter? So we need to modify these MCs?

But the 99-okd-worker-disable-mitigations MC (and the one for masters as well) does contain "ADD systemd.unified_cgroup_hierarchy=0". So wouldn't it re-add this parameter? So we need to modify these MCs?

I  had this same issue and just did this as a work around, seems to have persisted across a couple of updates now. https://github.com/okd-project/okd/issues/1310#issuecomment-1336006192

To me it looks like our clusters run in "mixed mode" cgroups v1+v2. But the garbage collector still works fine and keeps the cluster working.

I get some errors in DS pods. Is that ok? Master nodes use cgroup2, worker nodes use cgroup1.

Seems to be a Problem with SELinux. When the cleanup runs I also saw this audit messages:
AVC avc:  denied  { dac_override } for  pid=1656847 comm="ip" capability=1  scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:system_r:ifconfig_t:s0 tclass=capability permissive=0
Giving the SELinux Label to the pod solves the issue:
securityContext:
        seLinuxOptions:
          type: ifconfig_exec_t

Also, something like
updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1

should be applied to the DS. :-)

I've been following this for a while now, as I've also got a cluster affected by this (running unstable since the upgrade to 4.11; currently running 4.11.0-0.okd-2022-12-02-145640). Got several CronJobs running (with one of rather high frequency), which cause worker nodes to not even survive 24 hours without crashing.
I've now also deployed the DaemonSet workaround (using the latest version from @aneagoe https://github.com/okd-project/okd/issues/1310#issuecomment-1372157145), but also had to add the extra cleanup_misc_cgroups from @framelnl . Without the extra cleanup it was still crashing within a short period of time, the combined cleanup tasks now improved the situation a bit. Note that even with the DaemonSet deployed, the issue seems not to be fully resolved, it only delayed the crashes and the nodes are now at least lasting for a week and not crashing daily. But there's still a memory leak visible in the node metrics:

A second cluster still running on 4.10 which has almost the same workloads (most importantly, runs the same high frequency CronJobs) has none of these issues, the memory consumption curve is "perfectly" flat even over weeks of operation without any restarts or workarounds. Have been holding back with the upgrade to 4.11 on this second production system due to this major issue so far.
These clusters are running without cgroups v2 (upgraded from an older installation where this was still the default, i.e. kernel parameter systemd.unified_cgroup_hierarchy=0 is set in 99-okd-worker-disable-mitigations etc.). Disabling v1 is not an option, as there are some workloads which still rely on Java 8, for which v2 support is only scheduled to be available in April (see https://bugs.openjdk.org/browse/JDK-8297880 -> Timeline for 8u372 in https://wiki.openjdk.org/display/jdk8u/Main).

After upgrading to cgroupV2 (all our 6 nodes) without using garbage collector fix our systems now running for two weeks without any problems. Usually our nodes failed within one or two days after restart :-)
So this completely fixed our issues thanks :-)

After upgrading to cgroupV2 (all our 6 nodes) without using garbage collector fix our systems now running for two weeks without any problems. Usually our nodes failed within one or two days after restart :-) So this completely fixed our issues thanks :-)
Which version are you using precisely?

We are still using version: 4.11.0-0.okd-2022-12-02-145640

Run few tests w/ clean AWS cluster + cronjob spawning a container every minute on latest 4.11

and freshly released 4.12:

Seems it was caused by a combination of runc and kubelet job controller leaks, both fixed in latest 4.11/4.12. Latest kernel is known to leak memory when cgroupv1 is used, which needs to be handled upstream.
Closing, as original CPU/memory leak has been fixed. Feel free to open new issues if you're seeing memory leaks (don't forget to provide the metric results AND must-gather.

@vrutkovs Is there an open issue for the Kubernetes kernel problem so we can subscribe and wait until it's fixed?

still seeing issues on the before mentioned cluster, currently running 4.11.0-0.okd-2023-01-14-152430, although with cgroups v1 enabled. Not sure if it's really the linked blkio leak, as I don't see an increasing count as described there, but I didn't get to test the workaround from the blkio issue (still running the garbage collector daemon set from this issue).
Anyway, even if it's "only" the kernel issue remaining and not affecting cgroups v2, I don't see how this can be considered fixed in this case. Even though new clusters are installed with cgroups v2 by default, old upgraded instances may be still using v1 as detailed in earlier comments. So even if this cannot be fully fixed within OKD alone, I'd expect to see:

an automatic switch from cgroups v1 to v2 during upgrades, or a note in the documentation that this must be done manually during one of the upgrades (unless I missed that so far in the docs?)
and a big warning in the docs that v1 is currently broken and that it cannot be used in practice (i.e. in https://docs.okd.io/4.12/installing/install_config/enabling-cgroup-v2.html)

Wouldn't be really satisfying as it still means that OKD is almost unusable with certain workloads which cannot upgrade to v2, but at least it would be documented and provide hints on how to potentially workaround it.

Before we put warnings on docs we need more info - must-gather/metrics and reproducible issue.

it still means that OKD is almost unusable with certain workloads

I don't mind fixing these cases, but a) original issue is certainly not reproducible anymore and b) "certain" workloads needs a different issue with a distinct instruction. We've already let this issue spin from "high kubelet CPU usage" to "something is eating too much memory", lets stop it here and file new, more specific issues

@ssams same here. Every day pods stuck in container creating or terminating status. Restarting worker nodes helps, but it is annoying.

@ssams , @dkoci then try using the (modified) garbage collector as mentioned above.
Our clusters with cgroupsv1 are stable since using this.

then try using the (modified) garbage collector as mentioned above.

I'm using it already, as mentioned in https://github.com/okd-project/okd/issues/1310#issuecomment-1383770942 it certainly helped a lot although it doesn't seem to be perfect (but good enough to delay crashes enough so it becomes usable again). Nevertheless I'd still consider it to be more a temporary workaround, not a permanent fix, as it certainly hacks into some internals and may not be free of side effects as one can see in the logs (and I observed e.g. some slow startup times due to additional cleanup running after restarting a node manually from time to time, so I assume more would need to be cleanup up but didn't have the time for further investigation so far).

I don't mind fixing these cases, but
a) original issue is certainly not reproducible anymore

hm, as far as I can see it should be still reproducible, especially as the initial issue mentioned "after upgrading to 4.11" -> that's why I pointed out the different cgroups again which result from upgrades vs. new installations, so a fresh install may not be a sufficient reproduction. Unfortunately don't have the resources to spin up a new cluster and confirm it myself at the moment though. But of course can also create a new/separate issue regarding the alignment of old upgrades with new defaults if that's preferred.

"certain" workloads needs a different issue with a distinct instruction.

I was referring to my earlier comment in https://github.com/okd-project/okd/issues/1310#issuecomment-1383770942, sorry if that wasn't really clear. Was meant as example that if the fix for this requires to upgrade to cgroups v2, then that may not be possible to apply under some circumstances, one instance being the presence of Java 8 workloads which currently cannot detect v2 properly (https://bugs.openjdk.org/browse/JDK-8297880 -> release scheduled for April 2023)

So I'm currently trying to upgrade workloads in our cluster to prepare for a switch to cgroups v2 to finally move on from this issue, as according to the present information that should fix it. If not I'll create a new follow-up issue as requested.

Currently I have the following kernel args.
[root@worker1-cl1-dc99 /]# cat /boot/loader/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 36 411.36.202301141513-0 (ostree:0)
version 1
options mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/98fe0187ba7d8d444bc182df103375b8acb8ca1e5cef5e365cfba246755d5317/0 ignition.platform.id=vmware root=UUID=60eb22b6-114a-43e4-bb54-678710a626a2 rw rootflags=prjquota systemd.unified_cgroup_hierarchy=0 mitigations=off
linux /ostree/fedora-coreos-98fe0187ba7d8d444bc182df103375b8acb8ca1e5cef5e365cfba246755d5317/vmlinuz-6.0.18-200.fc36.x86_64
initrd /ostree/fedora-coreos-98fe0187ba7d8d444bc182df103375b8acb8ca1e5cef5e365cfba246755d5317/initramfs-6.0.18-200.fc36.x86_64.img
[root@worker1-cl1-dc99 /]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-98fe0187ba7d8d444bc182df103375b8acb8ca1e5cef5e365cfba246755d5317/vmlinuz-6.0.18-200.fc36.x86_64 mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/fedora-coreos/98fe0187ba7d8d444bc182df103375b8acb8ca1e5cef5e365cfba246755d5317/0 ignition.platform.id=vmware root=UUID=60eb22b6-114a-43e4-bb54-678710a626a2 rw rootflags=prjquota systemd.unified_cgroup_hierarchy=0 mitigations=off

As you can see systemd.unified_cgroup_hierarchy=0 is enabled, but why? Maybe somebody knows. I have no additional MachineConfigs. The rendered-worker configuration has only
  kernelArguments:
    - mitigations=off

I think all custom applications support cgroupv2 . But at the moment it looks like cgroupv1 is preferred.
To make it even more complicated, cgroupv2 seems to be enabled.
[root@worker1-cl1-dc99 /]# mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,misc)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)

But crio seems to be use cgroupv1
Jan 23 14:11:15 worker1-cl1-dc99 crio[1968]: time="2023-01-23 14:11:15.329551909Z" level=info msg="Node configuration value for cgroup v2 is false"

Systemd complains about cgroupv1
Jan 23 14:10:47 localhost systemd[1]: System is tainted: cgroupsv1

I would be grateful if someone could support me in finding a solution.

As you can see systemd.unified_cgroup_hierarchy=0 is enabled, but why? Maybe somebody knows. I have no additional MachineConfigs.

Assuming you got an upgraded cluster, have you checked 99-okd-worker-disable-mitigations (same for -master-, https://github.com/okd-project/okd/issues/1310#issuecomment-1336006192)? The data in contents.source is base64 encoded (at least in my instance), so it may not be obvious and findable with a simple textual search.

The data in contents.source is base64 encoded (at least in my instance), so it may not be obvious and findable with a simple textual search.

My rendered MachineConfigs aren't base64 encoded, unfortunately I have not found a reference. I didn't find anything on the system with grep either, except in the boot configuration.
Example snippet of the rendered-config
        - contents:
            source: >-
              data:,%5BManager%5D%0ADefaultEnvironment%3DGODEBUG%3Dx509ignoreCN%3D0%0A
          mode: 420
          overwrite: true
          path: /etc/systemd/system.conf.d/10-default-env-godebug.conf

My mitigations MachineConfig (some metadata are removed).
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: 'true'
    include.release.openshift.io/self-managed-high-availability: 'true'
    include.release.openshift.io/single-node-developer: 'true'
    release.openshift.io/create-only: 'true'
  name: 99-okd-worker-disable-mitigations
  generation: 1
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
    - mitigations=off

My rendered MachineConfigs aren't base64 encoded, unfortunately I have not found a reference. I didn't find anything on the system with grep either, except in the boot configuration.

not sure if you still have the problem, but I think i just ran into the same when trying to switch to cgroups v2, so maybe as a hint: in case the kernel argument was previously added via the ...-disable-mitigations with content in /etc/pivot/kernel-args (-> ADD systemd.unified_cgroup_hierarchy=0), it seems not sufficient to just remove that line. I think the relevant part which handles this file is https://github.com/openshift/pivot/blob/e31a72d53a9ccfd517b46febbf12e4048eeececc/cmd/root.go#L161-L190, and that only performs the additions / removals as instructed in the file, so if you remove an entry it's just left in the current state but not reverted to any other state automatically. So looks like you either have to add an explicit DELETE systemd.unified_cgroup_hierarchy=0, or remove the argument manually on each node via rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy=0 (https://github.com/okd-project/okd/issues/1310#issuecomment-1374411452, same as the pivot util would run)

Thanks for your reply in the meantime I have indeed manually removed the option with rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy=0 on all nodes.

11 days ago I deployed a test cluster with OKD 4.11-2023-01-14 from scratch. So it must have been deployed with cgroups V2 from the beginning. Yesterday, after 10 days even this cluster ran into our well known kubelet memory leaking problem. So I installed the garbage collector on this cluster now as well. So IMO, it is not just a cgroups V1 vs. V2 problem.

My observations show that memory and CPU problems occur much more quickly with cgroupv1. I hope that at least for cgroupv2 the problem was found in https://github.com/cri-o/cri-o/issues/6349 and will be fixed soon.

Why this issue is closed?

Why this issue is closed?

see:
https://github.com/okd-project/okd/issues/1310#issuecomment-1400465439
This issue has wandered quite a bit. By all accounts the origin of this issue has been handled but obviously some of the offshoots with similar behaviors (and workarounds?) persist so they likely deserve their own issues to be filed and diagnostics gathered.

I have a OKD 4.11-2023-01-14 cluster were memory is filling on nodes and crashing. Many messages with "Timeout waiting for systemd to create kubepods-burstable", looks like the issue here.
a) Is it correct to say that okd 4.12 fixes this, but not 4.11 (latest 4.11-2023-01-14)?
b) As 4.12 is not an option for me yet (no gluster storage support), for 4.11, is the fix to upgrade nodes to  cgroups v2 ?
https://docs.okd.io/4.11/nodes/clusters/nodes-cluster-cgroups-okd.html

I personally am not sure if cgroups v2 really completely fixes the problem (or 4.12).
Instead I rather let the "cgroups garbage collector" run on the clusters which I am responsible for.
It does not cost much if it is run but not needed.
See above for details about it. You will find this link above:
https://gist.github.com/aneagoe/6e18aaff48333ec059d0c1283b06813f
You may need to add some changes to it (see above as well):
*** cgroups-garbage-collector.yaml.orig	Fri Nov 25 13:17:24 2022
--- cgroups-garbage-collector.yaml	Fri Jan 13 20:55:39 2023
***************
*** 126,131 ****
--- 126,136 ----

          if [[ $? -ne 0 ]]; then
            # additional check for safety, making sure that if there's a problem with apiserver we don't blindly remove pods with running processes
+           KUBECONFIG=/var/lib/kubelet/kubeconfig kubectl get --raw='/readyz' &>/dev/null
+           if [[ $? -ne 0 ]]; then
+             logger "Kubernetes API unavailable. Could be false positive so skipping deletion of POD ${POD_NAME}"
+             continue
+           fi
            if [[ ${#POD_PIDS[@]} -eq 0 ]]; then
              logger "Found POD ${POD_NAME} unknown to k8s control plane and without any PIDs, will delete it..."

+       KUBECONFIG=/var/lib/kubelet/kubeconfig kubectl get --raw='/readyz' &>/dev/null
+       if [[ $? -ne 0 ]]; then
+         logger "Kubernetes API unavailable. Cancelling run."
+       else
          gc_pods
          gc_cgroups
          cleanup_misc_cgroups
+       fi
        logger "Sleeping for ${SLEEP_INTERVAL} seconds..."
        sleep ${SLEEP_INTERVAL}
      done
***************
*** 202,210 ****
    selector:
      matchLabels:
        app: k8s-gc
    template:
      metadata:
-       creationTimestamp: null
        labels:
          app: k8s-gc
      spec:
--- 212,223 ----
    selector:
      matchLabels:
        app: k8s-gc
+     updateStrategy:
+       type: RollingUpdate
+       rollingUpdate:
+         maxUnavailable: 1
    template:
      metadata:
        labels:
          app: k8s-gc
      spec:
***************
*** 221,226 ****
--- 234,241 ----
          securityContext:
            privileged: true
            runAsUser: 0
+           seLinuxOptions:
+             type: ifconfig_exec_t
          volumeMounts:
          - mountPath: /host
            name: host

Thanks.
 Your patch 126,136 and 190,203 are already integrated into the gist, but not the last three parts of the diff.
So you have 4.11 with the GC above, and no memory issues?

Our clusters now run with 4.13 or 4.12 but we had no problems any longer when we were running 4.11, either.

I have a OKD 4.11-2023-01-14 cluster were memory is filling on nodes and crashing. Many messages with "Timeout waiting for systemd to create kubepods-burstable", looks like the issue here.
a) Is it correct to say that okd 4.12 fixes this, but not 4.11 (latest 4.11-2023-01-14)?
b) As 4.12 is not an option for me yet (no gluster storage support), for 4.11, is the fix to upgrade nodes to cgroups v2 ? https://docs.okd.io/4.11/nodes/clusters/nodes-cluster-cgroups-okd.html

@Boran  Of topic question where did you read this about dropping of gluster support? Isn't it EOL end of 2024? Also nothing in the release notes of 4.12...

Just for fun)))
bash-5.2$ oc get clusterversion NAME      VERSION                          AVAILABLE   PROGRESSING   SINCE   STATUS version   4.12.0-0.okd-2023-04-16-041331   True        False         8d      Cluster version is 4.12.0-0.okd-2023-04-16-041331

We also ran into this problem in the summer. Then I removed systemd.unified_cgroup_hierarchy=0 on every machine, restarted them and everything got back into a good, working condition. After months of happyness, we had some maintenance work on the cluster where we had to shut down OKD machines in batches and later start them, and here we ran into the exact same situation, the FCOS used cgroup v1 again.
To make things persistent, I thought I'll just create a MachineConfig (based on the docs) and set systemd.unified_cgroup_hierarchy to 1 (instead of 0), because that hopefully overrides the original value and configures the machines to use cgroup v2.
Well, no. After the MCO restarted the nodes, they came up with both value, 0 and 1 as well, where the first took priority, what was 0, so they used v1.
Is there any way to either override the default value or delete that using MachineConfig?

We also ran into this problem in the summer. Then I removed systemd.unified_cgroup_hierarchy=0 on every machine, restarted them and everything got back into a good, working condition. After months of happyness, we had some maintenance work on the cluster where we had to shut down OKD machines in batches and later start them, and here we ran into the exact same situation, the FCOS used cgroup v1 again. To make things persistent, I thought I'll just create a MachineConfig (based on the docs) and set systemd.unified_cgroup_hierarchy to 1 (instead of 0), because that hopefully overrides the original value and configures the machines to use cgroup v2. Well, no. After the MCO restarted the nodes, they came up with both value, 0 and 1 as well, where the first took priority, what was 0, so they used v1. Is there any way to either override the default value or delete that using MachineConfig?

Hello, we encountered the same problem. We've got the machine configs that should enforce cgroupv2 but to no avail.
On reboot, after cluster upgrade or when provisioning new nodes, we end up with the same "hybrid" situation: both cgroup versions are mentioned in the kernel parameters:

sh-5.2# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-792b78931dec8d7b9ceb91e89c16e46096264f96cda0c3045898704235494de8/vmlinuz-6.1.14-200.fc37.x86_64
console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal
ostree=/ostree/boot.0/fedora-coreos/792b78931dec8d7b9ceb91e89c16e46096264f96cda0c3045898704235494de8/0
root=UUID=89415159-15c7-4fea-8cf5-5275defb7442 rw rootflags=prjquota
systemd.unified_cgroup_hierarchy=1 systemd.unified_cgroup_hierarchy=0

Only solution for us is to "clean up" manually with rpm-ostree and reboot ... that stuck for several cluster updates now:
sh-5.2# rpm-ostree kargs --delete-if-present=systemd.unified_cgroup_hierarchy=0 --append-if-missing=systemd.unified_cgroup_hierarchy=1
Staging deployment... done
Changes queued for next boot. Run "systemctl reboot" to start a reboot

sh-5.2# systemctl reboot

The issue persists on new nodes: we have to run this fix after provisioning them.
Would love to see a better fix as well :)

Hi @Bengrunt,
Thanks, that's what I also did, but I hope for a sophisticated solution (what I didn't ifnd yet) 😄
A small addition, the rpm-ostree command has --reboot parameter, if you use that you don't need the 'systemctl reboot' 😉

Good to know! rpm-ostree is so nebulous to me :/
A more permanent solution would be great for new nodes !

I ran into that problem yesterday as well.  Be sure to set nodes.config/cluster's spec.cgroupMode to v2.
https://docs.okd.io/latest/rest_api/config_apis/node-config-openshift-io-v1.html
I don't know, why it's not documented on the OKD documentation though.  That might warrant another bug report.

Interestingly the downstream documentation is more complete:
https://docs.openshift.com/container-platform/4.14/nodes/clusters/nodes-cluster-cgroups-2.html
@vrutkovs:  I thought the documentation was the same (modulo some replacements).

Check the contents of /etc/pivot/kernel-args on your nodes. In my case, the content of the file  was ADD systemd.unified_cgroup_hierarchy=0. I changed the file to REMOVE systemd.unified_cgroup_hierarchy=0 using a MachineConfig. Then I deleted the MachineConfig again. The whole process required several boot processes of the nodes. In the meantime, however, I have no problems with cgroupv1 and have completely migrated to cgroupv2.
The whole process:

Before you start, make sure that the file is not being edited by another MachineConfig. Check your rendered MachineConfigs
Add a MachineConfig which adds the file /etc/pivot/kernel-args with the content REMOVE systemd.unified_cgroup_hierarchy=0.
Wait for all nodes to reboot
Remove the MachineConfig
Wait for all nodes to reboot

/etc/pivot is an old file not used by newer machine-config-daemons.  If you do still have it, I assume you won't have nodes.config/cluster.

Interestingly the downstream documentation is more complete: https://docs.openshift.com/container-platform/4.14/nodes/clusters/nodes-cluster-cgroups-2.html
@vrutkovs: I thought the documentation was the same (modulo some replacements).

That documentation is puzzling me.
I'm used to seeing tmpfs for cgroupv1 and cgroup2fs for cgroupv2 but in the docs you mentioned it's the other way around.

Ah, right.  I noticed that as well.  It's the mixed up (also for the kernel arguments).

@ibotty yeah, seems its pointing to OKD-specific page - https://raw.githubusercontent.com/openshift/openshift-docs/master/installing/install_config/enabling-cgroup-v1.adoc
I think this is an artifact of time when OKD was cgroupsv2 by default and OCP wasn't. Please file a bug for https://github.com/openshift/openshift-docs to unify them since 4.14

/etc/pivot is an old file not used by newer machine-config-daemons. If you do still have it, I assume you won't have nodes.config/cluster.

Based on the current master branch, the file is still used with current machine-config-daemons.
https://github.com/openshift/machine-config-operator/blob/master/pkg/daemon/kernelargs.go

@Bengrunt what I usually do is I grep the journal log, searching for "cgroup v2":
[core@okd-preprod-1 ~]$ journalctl | grep "cgroup v2"
Nov 15 13:01:10 okd-preprod-1 crio[1799]: time="2023-11-15 13:01:10.802168939Z" level=info msg="Node configuration value for cgroup v2 is false"
Nov 15 13:29:50 okd-preprod-1 crio[1757]: time="2023-11-15 13:29:50.362239657Z" level=info msg="Node configuration value for cgroup v2 is true"

The last line is the relevant one, that is the value since the last update.

For info: I stabilized my 4.11 cluster by :
a) Isolation cronjobs on one node via taints/tolerations.
b) Restarting kubelet each morning on that node (and measuring kublet memory beforehand). FYI, Restarting does not affect running pods.
And... started working on a ceph storage cluster, to get off gluster and allow updating to 4.12/3/4.
I have not looked into Systemd.unified_cgroup_hierarchy=0 or /etc/pivot yet, which I probably should, looking at the recent commends above.

@Boran The cgroup garbage collector would probably help you solve the issue for your cluster.
https://github.com/okd-project/okd/issues/1310#issuecomment-1682049464

@Boran as you are working to set up Ceph on 4.11, it's better you know about the #1505 issue.

@Boran The cgroup garbage collector should probably have helped you to solve the issue for your cluster. It did not? #1310 (comment)

Does not solve the issue for me, see comment above, where I explain the GC and patches added. https://github.com/okd-project/okd/issues/1310#issuecomment-1682157225

@Boran as you are working to set up Ceph on 4.11, it's better you know about the #1505 issue.

That is really important, appreciated thanks! Its going to make the migration from gluster tricky.