kubelet breaks with cgroups v1 and newer Linux dropping kmem.limit_in_bytes

[smira]

What happened?

When cgroups v1 is enabled, and Linux is updated e.g. to 6.1.54 which contains commit which drops kmem.limit_in_bytes
(see https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/Documentation/admin-guide/cgroup-v1/memory.rst?h=linux-6.1.y&id=21ef9e11205fca43785eecf7d4a99528d4de5701), the kubelet fails to start with following messages:

172.20.0.2: {"ts":1695312052628.9187,"caller":"cm/container_manager_linux.go:881","msg":"Unable to get rootfs data from cAdvisor interface","err":"unable to find data in memory cache"}
172.20.0.2: {"ts":1695312052629.2424,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system/apid\": openat2 /sys/fs/cgroup/memory/system/apid/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052629.5186,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system/etcd\": openat2 /sys/fs/cgroup/memory/system/etcd/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052629.7024,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system/runtime\": openat2 /sys/fs/cgroup/memory/system/runtime/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052629.9346,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system/dashboard\": openat2 /sys/fs/cgroup/memory/system/dashboard/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052630.1235,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system\": openat2 /sys/fs/cgroup/memory/system/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052630.3171,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/system/trustd\": openat2 /sys/fs/cgroup/memory/system/trustd/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052630.504,"caller":"manager/container.go:586","msg":"Failed to update stats for container \"/podruntime/kubelet\": openat2 /sys/fs/cgroup/memory/podruntime/kubelet/memory.kmem.limit_in_bytes: no such file or directory, continuing to push stats\n","v":0}
172.20.0.2: {"ts":1695312052632.7512,"caller":"kubelet/kubelet.go:1511","msg":"Failed to start ContainerManager","err":"failed to get rootfs info: unable to find data in memory cache"}

What did you expect to happen?

kubelet to run happily without kmem.limit_in_bytes

How can we reproduce it (as minimally and precisely as possible)?

Linux version which drops kmem.limit_in_bytes + cgroupsv1, start kubelet.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
v1.28.2

Cloud provider

QEMU VMs

OS version

Talos Linux 6.1.54

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[smira]

/sig node

[frezbo]

Kernel 6.1.55 is out which should have the change reverted:

commit 45ea58f9dbf8faecc733e4f1db06a55fc0e7e6a7
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Wed Sep 20 12:57:00 2023 +0200

    Revert "memcg: drop kmem.limit_in_bytes"
    
    This reverts commit 21ef9e11205fca43785eecf7d4a99528d4de5701 which is
    commit 86327e8eb94c52eca4f93cfece2e29d1bf52acbf upstream.
    
    It breaks existing runc systems, as the tool always thinks the file
    should be present.
    
    Reported-by: Jeremi Piotrowski <jpiotrowski@linux.microsoft.com>
    Link: https://lore.kernel.org/r/20230920081101.GA12096@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Shakeel Butt <shakeelb@google.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Roman Gushchin <roman.gushchin@linux.dev>
    Cc: Muchun Song <muchun.song@linux.dev>
    Cc: Tejun Heo <tj@kernel.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

[mmiranda96]

/triage accepted
/priority important-soon

[SergeyKanzhelev]

runc changes:

• [1.1 backport] Handle kmem.limit_in_bytes removal opencontainers/runc#4028
• Handle kmem.limit_in_bytes removal opencontainers/runc#4018

Need to wait for the runc release and pick it up

[SergeyKanzhelev]

/assign

[SergeyKanzhelev]

/priority important-soon

[k8s-triage-robot]

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[AnishShah]

@bobbypage , do we need to cut a release for cAdvisor?

[cwangVT]

The issue happened in kernel version 6.1.54, where kmem.limit_in_bytes was dropped.
But that drop was reverted in a later version 6.1.55.

In the mean time, the runc also introduced a fix to that drop in 1.1.10.

On cAdvisor side, we already bump the runc version to 1.1.12 (same with the current runc version for k/k) and cut the release, The cAdvisor version in k/k is also bumped. So this issue should be resolved now.

[cwangVT]

/close

[k8s-ci-robot]

@cwangVT: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SergeyKanzhelev]

/close

[k8s-ci-robot]

@SergeyKanzhelev: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.