cgroup v1 deprecation

[Kevin Hannon]

Hello,

The latest trends in the linux community is to phase out cgroup v1.
Systemd 258 has announced the removal of cgroup v1 code so I propose
to deprecate cgroup v1 as stage 1 and in 1.38 we move to remove cgroup
v1 from the code base.

I drafted https://github.com/kubernetes/enhancements/issues/5573.

If you have concerns, please let sig-node know and comment on the KEP or issue.

Kevin Hannon

[Sean McGinnis]

I'd love to hear from some distro folks on current cgroup support plans.

If cgroupv1 is just discouraged in all current releases, that's one thing. But if it is actively being removed from upcoming releases, that raises the priority for Kubernetes to start strongly avoiding (and supporting) v1 support.

Of course it's also tricky on the other end of the spectrum. If there are other currently "supported" releases out there that default to v1, that will also cause friction. But it would be good to understand the scope of that so we can try to move forward with the least amount of pain for users.

Sean

[Tim Hockin]

The main reason to remove it is to avoid maintaining and testing it going forward. Whether some distros support it or not isn't relevant until we are sure that all of our needs are completely met by V2.  

So my question is what is preventing our users from adopting V2?

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/403b3a67-67bf-40da-b4b5-2b5f960ad37fn%40kubernetes.io.

[Benjamin Elder]

> Systemd 258 has announced the removal of cgroup v1 code so I propose

AIUI, the ecosystem has offerings that don't use systemd at all (e.g. talos?), I don't think we should necessarily be deciding based on systemd.

> The main reason to remove it is to avoid maintaining and testing it going forward. Whether some distros support it or not isn't relevant until we are sure that all of our needs are completely met by V2.  

Testing is pretty cheap in this respect, we already run CI for multiple container runtimes and versions and we don't need a 5k node scale cluster to test cgroups.

Maintenance might be less, but if we're not adding new features to this code already ...

> So my question is what is preventing our users from adopting V2?

1) If you're running minikube --vm-driver=docker (or podman), kind, k3d, etc. then you're using whatever the host distro provides. So the developer distros at minimum are relevant.

2) We instituted a breaking change in cgroupv2 to OOMKill group instead of individual processes, there's a kubelet config option now, but that requires access to configure kubelet, versus sticking to cgroupv1 nodes.
... This has caused a lot of problems for the project's own CI ... 

3) Again, distro defaults. I doubt many are tweaking kernel boot params on their nodes.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAO_RewY2MM%3Df9Rc17qQruirbDpO_X9TgygGMj%2B%2Ba0q%3Dj9aCb9A%40mail.gmail.com.

[Kevin Hannon]

I pushed a commit adding my research on linux distributions. https://github.com/kubernetes/enhancements/pull/5574/commits/d27a68bd959af18947f493b25ba10aa8e938bf08

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm9q%2BpWjmpBRYPDb1kOTYZHDMjz%3D5hfjUZ_AH23J4WSzOA%40mail.gmail.com.

[Benjamin Elder]

Thanks, also I want to note that I'm playing devil's advocate here ... KIND also gets simpler when we drop v1 so I understand the impetus, but I'm wary of triggering dockershim style blowback.

We should probably be pretty confident that impact will be low before any hard cutoff.

[Alvaro Aleman]

As an anecdotal example, Amazon Linux 2 does not support cgroup v2 and is supported until 2026-06-30, I would expect it to still see a decent bit of usage on AWS

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm-%3DwWHjALAwagEcKpLjfi%3Dn26YRHZUyPJXym0jRF3OkPQ%40mail.gmail.com.

[Benjamin Elder]

This might not be the best place to start, I recommend checking out our guide: https://www.kubernetes.dev/docs/guide/

On Thu, Oct 2, 2025 at 11:46 AM Sarah R <sarah...@gmail.com> wrote:

Hello all,

I realize this might sound unusual, but I was added to this mailing list a while ago, and due to my email not being active, I couldn’t follow the discussions. Now that I’m back, I’d really like to catch up and understand the current context.

I’m especially interested in contributing to the Linux community, and I noticed the ongoing discussion about deprecating cgroup v1. From what I gathered, systemd 258 has removed cgroup v1 code, and there’s a proposal to deprecate cgroup v1 as stage 1, followed by its removal in 1.38. I also came across the draft KEP here: https://github.com/kubernetes/enhancements/issues/5573.

Could someone help me get up to speed on this discussion and guide me on where I could best contribute?

Thanks,

Sarah 

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAAkVc2oqfuEcOJPOwrA9kp8%2BC6cjzQPfnVxMFKGv6RdcVzvrAQ%40mail.gmail.com.

[Carter McKinnon]

FWIW, EKS is dropping support for AL2 next month (Nov. 2025). Folks will absolutely use it until the lights go out, though. Any other RHEL7 derivative would be in the same boat (that distro seems immortal).

Carter

On Oct 2, 2025, at 11:25 AM, 'Alvaro Aleman' via kubernetes-sig-node <kubernete...@googlegroups.com> wrote:



You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAAkVc2oqfuEcOJPOwrA9kp8%2BC6cjzQPfnVxMFKGv6RdcVzvrAQ%40mail.gmail.com.

[Kevin Hannon]

For non systemd distributions, it looks like Talos deprecated cgroup v1 support. https://github.com/siderolabs/talos/discussions/9978

On Thursday, October 2, 2025, Carter McKinnon <carter....@gmail.com> wrote:

FWIW, EKS is dropping support for AL2 next month (Nov. 2025). Folks will absolutely use it until the lights go out, though. Any other RHEL7 derivative would be in the same boat (that distro seems immortal).

Carter

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-node+unsub...@googlegroups.com.

[Antonio Ojea]

I think that for any deprecation and removal, the most important thing
is to understand the user impact, that to me is about answering these
two questions:

1. What's a reasonable estimate of the number of users who will be
impacted, those that are still on cgroups v1? we can infer these
numbers by checking distros defaults and timelines.

2. For that group of users, what is preventing them from moving to cgroups v2?
This is exactly the question Tim raised, and I think it's the most
critical one we need to answer, and it will also be important to
define what represents a "valid blocker".

I think that a reasonable expectation is that the user's environment
should still be actively supported by the entire OSS ecosystem: the
Kernel, the golang toolchain and other dependencies.
IMHO I do not think that someone using an EOL kernel
https://www.kernel.org/category/releases.html or an unmaintained
project should be something that adds maintenance work to the
Kubernetes community and should block this process.

I also think the KEP is already doing a great job of documenting this
and gathering the necessary data to answer these questions.

On Fri, 3 Oct 2025 at 02:11, Kevin Hannon <kanno...@gmail.com> wrote:
>
> For non systemd distributions, it looks like Talos deprecated cgroup v1 support. https://github.com/siderolabs/talos/discussions/9978
>
> On Thursday, October 2, 2025, Carter McKinnon <carter....@gmail.com> wrote:
>>
>> FWIW, EKS is dropping support for AL2 next month (Nov. 2025). Folks will absolutely use it until the lights go out, though. Any other RHEL7 derivative would be in the same boat (that distro seems immortal).
>>
>> Carter
>>

>> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

>> To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAAkVc2oqfuEcOJPOwrA9kp8%2BC6cjzQPfnVxMFKGv6RdcVzvrAQ%40mail.gmail.com.
>

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-node/CANcg%3D9%2BciWtFEyFkNHU6Pf3HVm6Db6Qut4yE671JLYxaphZOZQ%40mail.gmail.com.

[Kevin Hannon]

I started a thread in slack on #sig-architecture about this.

But generally, my confusion lies in the kubernetes upstream commitment to supporting enterprise linux distributions.

Generally it sounds like Fedora or Debian are not well representative of the enterprise linux community. They are bleeding edge distributions.
At least in sig-node, we rely on these distributions for our e2e testing so I was thinking as these become harder to support cgroup v1 maybe we can consider

deprecation cgroup v1.
We still have midstream linux distributions like centos stream that provide support for cgroup v1 via support contracts.

For example, centos 8 went EOL in 2021 but distributions like RHEL 8, AlmaLinux 8 and Rocky 8 provide maintenance support that ends in 2029. 
I think the chance that there is a user that still uses these distributions with kubernetes is relatively high.

So do we follow EOL distributions of projects like RHEL, Alma or Rocky?

ref: https://github.com/opencontainers/runtime-spec/issues/1251

[Antonio Ojea]

> define what represents a "valid blocker".

> I do not think that someone using an EOL kernel
https://www.kernel.org/category/releases.html or an unmaintained
project should be something that adds maintenance work to the
Kubernetes community and should block this process.

IMHO I do not think we should be blocked by those specific distros and
that does not represent a "valid blocker"

IIUIC Rocky 8, Alma linux 8 and RHEL8 is kernel 4.18.0-553 that is
maintained as a forked kernel
https://wiki.rockylinux.org/rocky/version/#current-supported-releases
https://wiki.almalinux.org/release-notes/8.10.html#providing-feedback-and-getting-help

if we create a hard dependency on those distros we can end in
situations where some new features do not work there, and we can not
depend on the forker kernel maintainers to backport the necessary
patches.

> To view this discussion visit https://groups.google.com/d/msgid/kubernetes-sig-node/CALSq1yX%2BefYg9ctOraKxPfWKh4NPaZxkY%2BEXbY9xygn3HUkW3A%40mail.gmail.com.

[Dan Winship]

On 10/2/25 2:24 PM, 'Alvaro Aleman' via dev wrote:
> As an anecdotal example, Amazon Linux 2 does not support cgroup v2 and

> is supported until 2026-06-30, <https://aws.amazon.com/amazon-linux-2/
> faqs/#topic-0> I would expect it to still see a decent bit of usage on AWS

Enterprise distros are "supported" for a long time, but after a while
there's an assumption that you're supporting them in running their
existing workloads, not supporting them in adopting new bleeding-edge
software...

-- Dan

>
> On Thu, 2 Oct 2025 at 14:14, 'Benjamin Elder' via dev <d...@kubernetes.io
> <mailto:d...@kubernetes.io>> wrote:
>
> Thanks, also I want to note that I'm playing devil's advocate
> here ... KIND also gets simpler when we drop v1 so I understand the
> impetus, but I'm wary of triggering dockershim style blowback.
>
> We should probably be pretty confident that impact will be low
> before any hard cutoff.
>
> On Thu, Oct 2, 2025 at 10:36 AM Kevin Hannon <keha...@redhat.com
> <mailto:keha...@redhat.com>> wrote:
>
> I pushed a commit adding my research on linux
> distributions. https://github.com/kubernetes/enhancements/
> pull/5574/commits/d27a68bd959af18947f493b25ba10aa8e938bf08

> <https://github.com/kubernetes/enhancements/pull/5574/commits/
> d27a68bd959af18947f493b25ba10aa8e938bf08>

> ... This has caused a lot of problems <https://github.com/
> kubernetes-sigs/prow/issues/210> for the project's own CI ... 

>
> 3) Again, distro defaults. I doubt many are tweaking kernel
> boot params on their nodes.
>
>
> On Thu, Oct 2, 2025 at 7:49 AM 'Tim Hockin' via dev
> <d...@kubernetes.io <mailto:d...@kubernetes.io>> wrote:
>
> The main reason to remove it is to avoid maintaining and
> testing it going forward. Whether some distros support
> it or not isn't relevant until we are sure that all of
> our needs are completely met by V2.  
>
> So my question is what is preventing our users from
> adopting V2?
>
> On Thu, Oct 2, 2025, 7:03 AM Sean McGinnis
> <sean.m...@gmail.com

> enhancements/issues/5573 <https://github.com/
> kubernetes/enhancements/issues/5573>.

>
> If you have concerns, please let sig-node know
> and comment on the KEP or issue.
>
> Kevin Hannon
>
> --
> You received this message because you are subscribed
> to the Google Groups "dev" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to
> dev+uns...@kubernetes.io

> <mailto:dev+uns...@kubernetes.io>.

> To view this discussion visit https://
> groups.google.com/a/kubernetes.io/d/msgid/
> dev/403b3a67-67bf-40da-

> b4b5-2b5f960ad37fn%40kubernetes.io <https://
> groups.google.com/a/kubernetes.io/d/msgid/
> dev/403b3a67-67bf-40da-
> b4b5-2b5f960ad37fn%40kubernetes.io?
> utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to
> the Google Groups "dev" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to dev+uns...@kubernetes.io

> <mailto:dev+uns...@kubernetes.io>.

> To view this discussion visit https://groups.google.com/
> a/kubernetes.io/d/msgid/dev/

> CAO_RewY2MM%3Df9Rc17qQruirbDpO_X9TgygGMj%2B%2Ba0q%3Dj9aCb9A%40mail.gmail.com <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAO_RewY2MM%3Df9Rc17qQruirbDpO_X9TgygGMj%2B%2Ba0q%3Dj9aCb9A%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the
> Google Groups "dev" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to dev+uns...@kubernetes.io

> <mailto:dev+uns...@kubernetes.io>.

> To view this discussion visit https://groups.google.com/a/
> kubernetes.io/d/msgid/dev/

> CAOZRXm9q%2BpWjmpBRYPDb1kOTYZHDMjz%3D5hfjUZ_AH23J4WSzOA%40mail.gmail.com <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm9q%2BpWjmpBRYPDb1kOTYZHDMjz%3D5hfjUZ_AH23J4WSzOA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to dev+uns...@kubernetes.io

> <mailto:dev+uns...@kubernetes.io>.

> To view this discussion visit https://groups.google.com/a/

> kubernetes.io/d/msgid/dev/CAOZRXm-
> %3DwWHjALAwagEcKpLjfi%3Dn26YRHZUyPJXym0jRF3OkPQ%40mail.gmail.com
> <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm-
> %3DwWHjALAwagEcKpLjfi%3Dn26YRHZUyPJXym0jRF3OkPQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dev+uns...@kubernetes.io

> <mailto:dev+uns...@kubernetes.io>.

> To view this discussion visit https://groups.google.com/a/kubernetes.io/
> d/msgid/dev/

> CAAkVc2oqfuEcOJPOwrA9kp8%2BC6cjzQPfnVxMFKGv6RdcVzvrAQ%40mail.gmail.com
> <https://groups.google.com/a/kubernetes.io/d/msgid/dev/
> CAAkVc2oqfuEcOJPOwrA9kp8%2BC6cjzQPfnVxMFKGv6RdcVzvrAQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

[Tim Hockin]

I agree with Dan's sentiment, but what I want to understand most is if there's a real reason to remove code from Kubernetes, or if it's just an opportunistic cleanup. 

Don't get me wrong, I love deleting old unused unmaintained code.  The question is not whether it's worth doing, the question is when is it worth doing?  If we leave that code in tree for another year or even 3 years, do bad things happen? I honestly don't know that area of the code very well.

I know there are testing problems, and I know that every time we add a new feature or fix a bug, we have to decide whether to fix it in the old code or not.  Are there other considerations?

If the cost of leaving it for a while is not high, my preference is to leave it and let the usage drain as naturally as possible.  If it's actually a substantial problem to keep, then we can talk about how to ease the pain for as many customers as possible who are still using V1.

Tim

[Kevin Hannon]

So to me, the main problem is maintaining test infrastructure for cgroup v1 and how much effort we put in these jobs to keep running.

If there is a failure on a cgroup v1 related node, the interest in fixing this from the linux community is pretty low.
I was motivated to bring up cgroup v1 deprecation because of https://github.com/kubernetes/kubernetes/issues/134142#issuecomment-3325033801.

We could probably more easily support this if we use something like AlmaLinux 8 or Rocky 8 in our CI. Maybe AlmaLinux 9 or Rocky 9 would be fine as cgroup v1 is deprecated in those distributions but not yet removed. 

Benjamin Elder mentioned that we already use AlmaLinux 8 in some of our kind tests.

So I guess that could be an option if we still want to support cgroup v1 until the linux distributions fully remove it.

[Benjamin Elder]

We release a lot of things with testing gaps ... I think it would be inconsistent to rush to remove for that reason at the moment.

I investigated this a bit yesterday, and it turns out that the current SIG Node e2e release blocking signal is not replicated to any of the release branches, not even 1.34.
So we're clearly releasing cgroup v2 with coverage gaps as well for any backports.

How often are we actually seeing cgroupv1-specific bugs, with it in maintenance mode?

I reached out to #containerd-dev in the CNCF slack and AFAICT there are no plans to drop cgroup v1 there.

The kernel, runc, and docker don't seem to either, I think the tangents about kernels have gotten a bit off-track, what you need is an *init* that still works, the kernel is not dropping support.

[Kevin Hannon]

I pushed an update to the KEP where we removed the time when we push for removal.

The KEP proposes moving failCgroupv1 from false to true and clearly stating that cgroup v1 is deprecated. If someone is still using cgroup v1 they can turn off that flag.

We will still support testing on cgroup v1 but we can clearly state that we have plans to remove in the future.

I am not aware of any cgroup v1 bugs we have receive since we put cgroup v1 in maintenance mode.  

Kevin

>                     <mailto:dev+unsubscribe@kubernetes.io>.

>                     To view this discussion visit https://
>                     groups.google.com/a/kubernetes.io/d/msgid/
>                     dev/403b3a67-67bf-40da-
>                     b4b5-2b5f960ad37fn%40kubernetes.io <https://
>                     groups.google.com/a/kubernetes.io/d/msgid/
>                     dev/403b3a67-67bf-40da-
>                     b4b5-2b5f960ad37fn%40kubernetes.io?
>                     utm_medium=email&utm_source=footer>.
>
>                 --
>                 You received this message because you are subscribed to
>                 the Google Groups "dev" group.
>                 To unsubscribe from this group and stop receiving emails
>                 from it, send an email to dev+uns...@kubernetes.io

>                 <mailto:dev+unsubscribe@kubernetes.io>.

>                 To view this discussion visit https://groups.google.com/
>                 a/kubernetes.io/d/msgid/dev/
>                 CAO_RewY2MM%3Df9Rc17qQruirbDpO_X9TgygGMj%2B%2Ba0q%3Dj9aCb9A%40mail.gmail.com <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAO_RewY2MM%3Df9Rc17qQruirbDpO_X9TgygGMj%2B%2Ba0q%3Dj9aCb9A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
>             --
>             You received this message because you are subscribed to the
>             Google Groups "dev" group.
>             To unsubscribe from this group and stop receiving emails
>             from it, send an email to dev+uns...@kubernetes.io

>             <mailto:dev+unsubscribe@kubernetes.io>.

>             To view this discussion visit https://groups.google.com/a/
>             kubernetes.io/d/msgid/dev/
>             CAOZRXm9q%2BpWjmpBRYPDb1kOTYZHDMjz%3D5hfjUZ_AH23J4WSzOA%40mail.gmail.com <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm9q%2BpWjmpBRYPDb1kOTYZHDMjz%3D5hfjUZ_AH23J4WSzOA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
>     --
>     You received this message because you are subscribed to the Google
>     Groups "dev" group.
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to dev+uns...@kubernetes.io

>     <mailto:dev+unsubscribe@kubernetes.io>.

>     To view this discussion visit https://groups.google.com/a/
>     kubernetes.io/d/msgid/dev/CAOZRXm-
>     %3DwWHjALAwagEcKpLjfi%3Dn26YRHZUyPJXym0jRF3OkPQ%40mail.gmail.com
>     <https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAOZRXm-
>     %3DwWHjALAwagEcKpLjfi%3Dn26YRHZUyPJXym0jRF3OkPQ%40mail.gmail.com?
>     utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dev+uns...@kubernetes.io

> <mailto:dev+unsubscribe@kubernetes.io>.