cgroups v1: systemd & k8s on the same node using cgroupfs driver

[Andrei Enshin]

Hi node folks,

Recently I’ve got an issue with k8s 1.19.2 on a machine with systemd - https://github.com/kubernetes/kubernetes/issues/95488

Long story short: there are two systemd named cgroup mounts (hierarchies) with which kubelet has no idea what to do (me too):
expected:      /sys/fs/cgroup/systemd

unexpected:  /kubepods/burstable/pod7ffde41a-fa85-4b01-8023-69a4e4b50c55/8842def241fac72cb34fdce90297b632f098289270fa92ec04643837f5748c15

 

During investigation it turned out the kubelet can’t deal well with two mounts of the same controller (subsystem). Even though technically named mount has no controller, it’s reported by runc as with controller - https://github.com/opencontainers/runc/issues/2680.
Anyway Linux kernel allows to have more than one mount of the same controller. Kubelet expects to have only one mount per controller.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-relationships_between_subsystems_hierarchies_control_groups_and_tasks


Here is the small fix which just makes kubelet fail early and to not fool itself with using one «random» mount of the same type:
https://github.com/kubernetes/kubernetes/pull/96594

It will be good to understand the kubelet’s expectations of cgroups mounts on a machine. What is seen in code is: one mount per controller.
However main question of this topic is how does kubelet and systemd share the same hierarchies without conflicting.


I’ve read the article: https://systemd.io/CGROUP_DELEGATION/
And asked in systemd mailing list: https://lists.freedesktop.org/archives/systemd-devel/2020-November/045576.html

Seems writing into root cgroup in any hierarchy is what systemd can’t accept.

May someone explain why kubelet writes to root cgroup in all hierarchies (creates «kubepod» cgroup) and how it is in line with systemd?
Maybe there is some delegation which I missed?

I’m not experienced in all this stuff. The wordy and simply reply the better.

 

---

Best Regards,
Andrei Enshin