[slurm-users] Can't use cgroups on debian 11 : unable to get parameter 'tasks' for '/sys/fs/cgroup/cpuset/'

[Arthur Toussaint]

Hi,
I'm trying to use cgroups in my slurm config on Debian 11 (installed through
debian apt repo), but slurmd won't start, I've attached the relevant slurmd.log
as well as slurm.conf and cgroup.conf files

The error seems to be unable to get parameter 'tasks' for '/sys/fs/cgroup/cpuset/'
I've added the `cgroup_enable=memory swapaccount=1` to the cmdline but that
doesn't help

I've seen people having those kind of problems, but no one seem to be able to solve it and keep the cgroupsThanks a lot
Arthur

[Christopher Samuel]

On 11/16/21 8:04 am, Arthur Toussaint wrote:

> I've seen people having those kind of problems, but no one seem to be
> able to solve it and keep the cgroups

Debian Bullseye switched to cgroups v2 by default which Slurm doesn't
support yet, you'll need to switch back to the v1 cgroups. The release
notes have info on how to do this here:

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#openstack-cgroups

Short version is it says you need to add this to the kernel boot params:

systemd.unified_cgroup_hierarchy=false
systemd.legacy_systemd_cgroup_controller=false

The name of that second one looks a bit misleading, it's described in
the systemd man page as:

https://manpages.debian.org/testing/systemd/systemd.1.en.html

> Takes effect if the full unified cgroup hierarchy is not used (see
previous option). When specified without an argument or with a true
argument, disables the use of "hybrid" cgroup hierarchy (i.e. a
cgroups-v2 tree used for systemd, and legacy cgroup hierarchy[10],
a.k.a. cgroups-v1, for other controllers), and forces a full "legacy"
mode. When specified with a false argument, enables the use of "hybrid"
hierarchy.

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA

[Arthur Toussaint]

Hi Crhis,

That makes sense, thanks a lot !

Arthur

---

De: "Christopher Samuel" <ch...@csamuel.org>
À: "slurm-users" <slurm...@lists.schedmd.com>
Envoyé: Mercredi 17 Novembre 2021 02:36:36
Objet: Re: [slurm-users] Can't use cgroups on debian 11 : unable to get parameter 'tasks' for '/sys/fs/cgroup/cpuset/'

[Christoph Brüning]

Hi,

I encountered an issue a while back when Slurm would expect a cgroups v1
hierarchy, but Debian 11 had switched to cgroups v2.

I don't know whether this has been tackled in Slurm since then, but I
worked around it by adding "systemd.unified_cgroup_hierarchy=0" to the
kernel command line to enable hybrid cgroup mode.

See also
https://unix.stackexchange.com/questions/480747/how-to-find-out-if-systemd-uses-legacy-hybrid-or-unified-mode-cgroupsv1-vs-cgr
to find out which mode is active on your system.

Best,
Christoph

--
Dr. Christoph Brüning
Universität Würzburg
HPC & DataManagement @ ct.qmat & RZUW
Am Hubland
D-97074 Würzburg
Tel.: +49 931 31-80499