[slurm-users] How to limit # of execution slots for a given node

[David Henkemeyer]

All,

When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory.

What is the best way to do this with Slurm?  I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore.  This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard.  This is a less than ideal solution, it seems to me.  At least, it left me feeling like there has to be a better way. 

Thanks!
David

[Ole Holm Nielsen]

Hi David,

If your goal is to limit the amount of RAM memory per job, then kernel
Cgroups is probably the answer. I've collected some information in my
Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#cgroup-configuration

If some users need more RAM than available for 1 core, they have to submit
jobs for a larger number of cores to get it. This makes a lot of sense, IMHO.

SchedMD is working on the use of Cgroups v2, see the talk "Slurm 21.08 and
Beyond" by Tim Wickberg, SchedMD, https://slurm.schedmd.com/publications.html

You could probably "fool" Slurm as you describe it, but that shouldn't be
necessary.

I hope this helps.

/Ole

[Rémi Palancher]

I'm not sure you can lie to Slurm about the real number of CPUs on the nodes.

If you want to prevent Slurm from allocating more than n CPUs below the total number of CPUs of these nodes, I guess one solution is to use MaxCPUsPerNode=n at the partition level.

You can also mask "system" CPUs with CpuSpecList at node level.

The later is better if you need fine grain control over the exact list of reserved CPUs regarding NUMA topology or whatever.

--
Rémi Palancher
Rackslab: Open Source Solutions for HPC Operations
https://rackslab.io

[Paul Edmon]

You can actually spoof the number of cores and RAM on a node by using
the config_override option.  I've used that before for testing
purposes.  Mind you core binding and other features like that will not
work if you start spoofing the number of cores and ram, so use with caution.

-Paul Edmon-

[Paul Edmon]

Also I recommend setting:

CoreSpecCount

Number of cores reserved for system use. These cores will not be available for allocation to user jobs. Depending upon the TaskPluginParam option of SlurmdOffSpec, Slurm daemons (i.e. slurmd and slurmstepd) may either be confined to these resources (the default) or prevented from using these resources. Isolation of the Slurm daemons from user jobs may improve application performance. If this option and CpuSpecList are both designated for a node, an error is generated. For information on the algorithm used by Slurm to select the cores refer to the core specialization documentation ( https://slurm.schedmd.com/core_spec.html ).

and

MemSpecLimit

Amount of memory, in megabytes, reserved for system use and not available for user allocations. If the task/cgroup plugin is configured and that plugin constrains memory allocations (i.e. TaskPlugin=task/cgroup in slurm.conf, plus ConstrainRAMSpace=yes in cgroup.conf), then Slurm compute node daemons (slurmd plus slurmstepd) will be allocated the specified memory limit. Note that having the Memory set in SelectTypeParameters as any of the options that has it as a consumable resource is needed for this option to work. The daemons will not be killed if they exhaust the memory allocation (ie. the Out-Of-Memory Killer is disabled for the daemon's memory cgroup). If the task/cgroup plugin is not configured, the specified memory will only be unavailable for user allocations.

These will restrict specific memory and cores for system use.  This is probably the best way to go rather than spoofing your config.

-Paul Edmon-

On 1/7/2022 2:36 AM, Rémi Palancher wrote:

[Ryan Cox]

David,

There are several possible answers depending on what you hope to
accomplish.  What exactly is the issue that you're trying to solve? Do
you mean that you have users who need, say, 8 GB of RAM per core but you
only have 4 GB of RAM per core on the system and you want a way to
account for that?  Or is it something else?

Ryan