[slurm-users] Job allocating more CPUs than requested

[Nicolas Bock]

Hi,

A job run with

sbatch --ntasks=1 \
--ntasks-per-node=1 \
--cpus-per-task=1 \ --ntasks-per-core=1 \
--sockets-per-core=1 \
--cores-per-socket=1 \
--threads-per-core=1

shows as requesting 1 CPU when in queue, but then allocates all
CPU cores once running. Why is that?

Any suggestions would be greatly appreciated,

Nick

[DENOU, MEHDI]

Hello Nick,

What is the result with only -n 1 ?
Could you provide your slurm.conf ?

A lot of parameters are involved in the allocation process. The choice between a few cores or a whole node depends mostly of "SelectType" and "Shared" in the partition definition.

Regards,

[Dmitri Chebotarov]

Hi Nick

I'm curious what makes you think job is using all CPU cores once running?
Would you be able to share output from 'ps' command while job is running?:

ps -p <pid> -L -o pid,tid,psr,pcpu

Execute it on the compute node where your job is running.
<pid> is the process id of the job/task.

Thank you.

[Chris Samuel]

On Saturday, 22 September 2018 2:53:58 AM AEST Nicolas Bock wrote:

> shows as requesting 1 CPU when in queue, but then allocates all
> CPU cores once running. Why is that?

Do you mean that Slurm expands the cores requested to all the cores on the
node or allocates the node in exclusive mode, or do you mean that the code
inside the job uses all the cores on the node instead of what was requested?

The latter is often the case for badly behaved codes and that's why using
cgroups to contain applications is so important.

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

[Ryan Novosielski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/21/2018 11:22 PM, Chris Samuel wrote:
> On Saturday, 22 September 2018 2:53:58 AM AEST Nicolas Bock wrote:
>
>> shows as requesting 1 CPU when in queue, but then allocates all
>> CPU cores once running. Why is that?
>
> Do you mean that Slurm expands the cores requested to all the cores
> on the node or allocates the node in exclusive mode, or do you mean
> that the code inside the job uses all the cores on the node instead
> of what was requested?
>
> The latter is often the case for badly behaved codes and that's why
> using cgroups to contain applications is so important.

I apologize for potentially thread hijacking here, but it's in the
spirit of the original question I guess.

We constrain using cgroups, and occasionally someone will request 1
core (-n1 -c1) and then run something that asks for way more
cores/threads, or that tries to use the whole machine. They won't
succeed obviously. Is this any sort of problem? It seems to me that
trying to run 24 threads on a single core might generate some sort of
overhead, and that I/O could be increased, but I'm not sure. What I do
know is that if someone does this -- let's say in the extreme by
running something -n24 that itself tries to run 24 threads in each
task -- and someone uses the other 23 cores, you'll end up with a load
average near 24*24+23. Does this make any difference? We have NHC set
to offline such nodes, but that affects job preemption. What sort of
choices do others make in this area?

- --
____
|| \\UTGERS, |----------------------*O*------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark
`'
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAlulxpAACgkQmb+gadEcsb543gCeOnUj+raTuEjLdYe+rfmHDiPP
kfgAn0zY0Ykm3fEOha9P25Q4m0F0/yKQ
=kI8g
-----END PGP SIGNATURE-----

[Renfro, Michael]

Anecdotally, I’ve had a user cause load averages of 10x the node’s core count. The user caught it and cancelled the job before I noticed it myself. Where I’ve seen it happen live on less severe cases, I’ve never noticed anything other than the excessive load average. Viewed from ‘top’, the offending process was still confined to its 100% CPU or whatever it had reserved.

--
Mike Renfro / HPC Systems Administrator, Information Technology Services
931 372-3601 / Tennessee Tech University

[Chris Samuel]

On Saturday, 22 September 2018 2:35:34 PM AEST Ryan Novosielski wrote:

> We constrain using cgroups, and occasionally someone will request 1
> core (-n1 -c1) and then run something that asks for way more
> cores/threads, or that tries to use the whole machine. They won't
> succeed obviously. Is this any sort of problem?

At that point as all their processes will be contesting a particular core.
Load average is just about what's trying to run but can't - and in this case
it doesn't have a relationship to how difficult it can be for other jobs to do
things because of the core restriction.

I guess it's possible the next level caches might get a work out, but then
unless you're restricting OS daemon processes to cores that are not used by
Slurm then you're probably still going to get some amount of cache pollution
anyway.

All the best!