[slurm-users] GPUs not available after making use of all threads?

[Sebastian Schmutzhard-Höfler]

Dear all,

we have a node with 2 x 64 CPUs (with two threads each) and 8 GPUs, running slurm 22.05.5

In order to make use of individual threads, we changed


SelectTypeParameters=CR_Core
NodeName=nodename CPUs=256 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2

to

SelectTypeParameters=CR_CPU
NodeName=nodename CPUs=256

We are now able to allocate individual threads to jobs, despite the following error in slurmd.log:

error: Node configuration differs from hardware: CPUs=256:256(hw) Boards=1:1(hw) SocketsPerBoard=256:2(hw) CoresPerSocket=1:64(hw) ThreadsPerCore=1:2(hw)

However, it appears that since this change, we can only make use of 4 out of the 8 GPUs.
The output of "sinfo -o %G" might be relevant.

In the first situation it was

$ sinfo -o %G
GRES
gpu:A100:8(S:0,1)

Now it is:

$ sinfo -o %G
GRES
gpu:A100:8(S:0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126)

Has anyone faced this or a similar issue and can give me some directions?
Best wishes

Sebastian

 

[Hermann Schwärzler]

Hi Sebastian,

we did a similar thing just recently.

We changed our node settings from

NodeName=DEFAULT CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=32
ThreadsPerCore=2

to

NodeName=DEFAULT Boards=1 SocketsPerBoard=2 CoresPerSocket=32
ThreadsPerCore=2

in order to make use of individual hyper-threads possible (we use this
in combination with
SelectTypeParameters=CR_Core_Memory).

This works as expected: after this, when e.g. asking for
--cpus-per-task=4 you will get 4 hyper-threads (2 cores) per task
(unless you also specify e.g. "--hint=nomultithread").

So you might try to remove the "CPUs=256" part of your
node-specification to let Slurm do that calculation of the number of
CPUs itself.


BTW: on a side-note: as most of our users do not bother to use
hyper-threads or even do not want to as their programs might suffer from
doing so, we made "--hint=nomultithread" the default in our installation
by adding

CliFilterPlugins=cli_filter/lua

to our slurm.conf and creating a cli_filter.lua file in the same
directory as slurm.conf, that contains this

function slurm_cli_setup_defaults(options, early_pass)
options['hint'] = 'nomultithread'

return slurm.SUCCESS
end

(see also
https://github.com/SchedMD/slurm/blob/master/etc/cli_filter.lua.example).
So if user really want to use hyper-threads they have to add
"--hint=multithread" to their job/allocation-options.

Regards,
Hermann

[Sebastian Schmutzhard-Höfler]

Hi Hermann,

Using your suggested settings did not work for us.

When trying to allocate a single thread with --cpus-per-task=1, it still
reserved a whole CPU (two threads). On the other hand, when requesting
an even number of threads, it does what it should.

However, I could make it work by using

SelectTypeParameters=CR_Core
NodeName=nodename Sockets=2 CoresPerSocket=128 ThreadsPerCore=1

instead of

SelectTypeParameters=CR_Core
NodeName=nodename Sockets=2 CoresPerSocket=64 ThreadsPerCore=2

So your suggestion brought me in the right direction. Thanks!

If anyone thinks this is complete nonsense, please let me know!

Best wishes,

Sebastian

[Hermann Schwärzler]

Hi Sebastian,

I am glad I could help (although not exactly as expected :-).

With your node-configuration you are "circumventing" how Slurm behaves,
when using "CR_Core": if you read the respective part in

https://slurm.schedmd.com/slurm.conf.html

it says:

"CR_Core
[...] On nodes with hyper-threads, each thread is counted as a CPU to
satisfy a job's resource requirement, but multiple jobs are not
allocated threads on the same core."

That's why you got a full core (both threads) when allocating a singe
CPU. Or e.g. four threads when allocating three CPUs asf.

"Lying" to Slurm about the actual hardware-setup helps to avoid this
behaviour but are you really confident with potentially running two
different jobs on the hyper-threads of the same core?

Regards,
Hermann

[Brian Andrus]

Hermann makes a good point.

The concept of hyper-threading is not doubling cores. It is a single
core that can 'instantly' switch work from one process to another. Only
one is being worked on at any given time.

So if I request a single core on a hyper-threaded system, I would not be
pleased to find you are giving it to someone else 1/2 the time. I would
need to have the actual core assigned. If I request multiple cores and
my app is only going to affect itself, then I _may_ benefit from
hyper-threading.

In general, enabling hyper-threading is not the best practice for
efficient HPC jobs. The goal is that every process is utilizing the CPU
as close to 100% as possible, which would render hyper-threading moot.

Brian Andrus

[Sebastian Schmutzhard-Höfler]

Hi Brian and Hermann,

true. This makes a lot of sense. I will consider setting up Hermann's
configuration, defaulting to "--hint=nomultithread".

Thanks!

Sebastian

[Diego Zuccato]

I think that's incorrect:

> The concept of hyper-threading is not doubling cores. It is a single
> core that can 'instantly' switch work from one process to another.
> Only one is being worked on at any given time.

A core can have multiple (usually 2) independent execution pipelines, so
that multiple instructions from different threads run concurrently. It
does not switch from one to the other.
But it does have some shared resources, like the MMU and sometimes the
FPU (maybe only on older AMD processors). Having a single MMU means that
all the instructions running on a core must have the same "view" of the
memory space, and that means that they must come from a single process.
IOW that they're multiple threads of a single process.

If the sw you're going to run makes good use of multithreading, having
hyperthreading can pe a great boost. If the sw only uses multitasking,
then hyperthreading is a net loss (not only you can't use half the
available threads, you also usually get slower clock speeds).

Diego

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

[Brian Andrus]

Diego,

Not to start a debate, I guess it is in how you look at it.

From Intel's descriptions:

How does Hyper-Threading work? When Intel® Hyper-Threading Technology is active, the CPU exposes two execution contexts per physical core. This means that one physical core now works like two “logical cores” that can handle different software threads. The ten-core Intel® Core™ i9-10900K processor, for example, has 20 threads when Hyper-Threading is enabled.

Two logical cores can work through tasks more efficiently than a traditional single-threaded core. By taking advantage of idle time when the core would formerly be waiting for other tasks to complete, Intel® Hyper-Threading Technology improves CPU throughput (by up to 30% in server applications³).

So if we are creating code that is hypothetically 100% efficient (it can use the CPU 100% of the time), there would be no 'idle' time for another process. If work is done on that other process, it would be at the expense of the 100% efficiency enjoyed by our 'perfect' process.

Of course, the true performance answer lies in how any of the processes work, which is why some of us do so many experimental runs of jobs and gather timings. We have yet to see a 100% efficient process, but folks are improving things all the time.

Brian Andrus

[Diego Zuccato]

Not a debate, we're saying nearly the same thing but at different
"granularity". If you consider the core as a whole that's the effect you
see.

But a core is composed of different units (fetcher, decode/execute,
registers, ALU, FPU, MMU, etc). The concept behind hyperthreading is
having some of these units duplicated, while some of the others (notably
the MMU, caches and busses) remain shared.
The doubled FPU is a nearly perfect example where multithreading can
nearly double throughput: being a relatively slow component, if you
interleave correctly the two threads, while one FPU is working you load
the other with no (or minimal) interference. OTOH if you mostly do
integer math or logical operations, you'll see small/no improvement.

Diego

Il 14/02/2023 17:28, Brian Andrus ha scritto:
> Diego,
>
> Not to start a debate, I guess it is in how you look at it.
>
> From Intel's descriptions:
>
> How does Hyper-Threading work? When Intel® Hyper-Threading Technology is
> active, the CPU exposes two execution contexts per physical core. This
> means that one physical core now works like two “logical cores” that can
> handle different software threads. The ten-core Intel® Core™ i9-10900K

> <https://www.intel.com/content/www/us/en/gaming/i9-desktop-processors-for-gaming.html>processor, for example, has 20 threads when Hyper-Threading is enabled.

>
> Two logical cores can work through tasks more efficiently than a

> traditional single-threaded core. /By taking advantage of idle time when
> the core would formerly be waiting for other tasks to complete/, Intel®

> Hyper-Threading Technology improves CPU throughput (by up to 30% in

> server applications^3 ).