[slurm-users] Different GPU types on the same server

[Emyr James]

Dear all,

We currently have a single gpu capable server with 10x RTX2080Ti in it. One of our research groups wants to replace one of these cards with an RTX3090 but only if we can give them a higher priority on that particular card.

Is it possible to set up a queue that only includes a specific subset of the cards on a server? Any way to use gres's etc. to achieve this?

Many thanks,

Emyr James
Head of IT
CRG - Centre for Genomic Regulation
C/ Dr. Aiguader, 88
Edif. PRBB
08003 Barcelona, Spain

Phone Ext: #1098

[David Gauchard]

Hello, FWIW we did this with gres.conf and slurm.conf:

in node's /etc/slurm/gres.conf:

AutoDetect=off
Name=gpu Type=quadro_k620 File=/dev/nvidia0 CPUs=0-0
Name=gpu Type=nvs_510 File=/dev/nvidia1 CPUs=1-1
Name=gpu Type=nvs_510 File=/dev/nvidia2 CPUs=2-2

in server's slurm.conf:

NodeName=gputesthost CPUs=4 RealMemory=3500 Sockets=1 CoresPerSocket=4
ThreadsPerCore=1 Gres=gpu:quadro_k620:1,gpu:nvs_510:2

When submitting jobs, gpu is selected by user by that kind of request:
sbatch --gres=gpu:quadro_k620 ...

This is on a test host with old cards.
I don't know whether `CPUs` on each line should partition the host or
not (=> CPUs=0-3 for all lines)

david