[slurm-users] MIG-Slice: Unavailable GRES

[Vogt, Timon]

Dear Slurm Mailing List,

I am experiencing a problem which affects our cluster and for which I am
completely out of ideas by now, so I would like to ask the community for
hints or ideas.

We run a partition on our cluster containing multiple nodes with Nvidia
A100 GPUs (40GB), which we have sliced up using Nvidia Multi-Instance
GPUs (MIG) into one 3g.20gb slice and two 2g.10gb slices per GPU.

Now, when submitting a job to it and requesting the 3g.20gb slice (like
with "srun -p mig-partition -G 3g.20gb:1 hostname"), the job runs fine,
but when a job requests one of the 2g.10gb slices instead (like with
"srun -p mig-partition -G 2g.10gb:1 hostname"), the job does not get
scheduled and the controller repeatedly outputs the error:

slurmctld[28945]: error: _set_job_bits1: job 4780824 failed to find any
available GRES on node 1471
slurmctld[28945]: error: gres_select_filter_select_and_set job 4780824
failed to satisfy gres-per-job counter

Our cluster uses the AutoDetect=nvml feature for the nodes in the
gres.conf and both slice types are defined in "AccountingStorageTRES"
and in the GRES parameter of the node definition. The slurmd on the node
also finds both types of slices and reports the correct amounts. They
are also visible in the "Gres=" section of "scontrol show node", again
in correct amounts.

I have also ensured that the nodes are not used otherwise by creating a
reservation on them accessible only to me, and I have restarted all
slurmd's and the slurmctld.

By now, I am out of ideas. Does someone here have a suggestion on what
else I can try? Has someone already seen this error and knows more about it?

Thank you very much in advance and
best regards,
Timon

--
Timon Vogt
Arbeitsgruppe "Computing"
Nationales Hochleistungsrechnen (NHR)
Scientific Employee NHR
Tel.: +49 551 39-30146, E-Mail: timon...@gwdg.de
-------------------------------------------------------------------------
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Burckhardtweg 4, 37077 Göttingen, URL: https://gwdg.de

Support: Tel.: +49 551 39-30000, URL: https://gwdg.de/support
Sekretariat: Tel.: +49 551 39-30001, E-Mail: gw...@gwdg.de

Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen, Handelsregister-Nr. B 598

Zertifiziert nach ISO 9001 und ISO 27001
-------------------------------------------------------------------------

[Groner, Rob]

At some  point when we were experimenting with MIG, I was being entirely frustrated in getting it to work until I finally removed the autodetect from gres.conf and explicitly listed the stuff instead.  THEN it worked.  I think you can find the list of files that are the device files using nvidia-smi.

Here is the entry we use in our gres.conf for one of the nodes:

NodeName=p-gc-3037 Name=gpu Type=1g.5gb File=/dev/nvidia-caps/nvidia-cap[66,75,84,102,111,120,129,201,210,219,228,237,246,255]

Something to TRY anyway.  Odd that 3g.20gb works.  You might try reconfiguring the node for that instead and see if it works then.  We've used 3g.20gb and 1g.5gb on our nodes and it works fine, never tried 2g.10gb.

Rob

---

From: slurm-users on behalf of Vogt, Timon
Sent: Wednesday, July 19, 2023 3:08 PM
To: slurm...@schedmd.com
Subject: [slurm-users] MIG-Slice: Unavailable GRES

[Vogt, Timon]

Hi Rob,

thank you very much for that hint. I tried setting the MIG slices manually in the gres.conf and it works now.

Thank you very much.
Best regards,
Timon

-- 
Timon Vogt
Arbeitsgruppe "Computing"
Nationales Hochleistungsrechnen (NHR)
Scientific Employee NHR
Tel.: +49 551 39-30146, E-Mail: timon...@gwdg.de
-------------------------------------------------------------------------
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Burckhardtweg 4, 37077 Göttingen, URL: https://gwdg.de

Support: Tel.: +49 551 39-30000, URL: https://gwdg.de/support
Sekretariat: Tel.: +49 551 39-30001, E-Mail: gw...@gwdg.de

Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen, Handelsregister-Nr. B 598

Zertifiziert nach ISO 9001 und ISO 27001
-------------------------------------------------------------------------

Am 19.07.23 um 21:21 schrieb Groner, Rob: