[slurm-users] reserving TRES(gpus) fails via nodes busy

[Äkäslompolo Simppa via slurm-users]

Hi!

I've been trying to reserve a couple of GPUs permanently for a specific user, but I keep getting "Requested nodes are busy" (slurm 26.05.3)

Perhaps someone could give ideas what debug flags to enable or other ways to fish out more information why slurm says the nodes are busy?

Here is an example case, but I've not figured out the systematics: (And flags=ignore_jobs does not make a difference)

# scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g,gres/gpu:h200:4 corecnt=48
Error creating the reservation: Requested nodes are busy
# scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g corecnt=48
Reservation created: llmgw_permanent
#

Here is another case:

# scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:4 flags=ignore_jobs, nodes=gpu49
Error creating the reservation: Requested nodes are busy
# scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:0 flags=ignore_jobs, nodes=gpu49
Reservation created: llmgw_permanent
#

Thank you!

--
- Simppa -
Mr. Simppa Äkäslompolo
High performance computing specialist
Doctor of Science (Tech.)
Aalto Scientific Computing
School of Science, Aalto University, Finland

+358-50-5311327
https://scicomp.aalto.fi/

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Laura Hild via slurm-users]

Your cluster nodes wouldn't happen to use RestrictedCoresPerGPU? I tried without success to make a similar reservation a few months ago and I don't remember what my conclusions were but that's the association that's popping into my head.