[slurm-users] GPU usage in fair share factor

[Massimo Sgaravatto via slurm-users]

Dear all

I have these settings in my slurm.conf:

PriorityType=priority/multifactor
PriorityDecayHalfLife=7-0
PriorityFavorSmall=NO
PriorityMaxAge=10-0
PriorityWeightAge=100000
PriorityWeightFairshare=1000000
PriorityWeightJobSize=100000
PriorityWeightPartition=100000
PriorityWeightQOS=100000
PriorityFlags=ACCRUE_ALWAYS,FAIR_TREE
AccountingStorageTRES=gres/gpu,gres/gpu:nvidia-h100,gres/gpu:nvidia-l40s


If I have got it right, with these settings, when calculating the Job_priority (I am referring to the formula in
https://slurm.schedmd.com/priority_multifactor.html) the fair-share_factor is calculated only considering cores*seconds. So if a job also used some GPUs, this is not taken into account. Am I right ?


If I want to take memory and GPUs into account as well (with different weights for different GPU models), my understanding is that I should use the TRESBillingWeights attribute, setting e.g:

TRESBillingWeights="CPU=1.0,Mem=0.25G,gres/gpu=nvidia-h100=10.0,gres/gpu=nvidia-l40s=5.0"


Is this correct ?

Thanks, Massimo

[Paul Edmon via slurm-users]

That is correct. You need to set the TRESBillingWeight to get billing
for gpus.

-Paul Edmon-

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Massimo Sgaravatto via slurm-users]

Thanks a lot, but I am unable to set different weights for different partitions.

I tried with:

PartitionName=DEFAULT TRESBillingWeights="CPU=1.0,Mem=0.25G,gres/gpu=nvidia-h100=20.0,gres/gpu=nvidia-l40s=10.0"

but:

 [2025-12-12T14:59:14.741] error: Invalid unit type 'n'. Possible options are 'KMGTP' 

[2025-12-12T14:59:14.741] fatal: failed to parse tres weights str 'CPU=1.0,Mem=0.25G,gres/gpu=nvidia-h100=20.0,gres/gpu=nvidia-l40s=10.0'

while:

PartitionName=DEFAULT TRESBillingWeights="CPU=1.0,Mem=0.25G,gres/gpu=20"

works (or at least it is accepted)

Thanks, Massimo

[Paul Edmon via slurm-users]

I will admit I have never tried to bill on a mixed way like this for gpu. I have you tried doing gres/gpu:nvidia-l40s=10.0 I think the extra = is throwing things off.

-Paul Edmon-

[Massimo Sgaravatto via slurm-users]

Indeed I guess the right syntax is:

TRESBillingWeights=CPU=1.0,Mem=0.25G,gres/gpu:nvidia-h100=20.0,gres/gpu:nvidia-l40s=10.0

Thanks again !

Regards, Massimo