[slurm-users] Fairshare +FairTree Algorithm + TRESBillingWeights

[Yap, Mike]

Hi All

 

Need some clarification on Fairshare (multifactor priority plugin) and FairTree Algorithm

 

If I read correctly, the current default for slurm is FairTree algorithm in which

1. Priority can set on various level
2. No fairshare-actual usage is being consider
3. Job submitted will run according to Priority instead of fairshare-actual usage

Questions –

1. Rawshare is only a representation of weight where higher value equal to higher priority ?
2. The total of rawshare need not to be at 100 since it is not percentage?
3. Referring to image below, am I right to believe
1. Job from User7 will always run before User2 (including those in queue)
2. Job from User3 will always run before User2 (including those in queue)
3. Job from User7 will always run before User6 (including those in queue)
4. Job from User3 will always run before User8 (including those in queue)
5. Job from User2 will always run before User1 (including those in queue)

 

 

Only by configuring multifactor priority plugin, fairshare-usage will be consider in which (referring to above image again) in which it will be common for User1(adhoc user) to have job run before User4(user with massive clocktime)

Questions

1. In scenario if both user1 and user2 having the same fairshare value (sshare showing 0.001), do User4 will still have priority over User1 ?

 

Additional question

1. how do we enforce only user define in sacctmgr have the right to submit job ? restarted the services and system but user not in sacctmgr is still able to submit job
2. User with two account, how do they define which account to use for submission ?

 

TRESBillingWeights

 

If I wish to enable TRESBillingWeight (to use CPU:MEMORY:GPU) – referring to https://slurm.schedmd.com/tres.html

 

1. Do I just include the following line to enable the option
1. AccountingStorageTRES=gres/gpu
2. TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0"
2. Is there a command to print out the billing weight is indeed as requested instead of the default cpu ?

 

Many thanks

Mike

[Luke Yeager]

• Rawshare is only a representation of weight where higher value equal to higher priority ?
• The total of rawshare need not to be at 100 since it is not percentage?

 

Look at the output of this command on your cluster and things will probably become more clear:

sshare -a –format=cluster,account,user,partition,rawshare,normshare,normusage,levelfs

 

• Job from User7 will always run before User2 (including those in queue)

 

No, not “always.” It depends on how much each user (actually, user-to-project association) has been utilizing the cluster recently. See “normusage” in the command above. If levelfs is >1, then the priority of the job will be boosted (because they have been “under-served” recently – that’s how the manpage puts it).

 

• Is there a command to print out the billing weight is indeed as requested instead of the default cpu ?

 

Take a look at the output of this command. In particular, check out the value for GrpTRESMins for your “Association Records”:

scontrol show assoc_mgr

 

Hope that helps!

Luke

 

From: slurm-users <slurm-use...@lists.schedmd.com> On Behalf Of Yap, Mike
Sent: Wednesday, March 31, 2021 4:50 PM
To: slurm...@schedmd.com
Subject: [slurm-users] Fairshare +FairTree Algorithm + TRESBillingWeights

 

External email: Use caution opening links or attachments

[Yap, Mike]

Thanks Luke.. Will go through the  2 commands (will try to digest them)

 

Wondering if you’re able to advise on TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0". Tried to include it in slurm.conf but slurm fail to start

 

Also wondering if anyone can advise on the fairshare value.  I recall reading a page explaining the how the calculation work (which is quite complicated)

Just confused why the default weight for some parameter is set at difference value ?

In following example, am I correct to believe fairshare priority will play a main role compare to partition weight and age ?

Does this mean, a new job from a new user within the same group will have his job run before a pending  jobs from existing user (queue for 30days) and submitted to a partition with higher prioritytier ?

 

PriorityType=priority/multifactor

PriorityWeightFairshare=100000

PriorityWeightAge=1000

PriorityWeightPartition=10000

[Yap, Mike]

Fix the issue with TRESBillingWeights,

It seems like I will need to set PartitionName for it to work

https://bugs.schedmd.com/show_bug.cgi?id=3753

 

PartitionName=DEFAULT TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0"

[Yap, Mike]

Also found my answer for the weight value here

https://slurm.schedmd.com/priority_multifactor.html#fairshare

IMPORTANT: The weight values should be high enough to get a good set of significant digits since all the factors are floating point numbers from 0.0 to 1.0. For example, one job could have a fair-share factor of .59534 and another job could have a fair-share factor of .50002. If the fair-share weight is only set to 10, both jobs would have the same fair-share priority. Therefore, set the weights high enough to avoid this scenario, starting around 1000 or so for those factors you want to make predominant.