[slurm-users] Expected behavior for preempt/partition_prio

[Massimo Sgaravatto via slurm-users]

Dear all

I have these settings in slurm.conf:

.

...

PreemptType=preempt/partition_prio
PreemptMode=REQUEUE

PartitionName=gpus-opp Nodes=btc-ter-gpu-[03-05],btc-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=15

PartitionName=mpicms Nodes=btc-ter-03,btc-ter-gpu-05 AllowAccounts=cms,operators State=UP PriorityTier=20

....

My understanding was that the expected behavior is that a job submitted to the gpus-opp partition is always preempted (requeued) if the resources allocated by this job are requested by a job submitted to the mpicms partition (since 20 > 15)

Did I get it right?

I am asking because I see that sometimes this doesn't happen. I am still trying to debug the issue, but it looks like also the QoS is taken into account (and this doesn't make sense to me unless I am missing something)

Thanks, Massimo

[Loris Bennett via slurm-users]

Hi Massimo,

Massimo Sgaravatto via slurm-users

Here

https://slurm.schedmd.com/qos.html#preemption

it says

Partition QOS

A QOS can be attached to a partition. This means the partition will have all
the same limits as the QOS. This does not associate jobs with the QOS, nor
does it give the job any priority or preemption characteristics of the
assigned QOS. Jobs may separately request the same QOS or a different QOS to
gain those characteristics. However, the Partition QOS limits will override
the job's QOS. If the opposite is desired you may configure the job's QOS with
Flags=OverPartQOS which will reverse the order of precedence.

Is that relevant?

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
FUB-IT, Freie Universität Berlin

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Davide DelVento via slurm-users]

Massimo,

Slurm sometimes takes into account things in a surprising way. While debugging the issue, in addition to what Loris said, I suggest that you also check the accounts (which should have been called groups, but I digrees) and associations, to see if they got in the way or what should have been happening

Cheers

[Massimo Sgaravatto via slurm-users]

Thanks, but I still don't understand why the QoS matters ?

Shouldn't only the priorityTier of the partitions be considered ?

Thanks, Massimo