[slurm-users] Node OverSubscribe even if set to no

[Stéphane Larose]

Hello,

 

I have Slurm 17.11 installed on a 64 cores server. My 9 partitions are set with OverSubscribe=NO. I would expect that when all 64 cores are assigned to jobs, Slurm would just put new jobs in PENDING state. But it starts running new jobs so that more than 64 cores are assigned. Looking at the slurmctld log, we can see that cores 21, 22 and 24 to 38 are used in more than one partition right now:

 

[2018-04-16T15:00:00.439] node:katak cpus:64 c:8 s:8 t:1 mem:968986 a_mem:231488 state:11

[2018-04-16T15:00:00.439] part:ibismini rows:1 prio:10

[2018-04-16T15:00:00.439]   row0: num_jobs 6: bitmap: 4,6-12,16-33,48-55

[2018-04-16T15:00:00.439] part:ibisinter rows:1 prio:10

[2018-04-16T15:00:00.439]   row0: num_jobs 1: bitmap: 24-41

[2018-04-16T15:00:00.439] part:ibismax rows:1 prio:10

[2018-04-16T15:00:00.439]   row0: num_jobs 3: bitmap: 21-22,24-38,42-47,56-63

[2018-04-16T15:00:00.439] part:rclevesq rows:1 prio:10

[2018-04-16T15:00:00.439] part:ibis1 rows:1 prio:10

[2018-04-16T15:00:00.439] part:ibis2 rows:1 prio:10

[2018-04-16T15:00:00.439]   row0: num_jobs 1: bitmap: 32-37

 

So some jobs are now sharing the same cores but I don’t understand why since OverSubscribe is set to no.

 

Thanks for your help!

 

---

Stéphane Larose

Analyste de l'informatique

Institut de Biologie Intégrative et des Systèmes (IBIS)

Pavillon Charles-Eugène-Marchand

Université Laval

 

[Chris Samuel]

On Tuesday, 17 April 2018 5:26:26 AM AEST Stéphane Larose wrote:

> So some jobs are now sharing the same cores but I don’t understand why since
> OverSubscribe is set to no.

You might want to double check the config is acting as expected with:

scontrol show part | fgrep OverSubscribe

Also what does this say?

scontrol show config | fgrep SelectTypeParameters

I note that if you've got CR_Memory then:

CR_Memory
Memory is a consumable resource. NOTE: This
implies OverSubscribe=YES or OverSubscribe=FORCE
for all partitions. Setting a value for DefMem‐
PerCPU is strongly recommended.

cheers,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

[Stéphane Larose]

Hi Chris,

> You might want to double check the config is acting as expected with:
>
> scontrol show part | fgrep OverSubscribe

PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO

> Also what does this say?
>
> scontrol show config | fgrep SelectTypeParameters

SelectTypeParameters = CR_CPU_MEMORY

From the doc, it seems that only CR_Memory implies OverSubscribe=YES :
All CR_s assume OverSubscribe=No or OverSubscribe=Force EXCEPT for CR_MEMORY which assumes OverSubscribe=Yes

When I do "scontrol list jobs", all jobs have OverSubscribe=OK (which is not Yes). Again from the docs it seems fine: "OK" otherwise (typically allocated dedicated CPUs)

Thanks again,

Stéphane

-----Message d'origine-----
De : slurm-users <slurm-use...@lists.schedmd.com> De la part de Chris Samuel
Envoyé : 17 avril 2018 04:29
À : slurm...@lists.schedmd.com
Objet : Re: [slurm-users] Node OverSubscribe even if set to no

[Stéphane Larose]

Hi all,

I found out a way to avoid oversubscribing. I had to comment this configuration:

PreemptMode=Suspend,Gang
PreemptType=preempt/partition_prio

In my actual configuration, all the partitions are at the same priority. At times, I increase the priority of a partition and jobs in other partitions are suspended. That works fine. But I still do not understand why oversubscribing occurs when preemption is activated. I would like to keep preemption by suspending and not get oversubscription. If anyone have an idea of how to do this.

Thank you!

Stéphane

-----Message d'origine-----
De : Stéphane Larose
Envoyé : 17 avril 2018 10:02
À : 'Slurm User Community List' <slurm...@lists.schedmd.com>
Objet : RE: [slurm-users] Node OverSubscribe even if set to no

Hi Chris,

> You might want to double check the config is acting as expected with:
>
> scontrol show part | fgrep OverSubscribe

PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO
PriorityJobFactor=10 PriorityTier=10 RootOnly=NO ReqResv=NO OverSubscribe=NO

> Also what does this say?
>
> scontrol show config | fgrep SelectTypeParameters

SelectTypeParameters = CR_CPU_MEMORY

From the doc, it seems that only CR_Memory implies OverSubscribe=YES :
All CR_s assume OverSubscribe=No or OverSubscribe=Force EXCEPT for CR_MEMORY which assumes OverSubscribe=Yes

When I do "scontrol list jobs", all jobs have OverSubscribe=OK (which is not Yes). Again from the docs it seems fine: "OK" otherwise (typically allocated dedicated CPUs)

Thanks again,

Stéphane

-----Message d'origine-----
De : slurm-users <slurm-use...@lists.schedmd.com> De la part de Chris Samuel Envoyé : 17 avril 2018 04:29 À : slurm...@lists.schedmd.com Objet : Re: [slurm-users] Node OverSubscribe even if set to no