[slurm-users] Reservation to exceed time limit on a partition for a user

[Matthew BETTINGER]

Hello,

We are running slurm 17.02.6 with accounting on a cray CLE system.

We currently have a 24 hour job run limit on our partitions and a user needs to run a job which will exceed 24 hours runtime. I tried to make a reservation as seen below allocating the user 36 hours to run his job but it was killed at the 24 hour run limit. Can someone explain what is going on and what is the proper way to allow a user to exceed the partition time limit without having to modify slurm.conf and push it out to all of the nodes, run ansible plays and reconfigure etc. I thought that this is what reservation was for.

Here is the reservation I created that failed when it ran over 24 hours

scontrol show res
ReservationName=CoolBreeze StartTime=2018-12-27T10:08:11 EndTime=2018-12-28T22:08:11 Duration=1-12:00:00
Nodes=nid00[192-239] NodeCnt=48 CoreCnt=480 Features=(null) PartitionName=GPU Flags=
TRES=cpu=960
Users=coolbreeze Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a

Here is the partition with the resources the user needs to run on

PartitionName=GPU
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=1-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
Nodes=nid00[192-255]
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=EXCLUSIVE
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=1280 TotalNodes=64 SelectTypeParameters=NONE
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

Thanks!

[Matthew BETTINGER]

Answering my own question here. I created a hidden partition which shows like this

PartitionName=FOO
AllowGroups=ALL AllowAccounts=rt AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=YES GraceTime=0 Hidden=YES
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
Nodes=nid00[192-255]
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO

OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=1280 TotalNodes=64 SelectTypeParameters=NONE
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED

I can run jobs in there but if I set it to just a user (myself) then the job does not run. I may have to just make this partition like this until I can figure out the correct way since we need this to run today.

[Carsten Beyer]

Hi Matthew,

we use QOS for this and add it to the SLURM user, who needs to exceed
the partition time limit. You can also set a time limit in the QOS, so
that a user cannot exceed the 'limits' too much. Example from our system
with 8 hour runlimit per job:

# grep -i qos slurm.conf
PriorityWeightQOS=1000
AccountingStorageEnforce=limits,qos
#

# sacctmgr -s show qos ch0636 format=Name,Flags%32,MaxWall
      Name                            Flags     MaxWall
---------- -------------------------------- -----------
    ch0636   DenyOnLimit,PartitionTimeLimit    12:00:00
#
# scontrol show part compute
PartitionName=compute
   AllowGroups=ALL DenyAccounts=bmx825,mh1010 AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
   MaxNodes=512 MaxTime=08:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=m[10000-11420,11440-11545,11553-11577]

   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=EXCLUSIVE
   OverTimeLimit=NONE PreemptMode=OFF

   State=UP TotalCPUs=74496 TotalNodes=1552 SelectTypeParameters=NONE
   DefMemPerCPU=1280 MaxMemPerCPU=5360

With the flag PartitionTimeLimit a user can override the MaxTime of a
partition and with DenyOnLimit you can limit the maxwall of the job. The
QOS is added to the user like this:

# sacctmgr -s show user foo format=User,Account,MaxJobs,QOS%30
      User    Account MaxJobs QOS
---------- ---------- ------- ------------------------------
       foo     ch0636      20 ch0636,express,normal
       foo  noaccount       0 normal
       foo     mh0469      20 express,normal
#

The user also needs to add the QOS with '#SBATCH --qos=ch0636' in his
job description.

Cheers,
Carsten
--


Carsten Beyer
Abteilung Systeme

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45a * D-20146 Hamburg * Germany

Phone: +49 40 460094-221
Fax: +49 40 460094-270
Email: be...@dkrz.de
URL: http://www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784