[slurm-users] Include some cores of the head node to a partition

[Mahmood Naderan]

Hi,
Since our head node has 32 cores, I want to add some cores to a
partition. If I edit the parts file like this

PartitionName=SPEEDY AllowAccounts=em1 Nodes=compute-0-[2-4],rocks7

then it will include all cores. I think I have to edit slurm.conf like this then

NodeName=rocks7 NodeAddr=10.1.1.1 CPUS=20

is that confirmed?


Regards,
Mahmood

[Chris Samuel]

On Sunday, 22 April 2018 4:41:43 AM AEST Mahmood Naderan wrote:

> Since our head node has 32 cores, I want to add some cores to a
> partition. If I edit the parts file like this
>
> PartitionName=SPEEDY AllowAccounts=em1 Nodes=compute-0-[2-4],rocks7
>
> then it will include all cores.

All you need to do is add "MaxCPUsPerNode=20" to that to limit the number of
cores that the partition can use.

We do this for our non-GPU job partition to reserve some cores for the GPU job
partition.

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

[Mahmood Naderan]

I think that will limit other nodes to 20 too. Isn't that?

Currently computes have 32 cores per node and I want all 32 cores. The head node also has 32 core but I want to include only 20 cores.

[Benjamin Redling]

Hello Mahmood,

Am 22.04.2018 um 04:55 schrieb Mahmood Naderan:
> I think that will limit other nodes to 20 too. Isn't that?

you can declare less CPUs than phys. available. I do that for our
cluster; it is working robust for ages.

> Currently computes have 32 cores per node and I want all 32 cores. The
> head node also has 32 core but I want to include only 20 cores.

Agree with you. You don't need to restrict the whole
partition/cluster/etc. to only 20 cores when you only need that for a
single node.

> On Sun, Apr 22, 2018, 03:53 Chris Samuel <ch...@csamuel.org
> <mailto:ch...@csamuel.org>> wrote:
>
>
> All you need to do is add "MaxCPUsPerNode=20" to that to limit the
> number of
> cores that the partition can use.
>
> We do this for our non-GPU job partition to reserve some cores for
> the GPU job
> partition

That might be a valid solution for that cluster.
But it is a wasteful solution to Mahmood's.

Regards,
Benjamin
--
FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html
☎ +49 3641 9 44323

[Chris Samuel]

On Sunday, 22 April 2018 12:55:46 PM AEST Mahmood Naderan wrote:

> I think that will limit other nodes to 20 too. Isn't that?
>
> Currently computes have 32 cores per node and I want all 32 cores. The head
> node also has 32 core but I want to include only 20 cores.

Apologies, I misunderstood what you were trying to achieve. I assumed you
wanted a homogenous configuration for the partition. Yes, if you are happy
for the asymmetry then you can do that.

[Mahmood Naderan]

Chris,
So the problem still exists ;)

>Yes, if you are happy
>for the asymmetry then you can do that.

That is the question. MaxCPUsPerNode is for symmetrically set the max
core number for all nodes in the partition. That is not applicable for
asymmetric cases.
Regards,
Mahmood

[Mahmood Naderan]

Hi again
I see that the state of the frontend is Drained. Is that the default
state? The following line

PartitionName=OTHERS AllowAccounts=em1 Nodes=compute-0-[2-3],rocks7


Should include all core numbers of all nodes. The computes are set to
idle, but the frontend is drained.

Regards,
Mahmood

[Chris Samuel]

On Saturday, 28 April 2018 7:58:08 PM AEST Mahmood Naderan wrote:

> I see that the state of the frontend is Drained. Is that the default
> state?

Probably not. What does "sinfo --list-reasons" say?

[Mahmood Naderan]

[root@rocks7 ~]# sinfo --list-reasons
REASON USER TIMESTAMP NODELIST
Low socket*core*thre root 2018-04-19T16:46:39 rocks7


Regards,
Mahmood

[Chris Samuel]

On Sunday, 29 April 2018 2:34:09 AM AEST Mahmood Naderan wrote:

> [root@rocks7 ~]# sinfo --list-reasons
> REASON USER TIMESTAMP NODELIST
> Low socket*core*thre root 2018-04-19T16:46:39 rocks7

slurmd thinks that "rocks7" doesn't have enough hardware resources to meet
what you've told it.

What does "slurmd -C" say on rocks7 ?

[Mahmood Naderan]

[root@rocks7 ~]# slurmd -C
NodeName=rocks7 slurmd: Considering each NUMA node as a socket
CPUs=32 Boards=1 SocketsPerBoard=4 CoresPerSocket=8 ThreadsPerCore=1
RealMemory=64261
UpTime=15-21:30:53
[root@rocks7 ~]# scontrol show node rocks7
NodeName=rocks7 Arch=x86_64 CoresPerSocket=1
CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=0.03
AvailableFeatures=(null)
ActiveFeatures=(null)
Gres=(null)
NodeAddr=10.1.1.1 NodeHostName=rocks7 Version=17.11
OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017
RealMemory=64261 AllocMem=0 FreeMem=533 Sockets=1 Boards=1
State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=281775 Weight=1 Owner=N/A
MCS_label=N/A
Partitions=WHEEL,EMERALD
BootTime=2018-04-13T13:05:00 SlurmdStartTime=2018-04-13T13:05:17
CfgTRES=cpu=1,mem=64261M,billing=1
AllocTRES=
CapWatts=n/a
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
Reason=Low socket*core*thread count, Low CPUs [root@2018-04-19T16:46:39]


So, I don't know why only 1 core included

Regards,
Mahmood

[Chris Samuel]

On Sunday, 29 April 2018 4:11:39 PM AEST Mahmood Naderan wrote:

> So, I don't know why only 1 core included

What do you have in your slurm.conf for rocks7?

[Mahmood Naderan]

SlurmUser=root
SlurmdUser=root
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
CryptoType=crypto/munge
StateSaveLocation=/var/spool/slurm.state
SlurmdSpoolDir=/var/spool/slurmd
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/linuxproc
PluginDir=/usr/lib64/slurm
CacheGroups=0
JobCheckpointDir=/var/spool/slurm.checkpoint
#SallocDefaultCommand = "xterm"
GresTypes=gpu
#FirstJobId=
ReturnToService=2
#MaxJobCount=
#PlugStackConfig=
#PropagatePrioProcess=
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#Prolog=
#Epilog=
#SrunProlog=
#SrunEpilog=
#TaskProlog=
#TaskEpilog=
TaskPlugin=task/affinity
TrackWCKey=yes
TopologyPlugin=topology/none
#TreeWidth=50
TmpFs=/state/partition1
#UsePAM=
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=30
MinJobAge=300
KillWait=60
WaitTime=60
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory
#DefMemPerCPU=220
#MaxMemPerCPU=300
VSizeFactor=0
FastSchedule=0

JobCompType=jobcomp/none
JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30

####### Priority Begin ##################
PriorityType=priority/multifactor
PriorityDecayHalfLife=14-0
PriorityWeightFairshare=10000
PriorityWeightAge=10
PriorityWeightPartition=10000
PriorityWeightJobSize=100
PriorityMaxAge=1-0
PriorityWeightQOS=0
PriorityFlags=FAIR_TREE
PriorityWeightTRES=cpu=2000,mem=1,gres/gpu=400
AccountingStorageTRES=gres/gpu
AccountingStorageEnforce=all
FairShareDampeningFactor=5
####### Priority End ##################

SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm/slurmd.log
DebugFlags=Priority,NO_CONF_HASH,backfill,BackfillMap

NodeName=DEFAULT State=UNKNOWN
NodeName=rocks7 NodeAddr=10.1.1.1
PartitionName=DEFAULT AllocNodes=rocks7 State=UP
PartitionName=DEBUG

####### Power Save Begin ##################
# SuspendExcNodes=<your headnode>,compute-1-0
SuspendProgram=/etc/slurm/suspendhost.sh
SuspendRate=4
# SuspendTime = 600
SuspendTime=-1
SuspendTimeout=45
ResumeProgram=/etc/slurm/resumehost.sh
ResumeRate=4
ResumeTimeout= 450
####### Power Save End ##################

################ Do not edit below
#############################################################
include /etc/slurm/head.conf
include /etc/slurm/node.conf
include /etc/slurm/parts.conf
################################################################################################



Regards,
Mahmood

[Chris Samuel]

Hi Mahmood,

Not quite what I meant sorry.

What does this say?

scontrol show config | fgrep -i rocks7

cheers,
Chris

[Mahmood Naderan]

[root@rocks7 ~]# scontrol show config | fgrep -i rocks7
AccountingStorageHost = rocks7
ControlMachine = rocks7
JobCompHost = rocks7
Slurmctld(primary/backup) at rocks7/(NULL) are UP/DOWN



Chris,
Regarding this section

NodeName=DEFAULT State=UNKNOWN
NodeName=rocks7 NodeAddr=10.1.1.1
PartitionName=DEFAULT AllocNodes=rocks7 State=UP
PartitionName=DEBUG

Should I add something like

PartitionName=OTHERS AllocNodes=rocks7 State=UP MaxCPUsPerNode=20

or even 32?

Regards,
Mahmood

[Chris Samuel]

On Sunday, 29 April 2018 6:57:58 PM AEST Mahmood Naderan wrote:

> [root@rocks7 ~]# scontrol show config | fgrep -i rocks7

Ah, I'd forgotten that wouldn't list the NodeName lines from your config file.
Sorry.

> Chris,
> Regarding this section
>
> NodeName=DEFAULT State=UNKNOWN
> NodeName=rocks7 NodeAddr=10.1.1.1

OK, so there you are not telling Slurm anything about your hardware. As you
just want 20 cores for rocks7 then you probably want to add this to your
NodeName line:

CPUs=20 Boards=1 SocketsPerBoard=4 CoresPerSocket=5 ThreadsPerCore=1

I've never tried to do this though so it might not be optimal.

> Should I add something like
>
> PartitionName=OTHERS AllocNodes=rocks7 State=UP MaxCPUsPerNode=20
>
> or even 32?

No, it looks like Slurm hasn't got the right config info in it for the node.

Try updating your config and restarting slurmctld and slurmd everywhere and see
where that gets you. If it's still drained but the hardware config looks good
in Slurm then you can do "scontrol update node=rocks7 state=resume" to tell
Slurm to try using it again.

All the best,