[slurm-dev] default behavior of salloc drop into shell

[Jordan Willis]

Hi,

I have been adding more partitions to my cluster. I have two submission nodes, the primary controller and the backup controller. Before I added the new partitions, the behavior of salloc was to drop me into one of the nodes defined in the partition. I have no idea what I did, but the new behavior is to keep me on the submission node with the allocated resources now in my environment. After doing some searching, I found this thread - http://comments.gmane.org/gmane.comp.distributed.slurm.devel/5941

In it, Chrysovalantis Paschoulas states that "The default salloc (you can change it) returns you a shell on the node where the submission took place and then with srun commands you can execute programs on the compute nodes.”

I was wondering how I actually change the default behavior to drop into one of the allocated nodes. I’m sure it has something to with SallocDefaultCommand, but so far I can’t figure it out. Additionally, the documentation says various options may have influenced the way nodes are allocated (https://computing.llnl.gov/linux/slurm/cpu_management.html). Here is my slurm.conf:

# Put this file on all nodes of your cluster.

# See the slurm.conf man page for more information.

#SlurmctldPort=6817

#SlurmdPort=6818

# TIMERS

#KillWait=30

#MinJobAge=300

#SlurmctldTimeout=120

#SlurmdTimeout=300

#SchedulerPort=7321

#The time it takes for a completed job to disappear of the queue

CompleteWait=60

#The name of the user the controller executes as, can either be root or slurm user

SlurmdUser=root

#The machine name we use as the controller

ControlMachine=cot1

#Can also execute jobs on another machine using a backup controller

BackupController=cot2

#let srun figure out the mpi type

MpiDefault=none

#Track jobs by process group id

ProctrackType=proctrack/pgid

#A down node will automatically return to service except during a failure

ReturnToService=1

#Authenticate nodes with munge

AuthType=auth/munge

####SLURM FILE LOCATIONS####

SlurmctldPidFile=/var/run/slurmctl.pid

SlurmdPidFile=/var/run/slurmd.pid

SlurmdSpoolDir=/var/spool/slurmd

StateSaveLocation=/var/spool/slurm

SlurmctldLogFile=/var/log/slurmctl/controller.log

#Do we need special switch i.e elan (we use ehternet so no)

SwitchType=switch/none

#Type of task plugin - don't have special handling of cpu_bind or mem_bind jobs

TaskPlugin=task/none

#consdider the node definitions, if it is less, make the node go down

FastSchedule=1

#initiate lower priority jobs if they don't consume too much

#augments FIFO, first in first out

SchedulerType=sched/backfill

ClusterName=TSRI

#each cpu is considered a resource

SelectType=select/linear

### LOGGING AND ACCOUNTING

#Basic Logging to text file

AccountingStorageType=accounting_storage/filetxt

#and where should that logging go

AccountingStorageLoc=mypath/logging/account_text.txt

#use linux rules to gather, this doesn't do anything yet

#JobAcctGatherType=jobacct_gather/linux

#### COMPUTE NODES ####

NodeName=imperial-node[01-10] CPUs=24 RealMemory=64 Sockets=2 CoresPerSocket=12 ThreadsPerCore=1

NodeName=silver-node[01-28] CPUs=32 RealMemory=128 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2

#Note that ocean-node01 is different from the rest of the Oceans...dan..

NodeName=ocean-node01 CPUs=16 Procs=16 RealMemory=256 Sockets=2 CoresPerSocket=8 ThreadsPerCore=1

NodeName=ocean-node[02-05] CPUs=32 RealMemory=256 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2

NodeName=loma-node[01-10] CPUs=24 RealMemory=64 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2

### Partitions ####

PartitionName=all Nodes=imperial-node[01-10],silver-node[01-28],ocean-node[01-05],loma-node[01-10] State=UP Default=YES 

PartitionName=imperial Nodes=imperial-node[01-10] Default=NO MaxTime=INFINITE State=UP 

PartitionName=silver Nodes=silver-node[01-28] Default=NO MaxTime=INFINITE State=UP 

PartitionName=ocean Nodes=ocean-node[01-05] Default=NO MaxTime=INFINITE State=UP 

PartitionName=loma Nodes=loma-node[01-10] Default=NO MaxTime=INFINITE State=UP 

[Chrysovalantis Paschoulas]

Dear Jordan Willis,

as you already said you can change the default behaviour of salloc by setting the SallocDefaultCommand option in slurm.conf.

From the man pages, "man slurm.conf":
{{{
...
Normally, salloc(1) will run the user's default shell when
a command to execute is not specified on the salloc command line.
If SallocDefaultCommand is specified, salloc will instead
run the configured command. The command is passed to '/bin/sh \-c', so
shell metacharacters are allowed, and commands with multiple arguments
should be quoted. For instance:

    SallocDefaultCommand = "$SHELL"

would run the shell in the user's $SHELL environment variable. And

    SallocDefaultCommand = "srun -n1 -N1 --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"

would run spawn the user's default shell on the allocated resources, but not
consume any of the CPU or memory resources, configure it as a pseudo\-terminal,
and preserve all of the job's environment variables (i.e. and not over\-write
them with the job step's allocation information).
...
}}}

The default behaviour of salloc is similar to the first example above. And what you want is possible with the second example.

So, I would suggest you to set something like the following (of course you can play with the options, but the important is to have the --pty option of srun and a shell as the executable, $SHELL variable has the current shell during submission):

  SallocDefaultCommand="srun -n1 -N1 --cpu_bind=none --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"

With this option the salloc command will allocate the requested resources and give you a prompt on the first allocated compute node.

Best Regards,
Chrysovalantis Paschoulas

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------