[slurm-users] Question about sbatch options: -n, and --cpus-per-task

[David Henkemeyer]

Assuming -N is 1 (meaning, this job needs only one node), then is there a difference between any of these 3 flag combinations:

-n 64 (leaving cpus-per-task to be the default of 1)

--cpus-per-task  64 (leaving -n to be the default of 1)

--cpus-per-task 32 -n 2

As far as I can tell, there is no functional difference. But if there is even a subtle difference, I would love to know what it is!

Thanks

David 

--

Sent from Gmail Mobile

[Ralph Castain]

Well, there is indeed a difference - and it is significant.

> On Mar 24, 2022, at 12:32 PM, David Henkemeyer <david.he...@gmail.com> wrote:
>
> Assuming -N is 1 (meaning, this job needs only one node), then is there a difference between any of these 3 flag combinations:
>
> -n 64 (leaving cpus-per-task to be the default of 1)

Will launch 64 instances of your application, each bound to a single cpu

> --cpus-per-task 64 (leaving -n to be the default of 1)

Will run ONE instance of your application (no binding if the node has 64 cpus - otherwise, the proc will be bound to 64 cpu's)

> --cpus-per-task 32 -n 2

Will run TWO instances of your application, each bound to 32 cpu's

[David Henkemeyer]

“ Will launch 64 instances of your application, each bound to a single cpu”

This is true for srun, but not for sbatch. 

A while back, we did an experiment using “hostname” to verify. 

[Thomas M. Payerle]

Although all three cases ( "-N 1 --cpus-per-task 64 -n 1", "-N 1 --cpus-per-task 1 -n 64", and "-N 1 --cpus-per-task 32 -n 2") will cause Slurm to allocate 64 cores to the job, there can (and will) be differences in the other respects.

The variable SLURM_NTASKS will be set to the argument of the -n (aka --ntasks) argument, and other Slurm variables will differ as well.

More importantly, as others noted, srun will launch $SLURM_NTASKS processes.  The mpirun/mpiexec/etc binaries of most MPI libraries will (if compiled with support for Slurm) act similarly (and indeed, I believe most use srun under the hood).

If you are just using sbatch and launching a single process using 64 threads, then the different options are probably equivalent for most intent and purposes.  Similar if you are doing a loop to start 64 single threaded processes.  But those are simplistic cases, and just happen to "work" even though you are "abusing" the scheduler options.  And even the cases wherein it "works" is subject to unexpected failures (e.g. if one substitutes srun for sbatch). 

The differences are most clear when the -N 1 flag is not given.  Generally, SLURM_NTASKS should be the number of MPI or similar tasks you intend to start.  By default, it is assumed the tasks can support distributed memory parallelism, so the scheduler by default assumes that it can launch tasks on different nodes (the -N 1 flag you mentioned would override that).  Each such task is assumed to need --cpus-per-task cores which the scheduler assumes needs shared memory parallelism (i.e. must be on the same node).

So without the -N 1, "--cpus-per-task 64 -n 1" will require 64 cores on a single node, whereas "-n 64 --cpus-per-task 1" can result in the job being assigned 64 cores on a single node to a single core on 64 nodes or any combination in between with 64 cores.  The "--cpus-per-task 32 -n 2" will either assign one node with 64 cores or 2 nodes with 32 cores each.

As I said, although there are some simple cases where the different cases are mostly functionally equivalent, I would recommend trying to use the proper arguments --- "abusing" the arguments might work for a while but will likely bite you in the end.  E.g., the 64 thread case should do "--cpus-per-task 64", and the launching processes in the loop should _probably_ do "-n 64" (assuming it can handle the tasks being assigned to different nodes).

--

Tom Payerle
DIT-ACIGS/Mid-Atlantic Crossroads        pay...@umd.edu

5825 University Research Park               (301) 405-6135

University of Maryland
College Park, MD 20740-3831

[David Henkemeyer]

Thank you!  We recently converted from pbs, and I was converting “ppn=X” to “-n X”.  Does it make more sense to convert “ppn=X” to --“cpus-per-task=X”?

Thanks again

David

[Stephen Cousins]

If you want to have the same number of processes per node, like:

#PBS -l nodes=4:ppn=8

then what I am doing (maybe there is another way?) is:

#SBATCH --ntasks-per-node=8           
#SBATCH --nodes=4                     
#SBATCH --mincpus=8                   

This is because "--ntasks-per-node" is actually "maximum number of tasks per node" and "--nodes=4" means "minimum number of nodes". I'm sure other variations (specifying --ntasks=32, --mincpus=8 and --nodes=4-4 might do it too) but this one is what I've been using. I remember being surprised when coming over from Torque to find that "--ntasks-per-node" and --nodes did not mean what they so obviously seemed to mean.

Steve