[slurm-users] --partition requests ignored in scripts

[Bunis, Dan]

Hi there,

My colleagues and I have noticed that our compute cluster seems to ignore '--partition' requests when we give them as '#SBATCH --partition=<partition_name>' inside of our scripts, but it respects them when given in-line within our sbatch calls as 'sbatch --partition=<partition_name> script.sh'.  Based on some googling, it seems that both methods are meant to work, so I'm wondering if it's known what can cause the in-script methodology to NOT work for schedulers where the in-line methodology DOES work?

Thanks in advance!

Dan

_______________________________

Daniel Bunis, PhD

He/Him/His

Sr. Biological Data Scientist in Immunology

Data Science CoLab

University of California, San Francisco

daniel...@ucsf.edu

[Bjørn-Helge Mevik]

"Bunis, Dan" <Daniel...@ucsf.edu> writes:

> My colleagues and I have noticed that our compute cluster seems to
> ignore '--partition' requests when we give them as '#SBATCH
> --partition=<partition_name>' inside of our scripts, but it respects
> them when given in-line within our sbatch calls as 'sbatch
> --partition=<partition_name> script.sh'. Based on some googling, it
> seems that both methods are meant to work, so I'm wondering if it's
> known what can cause the in-script methodology to NOT work for
> schedulers where the in-line methodology DOES work?

My suspicion is that there is an environment variable SBATCH_PARTITION
set in your shells. Such a variable will override the #SBATCH directive,
but not the command line switch.

From man sbatch:

INPUT ENVIRONMENT VARIABLES
[...]
NOTE: Environment variables will override any options set in a batch
script, and command line options will override any environment
variables.

--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

[Michael Gutteridge]

The position of the #SBATCH directives also matters (emphasis mine):

> The batch script may contain options preceded with "#SBATCH" before any executable commands in the script.

We've been bit by that a couple times- a stray command before any #SBATCH lines will cause any of those directives to be ignored.

 - Michael

[Bunis, Dan]

Thanks so much for this answer!  Turned out that default partitions get set on this HPC via an environment variable, so armed with the knowledge you shared, I've been able to figure out a viable path for my use-case by making use of 'unset SBATCH_PARTITION' =)

-Dan

---

From: slurm-users on behalf of Bjørn-Helge Mevik
Sent: Thursday, November 9, 2023 2:38 AM
To: slurm...@schedmd.com
Subject: Re: [slurm-users] --partition requests ignored in scripts