[slurm-users] How do you make --export=NONE the default behavior for our cluster?

[Ransom, Geoffrey M.]

 

Hello

    We recently added new architectures to our compute and submit nodes and a PATH gets generated based on the type of machine our users log into. Unfortunately, this PATH is architecture dependent but it is getting copied with slurm jobs to the compute nodes which could be a different architecture as the submit node. Right now, running srun inside the sbatch script will fail with linking errors if you submit from a RHEL 8 box and land on a RHEL 7 box since it tries to run the RHEL 8 compiled version due to the PATH environment variable.

 

Adding “--export=NONE” to the job avoids the problem, but I’m not seeing a way to change this default behavior for the whole cluster.

 

How do you change/set default options for the cluster for all users so “--export=NONE” is the default behavior?

 

Is this change likely to break anything unexpected or cause problems?

Our users have NFS mounted home directories so if we removed the ssh non-admin user block they would get a similar/functional environment as they do on the submit nodes.

 

Thanks.

[Ward Poelmans]

Hi,

We're using a cli filter for doing this. But it's more tricky then just `--export=NONE`. For a srun inside a sbatch, you want `--export=ALL` again because MPI will break otherwise.

We have this in our cli filter:

function slurm_cli_pre_submit(options, pack_offset)

local default_env = "HOME,USER,TERM,PATH=/bin:/sbin"



if options["type"] == "sbatch" or (options["type"] == "srun" and options["pty"] == "set") then

-- for sbatch and interactive srun, set our minimal environment

if options["export"] == nil then

options["export"] = default_env

slurm.log_debug("Setting export to %s", tostring(options["export"]))

end

elseif options["type"] == "srun" then

-- in case of srun, and we're in our minimal environment, pass again everything (for mpi)

if options["export"] == default_env then

options["export"] = "ALL"

slurm.log_debug("Setting export to %s", tostring(options["export"]))

end

end

end

So we set `--export=HOME,USER,TERM,PATH=/bin:/sbin` if the user has not specified anything else. For a srun inside a sbatch, we switch again to `--export=ALL`.

Ward

[Christopher Samuel]

On 6/3/22 11:39 am, Ransom, Geoffrey M. wrote:

> Adding “--export=NONE” to the job avoids the problem, but I’m not seeing
> a way to change this default behavior for the whole cluster.

There's an SBATCH_EXPORT environment variable that you could set for
users to force that (at $JOB-1 we used to do that).

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA