[slurm-users] Execute parallel commands on all nodes running jobs of a particular user

[Ole Holm Nielsen]

Hi Slurm users,

We have found the need to execute a parallel command on all nodes
running jobs belonging to a particular user.

I have made a configuration to the excellent ClusterShell tool as
documented in https://wiki.fysik.dtu.dk/niflheim/SLURM#clustershell

If you add a "slurmuser" section to the
/etc/clustershell/groups.conf.d/slurm.conf file, you can now run
commands such as:

$ clush -bw@su:username 'df -Ph /scratch'

$ clush -bw@su:username 'du -s /scratch/username'

I hope that others may find this feature useful.

/Ole

[Bjørn-Helge Mevik]

There is also a Slurm plugin for pdsh (unfortunately not enabled in the
default redhat/centos RPMs) that lets you run a command on each node
belonging to a specific job with "pdsh -j <jobid> <command>". Not
exactly the same, though. :)

--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

[Ole Holm Nielsen]

On 06-08-2018 12:53, Bjørn-Helge Mevik wrote:
> There is also a Slurm plugin for pdsh (unfortunately not enabled in the
> default redhat/centos RPMs) that lets you run a command on each node
> belonging to a specific job with "pdsh -j <jobid> <command>". Not
> exactly the same, though. :)

Bjørn, that is a different task. I've documented pdsh usage with Slurm
in my Wiki page
https://wiki.fysik.dtu.dk/niflheim/SLURM#pdsh-parallel-distributed-shell.
However, I find it easier to work with ClusterShell, see
https://wiki.fysik.dtu.dk/niflheim/SLURM#clustershell.

The functionality I proposed on this list is to run a command on *all*
nodes belonging to *all* jobs of a particular user:

> If you add a "slurmuser" section to the /etc/clustershell/groups.conf.d/slurm.conf file, you can now run commands such as:
>
> $ clush -bw@su:username 'df -Ph /scratch'
>
> $ clush -bw@su:username 'du -s /scratch/username'

This functionality will be available in the next release 1.8.1 of
ClusterShell.

/Ole

[Bjørn-Helge Mevik]

Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> writes:

> Bjørn, that is a different task.

I know, but related. Just meant as a tip for people who already use
pdsh.