[slurm-dev] Question on process limits for jobs under SLURM

2 views

Skip to first unread message

Blosch, Edwin L

unread,

Aug 11, 2015, 2:36:01 PM8/11/15

to slurm-dev

With PBS Pro, it was customary and necessary to modify /etc/init.d/pbs_mom to add ulimit commands shown in (a) below, but I haven’t seen a need for that with SLURM yet. Recently there are some users at my organization who are focusing on process-limit issues as potential sources of application failures.

I was wondering if anyone has found need to be concerned with change (a) below, if change (b) has already been taken care of? In the user reports, I think they are using OpenMPI 1.6.5 and did not include the –without-slurm configuration, so I’m not certain of the launching mechanism for the children processes, but I think it may leverage SLURM as opposed to using ssh.

(a) adding the following lines to /etc/init.d/slurm

ulimit -l unlimited

ulimit -s unlimited

(b) adding these lines to /etc/security/limits.conf

* soft memlock unlimited

* hard memlock unlimited

* soft stack unlimited

* hard stack unlimited

* soft nofile 1000000

* hard nofile 1000000

Thanks for your help,

Trey Dockendorf

unread,

Aug 11, 2015, 4:06:55 PM8/11/15

to slurm-dev

I've found changes to limits.conf don't get applied to the slurm daemons when started via init.d scripts as the limits.conf is only used for PAM sessions. I ran into such an issue when debugging why VASP was failing [1].

We currently set all limits that need to be applied to jobs in /etc/sysconfig/slurm [2] which is sourced by /etc/init.d/slurm, at least is the case for RPM installs of SLURM.

- Trey

[1]: https://groups.google.com/d/msg/slurm-devel/Ibm2-fP-wUA/H34eKKLDzN0J

[2]:

ulimit -l unlimited

ulimit -n 8192

ulimit -s unlimited

CONFDIR="/etc/slurm"

SLURMCTLD_OPTIONS="-f /etc/slurm/slurm.conf"

SLURMD_OPTIONS="-f /etc/slurm/slurm.conf -M"

export SLURM_CONF="/etc/slurm/slurm.conf"

=============================

Trey Dockendorf