[slurm-users] srun/sbatch dependency not working

[Darin Gowan]

Dear distinguished list,

I am new to SLURM.  I have recently installed SLURM 20.11.3 on two separate three node clusters.  The first cluster was for testing purposes using three small RHEL 7.7 VMs (8 core, 8G RAM).  After a successful installation and some sbatch testing, I proceeded to the second cluster.

The production cluster is running on three RHEL 7.7 physical servers, two sockets, 24 cores each, 2 threads per core and 1TB RAM.  This installation was also successful.

Yesterday, a user brought an issue to my attention.  They reported that when submitting a job via srun using the dependency option (-d afterany:aaaa:bbbb:cccc:dddd...), the dependency was not being honored.

I began by testing the srun -d option in my test cluster, which worked like a charm.  The srun job went into the Pending state, waiting for resources.  Once jobid 1296 completed, the srun executed.  

$ srun /wks01/data/slurm-jobs/clean-up.sh -d afterany:1296

srun: job 1301 queued and waiting for resources

$ squeue

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

              1301     slurm clean-up zzgowand PD       0:00      1 (Resources)

              1291     slurm hostname zzgowand  R       0:56      1 r7slurm01

              1292     slurm hostname zzgowand  R       0:56      1 r7slurm01

              1293     slurm hostname zzgowand  R       0:56      1 r7slurm01

              1294     slurm hostname zzgowand  R       0:55      1 r7slurm01

              1295     slurm hostname zzgowand  R       0:55      1 r7slurm02

              1296     slurm hostname zzgowand  R       0:53      1 r7slurm02

 

However, when I ran the exact same test on the production cluster, it was like the '-d' option wasn't supplied.  The job went into the Running state, but never really executed.  It sat in this state for several minutes, when it should have run in a few seconds.  I finally ended up aborting the foreground execution of srun.  This resulted in the following messages:

srun launch/slurm: launch_p_step_launch: ... aborted before step completely launched.

Has anyone experienced this before?

Thank you.

Darin Gowan