[slurm-users] Background tasks in Slurm scripts?

[Steffen Grunewald via slurm-users]

Good morning,

yesterday I came across a Slurm (sbatch) script that, after doing some stuff
in the foreground, runs another executable in the background - and doesn't
"wait" for it to finish - literally the last line of the script is

executable &

(and that executable is supposed to take several 10 seconds or more to finish)

How would Slurm handle this? Will the end of the script immediately trigger
the job epilog, and what would happen to the leftover task? This certainly
is discussed somewhere in the manual pages and other documentation but up to
now I failed to find that place...

Thanks,
Steffen


--
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[mercan via slurm-users]

Good Morning;

This is not a slurm issue. This is a default shell script feature. If
you want to wait to finish until all background processes, you should
use wait command after all.

Regards;

C. Ahmet Mercan

On 26.07.2024 10:23, Steffen Grunewald via slurm-users wrote:
> Good morning,
>
> yesterday I came across a Slurm (sbatch) script that, after doing some stuff
> in the foreground, runs another executable in the background - and doesn't
> "wait" for it to finish - literally the last line of the script is
>
> executable &
>
> (and that executable is supposed to take several 10 seconds or more to finish)
>
> How would Slurm handle this? Will the end of the script immediately trigger
> the job epilog, and what would happen to the leftover task? This certainly
> is discussed somewhere in the manual pages and other documentation but up to
> now I failed to find that place...
>
> Thanks,
> Steffen
>
>

--

[Steffen Grunewald via slurm-users]

On Fri, 2024-07-26 at 10:42:45 +0300, Slurm users wrote:
> Good Morning;
>
> This is not a slurm issue. This is a default shell script feature. If you
> want to wait to finish until all background processes, you should use wait
> command after all.

Thank you - I already knew this in principle, and I also know that a login
shell will complain at an attempt to exit when there are leftover background
jobs. I was wondering though how Slurm's task control would react... Got to
try myself, I guess...

Best, S

[Brian Andrus via slurm-users]

Generally speaking, when the batch script exits, slurm will clean up (ie
kill) any stray processes.
So, I would expect that executable to be killed at cleanup.

Brian Andrus