[slurm-users] Grid engine slaughtering parallel jobs when any one of them fails (copy)

[Robert Peck]

Excuse me, I am trying to run some software on a cluster which uses the SLURM grid engine. IT support at my institution have exhausted their knowledge of SLURM in trying to debug this rather nasty bug with a specific feature of the grid engine and suggested I try here for tips.

I am using jobs of the form:

#!/bin/bash
#SBATCH --job-name=name       # Job name
#SBATCH --mail-type=END,FAIL             # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=my_e...@thing.thing     # Where to send mail  

#SBATCH --mem=2gb                        # Job memory request, not hugely intensive
#SBATCH --time=47:00:00                  # Time limit hrs:min:sec, the sim software being run from within the bash script is quite slow, extra memory can't speed it up and it can't run multi-core, hence long runs on weak nodes

#SBATCH --nodes=100
#SBATCH --ntasks=100
#SBATCH --cpus-per-task=1

#SBATCH --output=file_output_%j.log        # Standard output and error log
#SBATCH --account=code       # Project account
#SBATCH --ntasks-per-core=1 #only 1 task per core, must not be more
#SBATCH --ntasks-per-node=1 #only 1 task per node, must not be more
#SBATCH --ntasks-per-socket=1 #guessing here but fairly sure I don't want multiple instances trying to use same socket

#SBATCH --no-kill # supposedly prevents restart of other jobs on other nodes if one of the 100 gets a NODE_FAIL


echo My working directory is `pwd`
echo Running job on host:
echo -e '\t'`hostname` at `date`
echo
 
module load toolchain/foss/2018b
cd scratch
cd further_folder
chmod +x my_bash_script.sh

srun --no-kill -N "${SLURM_JOB_NUM_NODES}" -n "${SLURM_NTASKS}" ./my_bash_script.sh

wait
echo
echo Job completed at `date`

I use a bash script to launch my special software and stuff which actually handles each job, this software is a bit weird and two copies of it WILL NOT EVER play nicely if made to share a node. Hence this job acts to launch 100 copies on 100 nodes, each of which does its own stuff and writes out to a separate results file. I later proces the results files. 

In my scenario I want 100 jobs to run, but if one or two failed and I only got 99 or 95 back then I could work fine for further processing with just 99 or 95 result files. Getting back a few less jobs then I want is no tragedy for my type of work.

But the problem is that when any one node has a failure, not that rare when you're calling for 100 nodes simultaneously, SLURM would by default murder the WHOLE LOT of jobs, and even more confusingly then restart a bunch of them which ends up with a very confusing pile of results files. I thought the --no-kill flag should prevent this fault, but instead of preventing the killing of all jobs due to a single failure it only prevents the restart, now I get a misleading message from the cluster telling me of a good exit code when such slaughter occurs, but when I log in to the cluster I discover a grid engine massacre of my jobs, all because just one of them failed. 

I understand that for interacting jobs on many nodes then killing all of them because of one failure can be necessary, but my jobs are strictly parallel, no cross-interaction between them at all. each is an utterly separate simulation with different starting parameters. I need to ensure that if one job fails and must be killed then the rest are not affected.

I have been advised that due to the simulation software being such as to refuse to run >1 copy properly on any given node at once I am NOT able to use "array jobs" and must stick to this sort of job which requests 100 nodes this way.

Please can anyone suggest how to instruct SLURM not to massacre ALL my jobs because ONE (or a few) node(s) fails?

All my research is being put on hold by this bug which is making getting large runs out of the cluster almost impossible, a very large fraction of the jobs I submit has a failure on 1 of the 100 nodes and hence that very large fraction of my jobs get killed on all nodes even though only one is faulty. I don't get jobs lasting to give me many useful sets of the 100(ish) result files I need.

P.S. just to warn you, I'm not an HPC expert or a linux power user. I'm comfortable with linux and with command lines and technical details but will probably need a bit more explanation around answers than someone specialised in high performance computing would.

--

Thank You

Rob

P.S. unsure of whether this is how one is supposed to add forum posts to this google group, sent twice as I wasn't sure if the earlier one got through as I might not have been correctly subscribed at that time, thanks

[Renfro, Michael]

I can't speak to what happens on node failure, but I can at least get you a greatly simplified pair of scripts that will run only one copy on each node allocated:

 

 

#!/bin/bash

# notarray.sh

#SBATCH --nodes=28

#SBATCH --ntasks-per-node=1

#SBATCH --no-kill

echo "notarray.sh is running on $(hostname)"

srun --no-kill somescript.sh

 

 

and

 

 

#!/bin/bash

# somescript.sh

echo "somescript.sh is running on $(hostname)"

 

 

I can verify that after submitting the job with "sbatch notarray.sh":

• notarray.sh ran on only one allocated node, and
• somescript.sh ran once on each of the 28 nodes allocated, including the one that notarray.sh ran on.

 

No need to pass srun a set of parameters for how many tasks to run, since it can figure that out from the sbatch context.

 

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Robert Peck <rp1...@york.ac.uk>
Date: Friday, April 16, 2021 at 2:40 PM
To: slurm...@schedmd.com <slurm...@schedmd.com>
Subject: [slurm-users] Grid engine slaughtering parallel jobs when any one of them fails (copy)

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

---

[Robert Peck]

Michael, thanks for the tip. I can give that a go but don't know if it will solve my issue.

Jess, sorry I have no knowledge of how the university handles the cluster system in that sense.

Has anyone else been reporting bugs with the --no-kill flag recently on your forum?

--no-kill isn't all that heavily documented from online searching, are there any more specific parameters which can be used to create the desired behaviour of "if one node dies just ignore it and let the others keep going".

Would posting some cluster logs here help? Although I have found that the verbosity level doesn't seem to be able to be set higher than the default?

I did mention that I'd heard array jobs aren't ok due to my "separate job on separate node" requirement, but wondered if anyone could advise me of ways to build that need in to an array job setup? If that would avoid the problem I'm having where all jobs get slaughtered when just one of them fails?

Thank you

On Sat, 17 Apr 2021 at 02:16, Jess Arrington <je...@schedmd.com> wrote:

Hi Robert,

I hope your day is treating you well.


Thank you for your posts on the Slurm user list.  


Would there be interest on your side to see a Slurm support contract for your systems at University of York?

Sites running Slurm with support give us feedback that support is invaluable and a great return back to the organization with much better system utilization with optimized configs by our experts (which pays for the support contract in and of itself), guaranteed resolutions to their issues and their sites not having to rely on in-house best-effort support hacks that get very expensive and turn into complicated chaos and potential down systems.


Additionally, support keeps the Slurm project alive and going strong  


Please let me know your thoughts or if you would like me to reach out to another contact at University of York to chat about this further.

Take care,

Jess Arrington Executive Director:Global Sales & Alliances je...@schedmd.com |  801-616-7823 240 N 1200 E #203 Lehi, UT 84043

[Loris Bennett]

Hi, Robert,

Robert Peck <rp1...@york.ac.uk> writes:

> Michael, thanks for the tip. I can give that a go but don't know if it will solve my issue.
>
> Jess, sorry I have no knowledge of how the university handles the cluster system in that sense.
>
> Has anyone else been reporting bugs with the --no-kill flag recently on your forum?
>
> --no-kill isn't all that heavily documented from online searching, are there any more specific parameters which can be used to create the desired behaviour of "if one node dies just ignore it and let the others keep going".
>
> Would posting some cluster logs here help? Although I have found that the verbosity level doesn't seem to be able to be set higher than the default?
>
> I did mention that I'd heard array jobs aren't ok due to my "separate job on separate node" requirement, but wondered if anyone could advise me of ways to build that need in to an array job setup? If that would avoid the problem I'm having
> where all jobs get slaughtered when just one of them fails?

I think that, rather than trying to bend Slurm to do what you think you
want it to do, you should really rethink your "separate job on separate
node" requirement. It is just not normal behaviour for jobs to be
failing in the way you describe. You may want to look at issues like
how many processes a single job is actually starting compared with the
number of cores you are requesting, or what sort of files the program
generates. In the latter case, if a file name is not sufficiently
unique, files written to the same local file system by different jobs on
the same node may clobber each other.

Maybe people on the list could help you if you explained what your program is and a
bit about how it works.

Cheers,

Loris

PS: It is slightly confusing to use the phrase "grid engine" in
connection with Slurm, as Oracle's (previously Sun's) Grid Engine is a
different piece of software, albeit one with a similar purpose, namely
that of a resource manager.

> Thank you
>
> On Sat, 17 Apr 2021 at 02:16, Jess Arrington <je...@schedmd.com> wrote:
>
> Hi Robert,
>
> I hope your day is treating you well.
>
> Thank you for your posts on the Slurm user list.
>
> Would there be interest on your side to see a Slurm support contract for your systems at University of York?
>
> Sites running Slurm with support give us feedback that support is invaluable and a great return back to the organization with much better system utilization with optimized configs by our experts (which pays for the support contract in and
> of itself), guaranteed resolutions to their issues and their sites not having to rely on in-house best-effort support hacks that get very expensive and turn into complicated chaos and potential down systems.
>
> Additionally, support keeps the Slurm project alive and going strong
>
> Please let me know your thoughts or if you would like me to reach out to another contact at University of York to chat about this further.
>
> Take care,
>

[Christopher Samuel]

Hi Robert,

On 4/16/21 12:39 pm, Robert Peck wrote:

> Please can anyone suggest how to instruct SLURM not to massacre ALL my
> jobs because ONE (or a few) node(s) fails?

You will also probably want this for your srun: --kill-on-bad-exit=0

What does the scontrol command below show?

scontrol show config | fgrep KillOnBadExit

From the manual page:

-K, --kill-on-bad-exit[=0|1]
Controls whether or not to terminate a step if any task
exits with a non-zero exit code. If this option is not
specified, the default action will be based upon
the Slurm configuration parameter of KillOnBadExit.
If this option is specified, it will take precedence over
KillOnBadExit. An option argument of zero will not
terminate the job. A non-zero argument or no argument
will terminate the job. Note: This option takes
precedence over the -W, --wait option to terminate the
job immediately if a task exits with a non-zero exit
code. Since this option's argument is optional, for
proper parsing the single letter option must be followed
immediately with the value and not include a space between
them. For example "-K1" and not "-K 1".


Best of luck,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA

[Robert Peck]

Chris: thanks for that tip, I'm having a look at that now, it sounds promising.

Run on the login node I get:

scontrol show config | fgrep KillOnBadExit

KillOnBadExit           = 0

I've tried to put  -K0 in to a job to see if that helps.
But doing it on the command line
sbatch -K0 job_name.job
gives an error
sbatch: invalid option -- 'K'
and putting
#SBATCH --kill-on-bad-exit=0
in the top of the .job file gives the error
sbatch: unrecognized option '--kill-on-bad-exit=0'

Loris, thanks also. If Chris's tip can't solve my issues I'll post more detaeeld discussions of the software I'm working with, but it can get quite confusing and took me a long time to get this software running on the cluster even in the "one per node" form I currently use it, hence my preference to tinker with SLURM settings rather than try to change the software.

--
You received this message because you are subscribed to a topic in the Google Groups "slurm-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/slurm-users/I1T6GWcLjt4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to slurm-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/slurm-users/70b2e90b-4939-7105-8a15-eb5a60addd99%40csamuel.org.

--

Thanks

----------

Robert Peck

Robot Lab

Intelligent Systems and Nanoscience group

Department of Electronic Engineering

University of York

[Robert Peck]

P.S. the slurm version here is 20.02.3

[Robert Peck]

Submission did succeed when I put the -K0 inside the srun command within my job script though. Will be a while before my job runs though, so won't know for a little while whether the KillOnBadExit flag has helped.