[slurm-users] Slurm cannot kill a job which time limit exhausted

[Taras Shapovalov]

Hey guys,

When a job max time is exceeded, then Slurm tries to kill the job and fails:

[2019-03-15T09:44:03.589] sched: _slurm_rpc_allocate_resources JobId=1325 NodeList=rn003 usec=355 
[2019-03-15T09:44:03.928] prolog_running_decr: Configuration for JobID=1325 is complete 
[2019-03-15T09:45:12.739] Time limit exhausted for JobId=1325 
[2019-03-15T09:45:44.001] _slurm_rpc_complete_job_allocation: JobID=1325 State=0x8006 NodeCnt=1 error Job/step already completing or completed 
[2019-03-15T09:46:12.805] Resending TERMINATE_JOB request JobId=1325 Nodelist=rn003 
[2019-03-15T09:48:43.000] update_node: node rn003 reason set to: Kill task failed 
[2019-03-15T09:48:43.000] update_node: node rn003 state set to DRAINING 
[2019-03-15T09:48:43.000] got (nil) 
[2019-03-15T09:48:43.816] cleanup_completing: job 1325 completion process took 211 seconds 

This happens even on very simple "srun bash" jobs that exceed their time limits. Have you idea what does it mean? Upgrade to the latest did not help.

Best regards,

Taras

[Prentice Bisbal]

Slurm is trying to kill the job that is exceeding it's time limit, but the job doesn't die, so Slurm marks the node down because it sees this as a problem with the node. Increasing the value for GraceTime or  KillWait might help:

GraceTime

Specifies, in units of seconds, the preemption grace time to be extended to a job which has been selected for preemption. The default value is zero, no preemption grace time is allowed on this partition. Once a job has been selected for preemption, its end time is set to the current time plus GraceTime. The job's tasks are immediately sent SIGCONT and SIGTERM signals in order to provide notification of its imminent termination. This is followed by the SIGCONT, SIGTERM and SIGKILL signal sequence upon reaching its new end time. This second set of signals is sent to both the tasks and the containing batch script, if applicable. Meaningful only for PreemptMode=CANCEL. See also the global KillWait configuration parameter.

KillWait

The interval, in seconds, given to a job's processes between the SIGTERM and SIGKILL signals upon reaching its time limit. If the job fails to terminate gracefully in the interval specified, it will be forcibly terminated. The default value is 30 seconds. The value may not exceed 65533.

--
Prentice