[slurm-users] Epilog error

[Ratnasamy, Fritz via slurm-users]

Hello,

Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: 

ls -lsrt /scratch
drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe

The epilog script usually takes care of deleting it: 
echo -e "\n**** JOB EPILOG ****\n"

if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then
    rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID}
fi

Sometimes, the node get in draining state because of an epilog error for : 
[2026-08-12T16:36:30.215] error: epilog failed: rc:1 output:
**** JOB EPILOG ****
rm: cannot remove '/scratch/johndoe/464668': Permission denied

This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? 

Best,

Fritz Ratnasamy
Data Scientist
Information Technology

[John Hearns via slurm-users]

I don't know.

Maybe you could test the existence and permissions then log these before the deletion ?

Are you not better to use TMPDIR and define which filesystem TMPDIR uses?

Are the failed jobs very short?

Is this some caching weirdness with a parallel filesystem 

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Bjørn-Helge Mevik via slurm-users]

"Ratnasamy, Fritz via slurm-users" <slurm...@lists.schedmd.com>

writes:

> Sometimes, the node get in draining state because of an epilog error for :
> [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output:
> **** JOB EPILOG ****
> rm: cannot remove '/scratch/johndoe/464668': Permission denied
>
>
> This seems to happen on random jobs and eventually the files get deleted
> but the node remains in draining state. Does anyone know what is happening?

While I don't know why it happens, in the epilog, you could check the
exit status of "rm", and if it wasn't 0, you could do an "ls -la
/scratch/johndoe/" or similar and log that to a log file. Then you
might find out what the reason is. (We do things like this, and then we
have a script that checks for such log files from the epilogs.)

Also, if you don't want the node to be drained when this happens, just
make sure the epilog exits with exit status 0 whether the "rm" fails or
not.

--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo