[slurm-users] Avoid killing GPU jobs during preemption

[Ratnasamy, Fritz via slurm-users]

Hello,

We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles.

A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. 
The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up.
Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. 
Thanks,

Fritz Ratnasamy
Data Scientist
Information Technology

[Loris Bennett via slurm-users]

Hi Fritz,

"Ratnasamy, Fritz via slurm-users"

<slurm...@lists.schedmd.com> writes:

> Hello,
>
> We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is
> low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles.

Fairshare itself does not necessarily prevent users from using idle resources.
If there is no competition for the resources, then anyone should be able to
use them, regardless of the resources that person has used recently.
Fairshare just decides the priority when there is competition. We have
several users who continually submit jobs and have no almost shares, but their
jobs still run, because there are always periods during which resources become
idle.

> A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is
> submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of
> available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs.
> There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem
> without killing the job. I know that would require the users to set up checkpoints and re-start their jobs.
> The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when
> activity picks up.
> Has anyone implemented something similar? I am in general against preemption because that might lead to conflict
> between users.
> Thanks,

For me, the main argument against pre-emption is that computing time may be
lost, if users are unable to checkpoint their jobs. Regarding possible
conflict between users, it should be possible to avoid the by expectation
management. We have some resources which were bought by individual research
groups and to which members of those research groups have privileged access
via a corresponding QOS. Everyone else can also access the resources, but
only via a 'scavenger' QOS, which has shorter run-times and other
restrictions. The 'scavenger' QOS is fairly heavily used, particularly for
GPU resources, and these jobs can be pre-empted, but we haven't had any
conflicts so far.

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
FUB-IT, Freie Universität Berlin

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com