[slurm-users] Limit run time of interactive jobs

[Marko Markoc]

Hi All,

Quick question. Is there a way to limit the runtime on a partition only for salloc ? I would like for batch jobs to have a default max runtime of the partition but interactive jobs to have shortened allowed runtime. 

Thanks!

[Angel de Vicente]

Hi Marko,

I'm also interested in this (in my case to allow oversubscription only
for interactive jobs), but as far as I know, that is not possible in
general.

But one possible way to something similar is to have a partition only
for interactive jobs and a different partition for batch jobs, and then
enforce that each job uses the right partition. In order to do this, I
think we can use the Lua contrib module (check the job_submit.lua
example).

Cheers,
--
Ángel de Vicente
Research Software Engineer (Supercomputing and BigData)
Tel.: +34 922-605-747
Web.: http://research.iac.es/proyecto/polmag/

GPG: 0x8BDC390B69033F52

[Bjørn-Helge Mevik]

Angel de Vicente <angel.de...@iac.es> writes:

> But one possible way to something similar is to have a partition only
> for interactive jobs and a different partition for batch jobs, and then
> enforce that each job uses the right partition. In order to do this, I
> think we can use the Lua contrib module (check the job_submit.lua
> example).

Wouldn't it be simpler to just refuse too long interactive jobs in
job_submit.lua?

--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

[Angel de Vicente]

Hi,

Bjørn-Helge Mevik <b.h....@usit.uio.no> writes:

> Wouldn't it be simpler to just refuse too long interactive jobs in
> job_submit.lua?

Yes, I guess. I proposed the idea of having different partitions because
then the constraints are at the level of the partition, which is
probably easier to handle than modifying the job_submit.lua script?, but
probably can get the same result both ways.

Anyway, the goal was to point to the job_submit.lua stuff, because
without it, even if creating separate partitions for batch and
interactive jobs, it is not possible (or at least I wouldn't know how)
to force a certain policy only for interactive jobs.

[Ole Holm Nielsen]

On 5/8/23 08:39, Bjørn-Helge Mevik wrote:
> Angel de Vicente <angel.de...@iac.es> writes:
>
>> But one possible way to something similar is to have a partition only
>> for interactive jobs and a different partition for batch jobs, and then
>> enforce that each job uses the right partition. In order to do this, I
>> think we can use the Lua contrib module (check the job_submit.lua
>> example).
>
> Wouldn't it be simpler to just refuse too long interactive jobs in
> job_submit.lua?

This sounds like a good idea, but how would one identify an interactive
job in the job_submit.lua script? A solution was suggested in
https://serverfault.com/questions/1090689/how-can-i-set-up-interactive-job-only-or-batch-job-only-partition-on-a-slurm-clu


> Interactive jobs have no script and job_desc.script will be empty / not set.

So maybe something like this code snippet?

if job_desc.script == NIL then
-- This is an interactive job
-- make checks of job timelimit
if job_desc.time_limit > 3600 then
slurm.log_user("NOTICE: Interactive jobs are limited to 3600 seconds")
-- ESLURM_INVALID_TIME_LIMIT in slurm_errno.h
return 2051
end
end

/Ole

[Bjørn-Helge Mevik]

Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> writes:

> On 5/8/23 08:39, Bjørn-Helge Mevik wrote:
>> Angel de Vicente <angel.de...@iac.es> writes:
>>
>>> But one possible way to something similar is to have a partition only
>>> for interactive jobs and a different partition for batch jobs, and then
>>> enforce that each job uses the right partition. In order to do this, I
>>> think we can use the Lua contrib module (check the job_submit.lua
>>> example).
>> Wouldn't it be simpler to just refuse too long interactive jobs in
>> job_submit.lua?
>
> This sounds like a good idea, but how would one identify an
> interactive job in the job_submit.lua script?

Good question. :) I merely guessed it is possible. :)

> A solution was suggested in
> https://serverfault.com/questions/1090689/how-can-i-set-up-interactive-job-only-or-batch-job-only-partition-on-a-slurm-clu
>> Interactive jobs have no script and job_desc.script will be empty /
> not set.
>
> So maybe something like this code snippet?
>
> if job_desc.script == NIL then

That sounds like it should work, yes. (But perhaps double check that jobs
submitted with "sbatch --wrap" or taking the job script from stdin (if
that is still possible) get job_descr.script set.)

--
B/H

[Angel de Vicente]

Hello,

Bjørn-Helge Mevik <b.h....@usit.uio.no> writes:

>> A solution was suggested in
>> https://serverfault.com/questions/1090689/how-can-i-set-up-interactive-job-only-or-batch-job-only-partition-on-a-slurm-clu
>>> Interactive jobs have no script and job_desc.script will be empty /
>> not set.
>>
>> So maybe something like this code snippet?
>>
>> if job_desc.script == NIL then

In my case (merely a variation from some older post here at
slurm-users), I'm using the following to make sure jobs go to the right
queue (either 'batch' or 'interactive'), and it seems to work just
fine:


if (job_desc.script == nil or job_desc.script == '') then
if (job_desc.partition ~= interactive_partition) then
job_desc.partition = interactive_partition
slurm.log_user("%s: normal job seems to be interactive, moved to %s partition.", log_prefix, job_desc.partition)
end
else
if (job_desc.partition == interactive_partition) then
job_desc.partition = batch_partition
slurm.log_user("%s: batch jobs cannot be run in the interactive partition, moved to %s partition.", log_prefix, job_desc.partition)
end
end