[slurm-users] Partition dedicated to batch jobs only

[Moshe Mergy]

Hi all,

How could I define a partition dedicated to batch jobs only, i.e. to use with sbatch only, 

not permitting to run srun/interactive jobs on it?

Using Slurm options ? 

Using a Lua script blocking srun ? (though I don't understand much in Lua scripts)

I currently use Slurm 20.02

Thanks a lot for your help!

Moshe

[René Sitt]

Hello,

the easiest way to identify interactive jobs is probably via job_submit.lua - if a submitted job does not have an associated job script, it's most likely interactive.

As an example snippet, at our cluster we force interactive jobs into their own partition (has the advantage that you can attach an "interactive" QOS to that partition to enforce specific limits like e.g. only allowing one interactive job per user):

--- /etc/slurm/job_submit.lua ---

function slurm_job_submit(job_desc, part_list, submit_uid)

    local log_prefix = "slurm_job_submit"

    local interactive_partition = "interactive"

    -- ... more local variable definitions and setup ...

   

    -- check for interactive

    if (job_desc.script == nil or job_desc.script == '') then
        job_desc.partition = interactive_partition
        slurm.log_info("%s: normal job seems to be interactive, moved to %s partition.", log_prefix, job_desc.partition)
    end

   

    -- ...some more job filtering ...

    slurm.log_info("%s: for user %u, setting partition(s): %s.", log_prefix, submit_uid, job_desc.partition)
    slurm.log_user("Job \"%s\" queued to partition(s): %s.", job_desc.name, job_desc.partition)

    return slurm.SUCCESS

end

function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
   return slurm.SUCCESS
end

slurm.log_info("initialized")
return slurm.SUCCESS

--- /etc/slurm/job_submit.lua ---

Of course, if you only want to keep interactive jobs out of a specific partition, you could additionally check job_desc.partition for that partition's name in the above if-block and then decide what to do with it (change job partition to an allowed one, reject the job completely by returning slurm.ERROR, ...)

Best regards,
René Sitt

Am 09.05.22 um 14:17 schrieb Moshe Mergy:

-- 
Dipl.-Chem. René Sitt
Hessisches Kompetenzzentrum für Hochleistungsrechnen
Philipps-Universität Marburg
Hans-Meerwein-Straße
35032 Marburg

Tel. +49 6421 28 23523
si...@hrz.uni-marburg.de
www.hkhlr.de

[Moshe Mergy]

Thanks a lot René !

It helps me a lot!

An additional question about the log messages from job_submit.lua: 

With slurm.log_info(), I want to display the job ID instead of the job name "job_desc.name" in function  slurm_job_submit(),

but I didn't find such a relevant field in the job_desc type.

I find some example scripts using "job_ptr.job.id" or "job_rec.job_id", 

but as far as I understand, function slurm_job_submit can not handle parameters other than (job_desc, part_list, submit_uid)

Maybe you have an idea about displaying messages with JOB ID value via slurm_job_submit() ?

Thanks again!

Moshe

[René Sitt]

Hello Moshe,

I don't think that's possible (someone else may correct me if I'm wrong), as in my understanding, the job ID is assigned only after job_submit.lua has run through. The job_ptr struct (or job_rec for the lua API) is available for the slurm_job_modify() function but not for slurm_job_submit(), since it has fields that can only have values after a successful job submit and resource allocation by the scheduler.

Best regards,
René Sitt

Am 10.05.22 um 13:45 schrieb Moshe Mergy: