[slurm-users] Restrict and prioritize usage of certain nodes according to accounts

[thomas.hartmann--- via slurm-users]

Hi,
I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account.

I have two sets of physical nodes:
1. node01 - nodeXX: General Purpose Nodes. Everybody can use them
2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them.

Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts.

I have three sets of accounts (each can have child accounts):
1. "General" accounts: These are allowed to use all physical nodes.
2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes
3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.

My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.

In the future, there might be more project specific nodes for other projects.

I would be grateful for any alternative solution or hints, how this could be solved optimally.

Thanks a lot!
Thomas

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Sean Mc Grath via slurm-users]

Hi,

> My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.

That is what we do. We also set a higher priority on the restricted access partition so jobs in it get queued faster. I don't know if it is the best way to do this though.

Thanks

Sean

---
Sean McGrath

Systems Administrator
Research IT
IT Services
Trinity College Dublin

00 353 - (0)1 896 3725
https://www.tcd.ie/itservices/
https://www.tchpc.tcd.ie/

---

From: thomas.hartmann--- via slurm-users <slurm...@lists.schedmd.com>
Sent: Wednesday 21 May 2025 09:25
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: [slurm-users] Restrict and prioritize usage of certain nodes according to accounts

 

[External Email] This email originated outside of Trinity College Dublin. Do not click links or open attachments unless you recognise the sender and know the content is safe.

[Bjørn-Helge Mevik via slurm-users]

"thomas.hartmann--- via slurm-users" <slurm...@lists.schedmd.com>
writes:

> I have three sets of accounts (each can have child accounts):
> 1. "General" accounts: These are allowed to use all physical nodes.
> 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes
> 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.
>
> My first idea would be to create two partitions, one with all nodes in
> there and a second one with only those nodes that the "ForProfit" are
> allowed to use. Using "AllowAccounts" or "DenyAccounts" would
> implement the restriction I need.

That will work, but will not ensure that Project_A jobs first land on
the projectt_A_* nodes (unless you give these nodes a lower weight - but
then the General jobs will also start there first).

You could add a third partition with the project_A_* nodes, and only
give Project_A access to it. You can then give this partition a higher
PriorityTier than the other partitions, meaning that jobs asking for
that partitions will be scheduled first. Users of project_A should then
submit jobs with -p project_A_part,General_part, or you could add some
logic in your job_submit.lua that makes sure this happens.

--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

[Daniel Letai via slurm-users]

You could add features (constraints) to project_A_* nodes in slurm.conf:

NodeName=DEFAULT ...

NodeName=project_A_[01-80] Features=project_A

and then in your batch script template for project_A add something like

#SBATCH --prefer=project_A

Of course this will require project_A accounts to use your template, since Slurm doesn't have a builtin mechanism to require accounts to use constraints.

References:

https://slurm.schedmd.com/slurm.conf.html#OPT_Features

https://slurm.schedmd.com/sbatch.html#OPT_prefer

I didn't test this solution, just basing it on documentation.

This should overcome the issue with using multiple partitions in sbatch -p, namely:

If project_A job requires 2 nodes, but currently only 1 node is available in partition project_A_part, the job will be allocated both nodes from general_part, and there is no guarantee the free project_A_ node will be used.

HTH,

--Dani_L.

[Prentice Bisbal via slurm-users]

> Users are associated with one or more accounts.

If you can avoid this ( I know that's a big "if"), avoid it! Humans
being humans (lazy) and users begin users (copying existing work as much
as possible), if a user has multiple accounts, there is a damn good
chance they'll submit jobs to the wrong accounts eventually, messing up
priority/fairshare and accounting/billing, which will eventually add up
to headaches for you.

I know it's not always possible to eliminate it, and it can be
needed/desirable to have this capability. If you can minimize it as much
as possible, though, it will minimize headaches for you.

I did work at one place where a Slurm user could only belong to one
account at a time to prevent those sort of issues.

Prentice

Prentice Bisbal
HPC Systems Engineer III
Computational & Information Systems Laboratory (CISL)
NSF National Center for Atmospheric Research (NSF NCAR)
https://www.cisl.ucar.edu
https://ncar.ucar.edu