[slurm-users] dynamic node slurm node list order

[Assmann, Greta Marie via slurm-users]

Hello, 

we would like to understand how the internal SLURM node list order works. More detailed info: 

Our setup: 

We have a Slurm cluster with N dynamic nodes (heterogeneous node types)  in a partition and we observe that if all resources are free, jobs always get submitted to one specific node first. This node is also the node that turns up at the top of the list  when doing an $ scontrol show nodes .  As we do not have a defined node list in slurm.conf as it is done for non-dynamic nodes, we were wondering how the order is set up. 

Interestingly, when deleting (and unregistering) this node from the cluster and re-registering it again, the node is still at the same position when doing scontrol show nodes. 

Is there some internal node list order caching or similar? 
How is the node list order defined ?   

Thanks , 
Greta 

Dr. Greta Assmann

Data Analysis and Research Infrastructure

OBBA 230 

Forschungsstrasse 111
5232 Villigen-PSI

email: greta....@psi.ch 

[Patrick Pun via slurm-users]

You can get a node list by
scontrol show node
Not sure if that order is a definitive one

But to resolve this issue you can try this
all node probably have the same node weight
Slurm pick first available node of the minimum node weight
You can manipulate the node weight change which node is picked
Either a cron job to randomise it or a prologslurmctld script to increment it.

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Ole Holm Nielsen via slurm-users]

On 10/22/25 14:41, Patrick Pun via slurm-users wrote:
> You can get a node list by
> scontrol show node
> Not sure if that order is a definitive one
>
> But to resolve this issue you can try this
> all node probably have the same node weight
> Slurm pick first available node of the minimum node weight

There's a description of this in
https://slurm.schedmd.com/slurm.conf.html#OPT_Weight

> All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements.

/Ole

[Assmann, Greta Marie via slurm-users]

Dear Patrick and Ole, 

Thanks a lot for your fast answers! 

As I mentioned in my email, I used scontrol show node(s)  in order to display the current order - I was more interested in understanding how the order is actually defined/set up by SLURM , as it is not defined in slurm.conf for dynamic nodes.  
We assumed that it might be related to the time the node registers, but as I unregistered and re-registered and the node stayed in the same "order" I was wondering if there is some internal caching...

Also -Yes, you are right, we have the same Weights for all nodes - but we would like to avoid setting weights for our setup, only if not possible otherwise...- but thanks again for the hint! 

So I would still like to clarify: 

• How SLURM determines the internal node order when nodes register dynamically given all weights/configs are the same ? (kinda : who/what defines the output of scrontrol show nodes) 
•  Whether this order persists across node or controller restarts (i.e., if SLURM caches previous node information)?

Best, Greta 

Dr. Greta Assmann

Data Analysis and Research Infrastructure

OBBA 230 

Forschungsstrasse 111
5232 Villigen-PSI

email: greta....@psi.ch 

---

From: Ole Holm Nielsen via slurm-users <slurm...@lists.schedmd.com>
Sent: 22 October 2025 15:02
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: [slurm-users] Re: dynamic node slurm node list order

 

[Brian Andrus via slurm-users]

I suspect it is the order of registration.

So, even if you unregister a node, it still exists internally. Even if you delete the node, it would still be in the db with a 'deleted' flag. I haven't dug into the code to validate, but that is what I have gone with and it hasn't been off for me.

Brian Andrus

[Assmann, Greta Marie via slurm-users]

Thanks, Brian, 
That matches our experience.. 

Dr. Greta Assmann

Data Analysis and Research Infrastructure

OBBA 230 

Forschungsstrasse 111
5232 Villigen-PSI

email: greta....@psi.ch 

---

From: Brian Andrus via slurm-users <slurm...@lists.schedmd.com>
Sent: 22 October 2025 22:39