You should look at LLN (least loaded nodes):
https://slurm.schedmd.com/slurm.conf.html
That should do what you want.
Kesim,
what you are saying is that Slurm schedukes tasks based on the number of allocated CPUs. Rather than the actual load factor on the server.
As I recall Gridengine actually used the load factor.
However you comment that "users run programs on the nodes" and "the slurm is aware about the load of non-slurm jobs"
IMHO, in any well-run HPC setup any user running jobs without using the scheduler would have their fingers broken. or at least bruised using the clue stick.
Seriously, three points:
a) tell users to use 'salloc' and 'srun' to run interactive jobs. They can easily open a Bash session on a compute node and do what they like. Under the Slurm scheduler.
b) implement the pam-slurm PAMmodule. It is a few minutes work. This means your users cannot go behind the sluem scheduler and log into the nodes
c) on Bright clusters which I configure, you have a healtcheck running which wans you when a user is detected as logging in withotu using Slurm
Seriously again. You have implemented an HPC infrastructure, and have gone to the time and effort to implement a batch scheduling system.
A batch scheduler can be adapted to let your users do their jobs, including interactive shell sessions and remote visualization sessions.
Do not let the users ride roughshod over you.
________________________________________
From: kesim [keti...@gmail.com]
Sent: 18 March 2017 16:16
To: slurm-dev
Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load
Unbelievable but it seems that nobody knows how to do that. It is astonishing that such sophisticated system fails with such simple problem. The slurm is aware about the cpu load of non-slurm jobs but it does not use the info. My original understanding of LLN was apparently correct. I can practically kill the CPUs on particular node with nonslurm tasks but slurm will diligently submit 7 jobs to this node leaving other idling. I consider this as a serious bug of this program.
On Fri, Mar 17, 2017 at 10:32 AM, kesim <keti...@gmail.com<mailto:ketiw...@gmail.com>> wrote:
Dear All,
Yesterday I did some tests and it seemed that the scheduling is following CPU load but I was wrong.
My configuration is at the moment:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU,CR_LLN
Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD info
node1 0.08 7/0/0/7
node2 0.01 7/0/0/7
node3 0.00 7/0/0/7
node4 2.97 7/0/0/7
node5 0.00 7/0/0/7
node6 0.01 7/0/0/7
node7 0.00 7/0/0/7
node8 0.05 7/0/0/7
node9 0.07 7/0/0/7
node10 0.38 7/0/0/7
node11 0.01 0/7/0/7
As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs on idling node11. Why such simple thing is not a default? What am I missing???
-- Andy Riebs andy....@hpe.com Hewlett-Packard Enterprise High Performance Computing Software Engineering +1 404 648 9024 My opinions are not necessarily those of HPE May the source be with you!