[slurm-users] gres.conf and select/cons_res plugin

[Patrick Goetz]

I think reading the documentation is making me more confused; maybe this
has to do with version changes. My current slurm cluster is using
version 17.x

Looking at the man page for gres.conf
(https://slurm.schedmd.com/gres.conf.html) I see this:

NOTE: Slurm support for gres/[mps|shard] requires the use of the
select/cons_tres plugin.

On my current (inherited) Slurm cluster we have:

SelectType=select/cons_res

but users are primarily using GPU resources, so I know Gres is working.
Why then is select/cons_tres required?

[Kamil Wilczek]

Hello,

unfortunately imho Slurm's documentation is "scattered".
What I mean is that a single subject is not defined in one place but
described in several places. It is so for cons_tres.

Among others cons_tres allows for individual allocation of
GPUs and other TRES, provides additional submit options to jobs, such
as --gpus, allows for default CPU or memory allocation per GPU, etc.
cons_res allowed only for cores and memory.
This is at least my understading. I ran 19.X and now I'm running 22.05.

See documents below and search for "cons_tres", there are multiple
entries in each document. I remember searching in the whole
manual dozens of times when searching for answers :) :/
And sometimes answers are not quite self-explanatory and confusing.

https://slurm.schedmd.com/gres.html
https://slurm.schedmd.com/slurm.conf.html
https://slurm.schedmd.com/cons_res.html

Kind regards
--


W dniu 13.09.2022 o 23:04, Patrick Goetz pisze:

--
Kamil Wilczek [https://keys.openpgp.org/]
[6C4BE20A90A1DBFB3CBE2947A832BF5A491F9F2A]
Laboratorium Komputerowe
Wydział Matematyki, Informatyki i Mechaniki
Uniwersytet Warszawski

ul. Banacha 2
02-097 Warszawa

Tel.: 22 55 44 392
https://www.mimuw.edu.pl
https://www.uw.edu.pl

[Ole Holm Nielsen]

Please note that the on-line Slurm documentation refers to version 22.05
(the latest version)! For your outdated version 17.x you will have to
find the old documentation.

Of course, upgrading to 22.05 is very strongly recommended! Please note
that you must upgrade by no more than 2 major releases at a time!! See
some notes in
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

/Ole

[Yair Yarom]

Hi,

I don't remember the exact details, but we started with cons_res a while back, and at one of the upgrades we moved to cons_tres which was newer and supported more options (I don't remember which, but I think gpus were generally supported with cons_res). I don't think we lost any features when we switched.

Indeed you need to look at your version's documentation, e.g. cons_tres doesn't appear in:

https://slurm.schedmd.com/archive/slurm-17.11.0/cons_res.html

but appears in

https://slurm.schedmd.com/cons_res.html

And from the latter:

The Consumable Trackable Resources (cons_tres) plugin provides all the same functionality provided by the Consumable Resources (cons_res) plugin. It also includes additional functionality specifically related to GPUs.

--

  /|       |
  \/       | Yair Yarom | System Group (DevOps)
  []       | The Rachel and Selim Benin School
  [] /\    | of Computer Science and Engineering
  []//\\/  | The Hebrew University of Jerusalem
  [//  \\  | T +972-2-5494522 | F +972-2-5494522
  //    \  | ir...@cs.huji.ac.il
 //        |