[slurm-users] free resources

[Pankaj Dorlikar]

Hi,

 

We are using slurm-20.11..7 on ubuntu system having GPUs. What is the equivalent of “showbf –S” in maui or any command in slurm for checking the free resources ?

 

Regards,

pvd

------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------

[Ole Holm Nielsen]

On 26-08-2021 08:01, Pankaj Dorlikar wrote:
> We are using slurm-20.11..7 on ubuntu system having GPUs. What is the
> equivalent of “showbf –S” in maui or any command in slurm for checking
> the free resources ?

Maybe "sinfo -t idle"?

The showbf manual doesn't document any -S flag:
https://docs.adaptivecomputing.com/maui/commands/showbf.php

/Ole

[Pankaj Dorlikar]

Thank you.

However, sinfo -t idle /sinfo -t mix is only giving the node names which are
idle/mix. To get exact number of cpus / gpus, what command need to be used.


showbf -S
backfill window (user: 'root' group: 'root' partition: ALL) Thu Aug 26
12:53:23

410 procs available for 3:57:21
218 procs available for 9:38:20
194 procs available for 1:10:51:51
192 procs available with no timelimit

-----Original Message-----
From: slurm-users <slurm-use...@lists.schedmd.com> On Behalf Of Ole
Holm Nielsen
Sent: 26 August 2021 12:41
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] free resources

On 26-08-2021 08:01, Pankaj Dorlikar wrote:
> We are using slurm-20.11..7 on ubuntu system having GPUs. What is the

> equivalent of "showbf -S" in maui or any command in slurm for checking

> the free resources ?

Maybe "sinfo -t idle"?

The showbf manual doesn't document any -S flag:
https://docs.adaptivecomputing.com/maui/commands/showbf.php

/Ole

[Ole Holm Nielsen]

On 26-08-2021 09:34, Pankaj Dorlikar wrote:
> However, sinfo -t idle /sinfo -t mix is only giving the node names which are
> idle/mix. To get exact number of cpus / gpus, what command need to be used.

If sinfo doesn't show what you want, perhaps you can try installing my
"pestat" command from
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat

The pestat can display very many job and node properties, see "pestat
-h". To display GPU resources available and used, try this one:

$ pestat -t mix -G
Select only nodes with state=mix
GRES (Generic Resource) is printed after each jobid
Hostname Partition Node Num_CPU CPUload Memsize Freemem
GRES/node Joblist
State Use/Tot (15min) (MB) (MB)
JobID(JobArrayID) User GRES/job ...
s002 sm3090 mix 64 80 8.15* 191000 156548
gpu:RTX3090:10 3941466 s184362 gpu:RTX3090:1 3941465 s184362
gpu:RTX3090:1 3941387 s184362 gpu:RTX3090:1 3940065 s184362
gpu:RTX3090:1 3939748 s184362 gpu:RTX3090:1 3939743 s184362
gpu:RTX3090:1 3941386 s184362 gpu:RTX3090:1 3941385 s184362 gpu:RTX3090:1

> showbf -S
> backfill window (user: 'root' group: 'root' partition: ALL) Thu Aug 26
> 12:53:23
>
> 410 procs available for 3:57:21
> 218 procs available for 9:38:20
> 194 procs available for 1:10:51:51
> 192 procs available with no timelimit

Again, the Maui command "showbf -S" is undocumented! Do you have
documentation for it?

/Ole