[slurm-users] derived counters

[Heckes, Frank]

Hi all,

Sorry, if this has been asked and answered before.
Does someone created a script/sql-query or maybe can provide combination of command line flags to create a ‘report’ for:

 

• partition utilization
• (average) waittime for job send to a certain partition

• (average) queue length for a certain partition

 

for different time scalings (perDay, perWeek, perTimeInterval) using historic data of the stored slurmdb records, and could share it?

Additional question: Does anyone got PySLURM running for slurm version 20.x.x?

Many thanks in advance.

 

Cheers,

-Frank

[Ole Holm Nielsen]

On 4/11/21 6:17 PM, Heckes, Frank wrote:
> Sorry, if this has been asked and answered before.
> Does someone created a script/sql-query or maybe can provide combination
> of command line flags to create a ‘report’ for:

I'm not sure my Slurm tools do what you want, but maybe you can get
partial answers for your questions using the tools in folders in
https://github.com/OleHolmNielsen/Slurm_tools:

> * partition utilization

showuserjobs -p <partition> (in
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserjobs)
show the current usage

> * (average) waittime for job send to a certain partition

topreports (in
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmacct) shows
top users and average job waiting time.

> * (average) queue length for a certain partition

>
> for different time scalings (perDay, perWeek, perTimeInterval) using
> historic data of the stored slurmdb records, and could share it?
>
> Additional question: Does anyone got PySLURM running for slurm version 20.x.x?

/Ole

[Heckes, Frank]

Hello Ole, many thanks for sharing your scripts, they cover most of the topics I was looking for. (my apologies, I noticed them already, but didn't checked them careful enough). The script are very clean coded and documented. Great work.
Cheers,
-Frank

[Ole Holm Nielsen]

Hi Frank,

On 4/12/21 9:53 AM, Heckes, Frank wrote:
> Hello Ole, many thanks for sharing your scripts, they cover most of the topics I was looking for. (my apologies, I noticed them already, but didn't checked them careful enough). The script are very clean coded and documented. Great work.

Thanks for your nice comments. I'm glad these scripts can be of use.

>> -----Original Message-----
>>> * (average) queue length for a certain partition

I wonder what exactly does your question mean? Maybe the number of jobs
or CPUs in the Pending state? Maybe relative to the number of CPUs in the
partition?

The "slurmacct" command prints (possibly for a specified partition) the
average job waiting time while Pending in the queue, but not the queue
length information.

It may be difficult to answer your question from the Slurm database. The
sacct command displays accounting data for all jobs and job steps, but not
directly for partitions.

There are other Slurm monitoring tools which perhaps can supply the data
you are looking for. You could ask this list again.

/Ole

[Heckes, Frank]

Hello Ole,

> >> -----Original Message-----
> >>> * (average) queue length for a certain partition
>
> I wonder what exactly does your question mean? Maybe the number of jobs or
> CPUs in the Pending state? Maybe relative to the number of CPUs in the
> partition?
>

This result from a mgmt. - question. How long jobs have to wait (in s, min, h, day) before they getting executed and
how many jobs are waiting (are queued) for each partition in a certain time interval.
The first one is easy to find with sacct and submit, start counts + difference + averaging.
The second is a bit cumbersome, so I wonder whether a 'solution' is already around. The easiest way is to monitor from the beginning and store the squeue ouput for later evaluation. Unfortunately I didn’t do that.

Cheers,
-Frank

[Hadrian Djohari]

Hi Frank,

A way to get "how long jobs wait in the queue" is to import the data to XDMOD (https://open.xdmod.org/9.0/index.html).

The nifty reporting tool has many features to make it easier for us to report out the cluster usage.

Hadrian

--

Hadrian Djohari
Manager of Research Computing Services, [U]Tech
Case Western Reserve University
(W): 216-368-0395
(M): 216-798-7490

[Juergen Salk]

* Heckes, Frank <hec...@mps.mpg.de> [210413 12:04]:

> This result from a mgmt. - question. How long jobs have to wait (in s, min, h, day) before they getting executed and
> how many jobs are waiting (are queued) for each partition in a certain time interval.
> The first one is easy to find with sacct and submit, start counts + difference + averaging.

Hi Frank,

depending on the definition of "waiting time", the "reserved" field
from sacct may be more appropriate than "start" minus "submit". For
example for dependency jobs (aka chain jobs) the latter does also
count the time a job had to wait for another job to finish
whereas "reserved" will only start counting when a job becomes
eligible.

However, the "eligible" and "reserved" fields in sacct will be
set or increased also if a job has hit a resource throttling limit,
which may be something you want to factor out of the job waiting time
as well.

Unfortunaty, I haven't found any metrics in sacct that does only
count (or allows to derive) the time a job had to wait just for
sufficent resources to become available. Maybe someone else?

> The second is a bit cumbersome, so I wonder whether a 'solution' is
> already around. The easiest way is to monitor from the beginning and
> store the squeue ouput for later evaluation. Unfortunately I didn’t
> do that.

Not sure if this is a solution for you but I think you can at
least resample this retrospectively from sacct by using something like

sacct -a -X -S 2021-04-01T00:00:00 -s PD -o JobID,User,Partition

This will return job records for all jobs that were in pending state
at the specified time.

Best regards
Jürgen

[Renfro, Michael]

I'll never miss an opportunity to plug XDMoD for anyone who doesn't want to write custom analytics for every metric. I've managed to get a little bit into its API to extract current values for number of jobs completed and the number of CPU-hours provided, and insert those into a single slide presentation for introductory meetings.

 

You can see a working version of it for the NSF XSEDE facilities at https://xdmod.ccr.buffalo.edu

 

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Hadrian Djohari <hx...@case.edu>
Date: Tuesday, April 13, 2021 at 8:11 AM
To: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] derived counters

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

---

[Heckes, Frank]

Hi all, many thanks for all hints. The link in the latest pointing points to an impressive switch-board.

Cheers,

-Frank

[Matthew BETTINGER]

Before you get all excited about it,  we have had a terrible time trying to get gppu metrics.  Finally abandoned and switch to  Grafana, Prometheus influx.  Good luck to you though.

[Ole Holm Nielsen]

Hi Jürgen,

On 4/13/21 6:29 PM, Juergen Salk wrote:
> * Heckes, Frank <hec...@mps.mpg.de> [210413 12:04]:
>
>> This result from a mgmt. - question. How long jobs have to wait (in s, min, h, day) before they getting executed and
>> how many jobs are waiting (are queued) for each partition in a certain time interval.
>> The first one is easy to find with sacct and submit, start counts + difference + averaging.
>
> Hi Frank,
>
> depending on the definition of "waiting time", the "reserved" field
> from sacct may be more appropriate than "start" minus "submit". For
> example for dependency jobs (aka chain jobs) the latter does also
> count the time a job had to wait for another job to finish
> whereas "reserved" will only start counting when a job becomes
> eligible.

The slurmacct tool
(https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmacct)
calculates the waiting time as you recommend:

wait = start - eligible

I have experienced eligible == "Unknown", in which case I use the submit
time as the best guess.

> However, the "eligible" and "reserved" fields in sacct will be
> set or increased also if a job has hit a resource throttling limit,
> which may be something you want to factor out of the job waiting time
> as well.
>
> Unfortunaty, I haven't found any metrics in sacct that does only
> count (or allows to derive) the time a job had to wait just for
> sufficent resources to become available. Maybe someone else?

Good point! I don't have an answer...

>> The second is a bit cumbersome, so I wonder whether a 'solution' is
>> already around. The easiest way is to monitor from the beginning and
>> store the squeue ouput for later evaluation. Unfortunately I didn’t
>> do that.
>
> Not sure if this is a solution for you but I think you can at
> least resample this retrospectively from sacct by using something like
>
> sacct -a -X -S 2021-04-01T00:00:00 -s PD -o JobID,User,Partition
>
> This will return job records for all jobs that were in pending state

That's a nice trick! According to the sacct man-page, when you specify
the state (-s PD) and the starttime with -S, the DEFAULT TIME WINDOW in
this case sets endtime=starttime. Thus you get a snapshot of the Pending
jobs at the instant given by -S. This could definitely be used to make
graphs of Pending jobs in each partition as a function of time.

/Ole