[slurm-users] Performance tracking of array tasks

[William Dear]

Could anyone please recommend methods of tracking the performance of individual tasks in a task array job?  I have installed XDMoD but it is focused solely on the Job level with no information about tasks.

My users almost exclusively use task arrays to run embarrassingly parallel jobs.  After the job is complete I would like to see run time and peak RAM usage per task so that we can correctly size the reservations for future jobs.  It would also be very helpful to break this down by node so that I can identify poorly performing nodes.

William Dear

_____________________________________
Confidentiality Notice - The information contained in this e-mail and any attachments to it may be legally privileged and include confidential information. If you are not the intended recipient, be aware that any disclosure, distribution or copying of this e-mail or its attachments is prohibited. If you have received this e-mail in error, please notify the sender immediately of that fact by return e-mail and permanently delete the e-mail and any attachments to it.

[Brian Andrus]

I think you may be conflating the array task with slurmstepd tasks.

Every array task does have it's own job id, usually <JOBID>_<TASKID>

It will also have a 'normal' job id that goes with that.

You can get detailed information from sacct if you have configured it. In fact, it also breaks things down into slurm steps, so you would be able to get pretty good detailed info.

Brian Andrus

[Loris Bennett]

Hi William,

William Dear <willia...@i3-corps.com> writes:

> Could anyone please recommend methods of tracking the performance of individual tasks in a task array job? I have installed XDMoD but it is focused solely on the Job level with no information about
> tasks.
>
> My users almost exclusively use task arrays to run embarrassingly parallel jobs. After the job is complete I would like to see run time and peak RAM usage per task so that we can correctly size the
> reservations for future jobs. It would also be very helpful to break this down by node so that I can identify poorly performing nodes.
>
> William Dear

I'm not sure what you mean by a 'task array job'. A job can have
multiple tasks within it - I don't think you will be able to get data on
such individual tasks very easily. However, a job array is just a sort
of convenient wrapper around a bunch of jobs. Each element of a job
array still has its own job ID, so you can extract job data the same way
you do for a non-array job.

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin Email loris....@fu-berlin.de

[William Dear]

It looks like Brian's suggestion of using SACCT will be the fast answer in the short term so I'll just have to write my own script to aggregate the output.  I was hoping for a canned solution such as XDMoD but haven't found one that quite fits our needs.  If there's a list of recommended supporting applications for SLURM I would appreciate that.

One example of how the canned reporting doesn't meet our needs is that my users self limit their arrays such as "--array=1-12000%100".  Technically, the initial job isn't waiting on anything but itself since it only runs 100 at a time but all the pending array jobs still show up as waiting.  If the partition resources are too low and the job is running less than 100 then it actually is waiting on another job.  The challenge will be determining when a job is self limiting vs waiting on a different job.

Thanks,

William Dear

---

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Loris Bennett <loris....@fu-berlin.de>
Sent: Monday, May 16, 2022 9:04 AM
To: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Performance tracking of array tasks

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

[Loris Bennett]

Hi,

William Dear <willia...@i3-corps.com> writes:

> It looks like Brian's suggestion of using SACCT will be the fast answer in the short term so I'll just have to write my own script to aggregate the output. I was hoping for a canned solution such as XDMoD but haven't found one that quite
> fits our needs. If there's a list of recommended supporting applications for SLURM I would appreciate that.
>
> One example of how the canned reporting doesn't meet our needs is that my users self limit their arrays such as "--array=1-12000%100". Technically, the initial job isn't waiting on anything but itself since it only runs 100 at a time but
> all the pending array jobs still show up as waiting. If the partition resources are too low and the job is running less than 100 then it actually is waiting on another job. The challenge will be determining when a job is self limiting vs
> waiting on a different job.

What is the use-case for having users need to self-limit? We just rely
on the cap for the maximum number of jobs in an array and on fairshare
to do the rest.

Cheers,

Loris

> Thanks,
>
> William Dear
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[William Dear]

> What is the use-case for having users need to self-limit?

Our users self limit jobs with extremely high disk IO requirements.  Some batch jobs read/write over 15TB a day and I haven't identified an effective method of capping IOPS per user.  We still have issues with the occasional user deciding to use SLURM to extract hundreds of 60GB tar.gz files in parallel with no task limits.  One of my current goals is to find a method of quickly identifying jobs with high IO Wait so that a single user can't DDOS the storage.  Unfortunately, all jobs using the same storage device end up with high IO wait so identifying the culprit requires also comparing total IO per job.

William Dear

---

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Loris Bennett <loris....@fu-berlin.de>

Sent: Tuesday, May 17, 2022 12:46 AM