[slurm-users] seff for GPU

[Josu Lazkano Lete via slurm-users]

Hello,

We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?

It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.

Kind regards.

			Josu Lazkano Lete Systems Manager Infrastructures and General Services jlaz...@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it.  Avoid printing this message if it is not strictly necessary.

[John Hearns via slurm-users]

Well you do not say which type of GPU you use...

If you use AMD this may be useful

https://github.com/ROCm/device-metrics-exporter

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Loris Bennett via slurm-users]

Hi,

Josu Lazkano Lete via slurm-users

<slurm...@lists.schedmd.com> writes:

> Hello,
>
> We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
>
> It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.

Various sites have produced their own versions of 'seff'-like programs.
We currently use

https://github.com/PrincetonUniversity/jobstats

which reports CPU, memory and GPU utility as well as providing
suggestions to users about the amount of resources users should request
for similar future jobs.

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
FUB-IT, Freie Universität Berlin

[Fulcomer, Samuel via slurm-users]

Brown also uses jobstats, with prometheus. 

[Josu Lazkano Lete via slurm-users]

Thanks for all your replies, we will check them.

We use Nvidia GPUs.

I will inform you about the updates.

Best regards.

			Josu Lazkano Lete Systems Manager Infrastructures and General Services jlaz...@vicomtech.org +(34) 943 30 92 30
The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it.  Avoid printing this message if it is not strictly necessary.

[Prentice Bisbal via slurm-users]

On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
> Hi,
>
> Josu Lazkano Lete via slurm-users
> <slurm...@lists.schedmd.com> writes:
>
>> Hello,
>>
>> We are looking to optimize the GPU jobs of our HPC users, is it possible to add GPU info in the seff?
>>
>> It will be great to know how much GPU resources the users request and compare with how much GPU resources they use.
> Various sites have produced their own versions of 'seff'-like programs.
> We currently use
>
> https://github.com/PrincetonUniversity/jobstats
>
> which reports CPU, memory and GPU utility as well as providing
> suggestions to users about the amount of resources users should request
> for similar future jobs.
>
> Cheers,
>
> Loris
>
>

Both seff and jobstats were created by the same group of people at
Princeton University.

Prentice

[Prentice Bisbal via slurm-users]

On 9/10/25 11:37 AM, Prentice Bisbal wrote:
> On 9/2/25 10:01 AM, Loris Bennett via slurm-users wrote:
>> Hi,
>>
>> Josu Lazkano Lete via slurm-users
>> <slurm...@lists.schedmd.com> writes:
>>
>>> Hello,
>>>
>>> We are looking to optimize the GPU jobs of our HPC users, is it
>>> possible to add GPU info in the seff?
>>>
>>> It will be great to know how much GPU resources the users request
>>> and compare with how much GPU resources they use.
>> Various sites have produced their own versions of 'seff'-like programs.
>> We currently use
>>
>>    https://github.com/PrincetonUniversity/jobstats
>>
>> which reports CPU, memory and GPU utility as well as providing
>> suggestions to users about the amount of resources users should request
>> for similar future jobs.
>>
>> Cheers,
>>
>> Loris
>>
>>
> Both seff and jobstats were created by the same group of people at
> Princeton University.
>
> Prentice
>

I just e-mailed one of the developers/maintainers of seff and jobstats.
seff is being deprecated because it uses the Slurm Perl API which
SchedMD is moving away from. For their own in-house use, jobstats has
largely replaced seff, but moving to jobstats requires setting up
additional machinery which seff doesn't require, so jobstats is not a
1:1 replacement for seff.

Unfortunately, it doesn't look like GPU support will be added to seff.

[Paul Edmon via slurm-users]

That's sad that seff is being deprecated due to dropping the perl api.
It would be great if it was reimplemented via the C API, REST API, or
via command parsing using python.

As I understand it OSC has implemented a seff-gpu along the same lines
of Princeton JobStats (they require the same collectors) but the source
code is not public. It would be great to merge that into the general job
stats platform.

Suffice it to say seff is a really useful command and it would be a
shame to see it die, it should in fact be extended to include GPUs.

-Paul Edmon-

[Hagdorn, Magnus Karl Moritz via slurm-users]

On Tue, 2025-09-02 at 15:38 +0200, Josu Lazkano Lete via slurm-users
wrote:

> We are looking to optimize the GPU jobs of our HPC users, is it
> possible to add GPU info in the seff?
>
> It will be great to know how much GPU resources the users request and
> compare with how much GPU resources they use.

Hi Josu,
I have forked sps and added support for Nvidia and AMD GPUs

https://github.com/mhagdorn/sps

Regards
magnus


--
Dr. Magnus Hagdorn
Charité – Universitätsmedizin Berlin
Geschäftsbereich IT | Scientific Computing
 
Campus Charité Mitte
BALTIC - Invalidenstraße 120/121
10115 Berlin
 
magnus....@charite.de
https://www.charite.de
HPC Helpdesk: sc-hpc-...@charite.de