[slurm-users] Store sstat information permanently on job completion?

[Jacob Chappell]

Is there a simple way to store sstat information permanently on job completion? We already have job accounting on, but the information collected from cgroups doesn't seem to be stored once a job finishes (sstat -j $JOB_ID on a dead job returns an error).

Thanks,

__________________________________________________
Jacob D. Chappell, CSM
Research Computing Associate
Research Computing | Research Computing Infrastructure
Information Technology Services | University of Kentucky
jacob.c...@uky.edu

[Paul Edmon]

All the aggregate historic data should be accessible via sacct.  sstat is for live jobs but sacct is for completed jobs.

-Paul Edmon-

[Jacob Chappell]

Ah! I didn't realize sacct included that data, but I see the fields listed now. Thanks!

__________________________________________________
Jacob D. Chappell, CSM
Research Computing Associate
Research Computing | Research Computing Infrastructure
Information Technology Services | University of Kentucky
jacob.c...@uky.edu

[Brian Andrus]

Except sstat can give you the MaxRSS without having cgroups and it will give you a simple MaxRSS, whereas sacct provides a MaxRSS for every step... have to play with that data to get the high water mark.... grrr.

I had tried to use sstat in an epilogue but apparently that is too late...

Brian Andrus

[Paddy Doyle]

We use sstat in the EpilogSlurmctld, and it gives info; so am not sure why
it would be too late?

But you might want to also look at seff in the slurm-contribs package, as
it gives a nicer user-friendly summary.

Paddy

On Wed, Oct 30, 2019 at 03:04:50PM -0700, Brian Andrus wrote:

> Except sstat can give you the MaxRSS without having cgroups and it will give
> you a simple MaxRSS, whereas sacct provides a MaxRSS for every step... have
> to play with that data to get the high water mark.... grrr.
>
> I had tried to use sstat in an epilogue but apparently that is too late...
>
> Brian Andrus
>
> On 10/30/2019 11:26 AM, Paul Edmon wrote:
> >
> > All the aggregate historic data should be accessible via sacct.  sstat
> > is for live jobs but sacct is for completed jobs.
> >
> > -Paul Edmon-
> >
> > On 10/30/2019 2:13 PM, Jacob Chappell wrote:
> > > Is there a simple way to store sstat information permanently on job
> > > completion? We already have job accounting on, but the information
> > > collected from cgroups doesn't seem to be stored once a job finishes
> > > (sstat -j $JOB_ID on a dead job returns an error).
> > >
> > > Thanks,
> > > __________________________________________________

> > > *Jacob D. Chappell, CSM*
> > > *Research Computing Associate*

> > > Research Computing | Research Computing Infrastructure
> > > Information Technology Services | University of Kentucky

> > > jacob.c...@uky.edu <mailto:jacob.c...@uky.edu>

--
Paddy Doyle
Research IT / Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
https://www.tchpc.tcd.ie/