[slurm-users] Cluster usage, filtered by partition
Diego Zuccato
Is it possible to extract a "partition usage summary", like the one
generated by "sreport cluster usage" but limited to a single partition
(or a partition set)?
Alternatively, is there some recommended 3rd party tool to generate
"executive" reports (or at least the data to put in a spreadsheet for
further processing)?
Tks.
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Ole Holm Nielsen
> Is it possible to extract a "partition usage summary", like the one
> generated by "sreport cluster usage" but limited to a single partition (or
> a partition set)?
> Alternatively, is there some recommended 3rd party tool to generate
> "executive" reports (or at least the data to put in a spreadsheet for
> further processing)?
https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmacct
Maybe you can use the "topreports" tool?
/Ole
Diego Zuccato
Tks for the very fast answer.
> I have written some accounting tools which are in
> https://github.com/OleHolmNielsen/Slurm_tools/tree/master/slurmacct
> Maybe you can use the "topreports" tool?
witdh: our usernames are quite long, being from AD), but first I have to
check if it extracts the info our users want to see :)
Renfro, Michael
On May 11, 2021, at 5:18 AM, Diego Zuccato <diego....@unibo.it> wrote:
Il 11/05/21 11:21, Ole Holm Nielsen ha scritto:
Tks for the very fast answer.
I have written some accounting tools which are in
Paul Edmon
Yup, we use XDMod for this sort of data as well.
-Paul Edmon-
Kilian Cavalotti
>
> XDMoD [1] is useful for this, but it’s not a simple script. It does have some user-accessible APIs if you want some report automation. I’m using that to create a lightning-talk-style slide at [2].
>
> [1] https://open.xdmod.org/
> [2] https://github.com/mikerenfro/one-page-presentation-hpc
Thanks,
--
Kilian
Renfro, Michael
In a word, nothing that's guaranteed to be stable. I got my start from this reply on the XDMoD list in November 2019. Worked on 8.0:
Mike,
The recommended way of doing this would be to use XDMoD's Report Generator to periodically email you a document containing the chart images.
https://xdmod.ccr.buffalo.edu/user_manual/?t=Report%20Generator
This will only get you the images, though and not the numerical values.
The more complex alternative is to use curl to query XDMoD directly. An example of how to download chart images is in the automated regression tests that verify image export:
https://github.com/ubccr/xdmod/blob/xdmod9.0/tests/regression/lib/Controllers/UsageChartsTest.php
See the chartSettingsProvider() for how to create the data to send and testChartSettings() for where to POST it. To get the raw numbers you can change the 'format'
setting from 'png' to 'csv' to get the raw data in csv format. Note that you would be accessing an internal XDMoD api which could change or even be removed in new releases.
From:
slurm-users <slurm-use...@lists.schedmd.com> on behalf of Kilian Cavalotti <kilian.cav...@gmail.com>
Date: Tuesday, May 11, 2021 at 1:57 PM
To: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Cluster usage, filtered by partition
On Tue, May 11, 2021 at 5:55 AM Renfro, Michael <Ren...@tntech.edu> wrote:
>
> XDMoD [1] is useful for this, but it’s not a simple script. It does have some user-accessible APIs if you want some report automation. I’m using that to create a lightning-talk-style slide at [2].
>
> [1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopen.xdmod.org%2F&data=04%7C01%7Crenfro%40tntech.edu%7C406944c7097041e8402f08d914ae978f%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637563562374746702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Mla%2BlItr2M4XNEzP7UoAAP2z3P%2FPPmfE2%2B8zTwTE4W4%3D&reserved=0
> [2]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikerenfro%2Fone-page-presentation-hpc&data=04%7C01%7Crenfro%40tntech.edu%7C406944c7097041e8402f08d914ae978f%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637563562374746702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pkOtpZ7L7qyiK8kICUmJGPioIwzxvCRC0i%2BQOC9usHM%3D&reserved=0
Diego Zuccato
> In a word, nothing that's guaranteed to be stable. I got my start from
> this reply on the XDMoD list in November 2019. Worked on 8.0:
XDMoD seems interesting and I'll try to have a look. But a scientific
report w/o access to the bare numbers is definitely a no-no :)
Greg Wickham
Hi Diego,
Disclaimer: A little bit of shameless self-promotion.
We're using an application I wrote to inject slurm accounting records into a PostreSQL database. The
data is extracted from Slurm using "sacct".
From there it's possible to use SQL queries to mine the raw slurm data.
https://gitlab.com/greg.wickham/sminer
This tool _only_ injects Slurm data into PostgreSQL, unlike XDMoD (which can do this and more).
However the a big benefit for us is sminer can inject records into an existing database (no need for a separate database).
CSV dumps can be obtained using native PostgreSQL commands.
Graphs are created using python scripts (querying the data) and then plotted with gnuplot.
-Greg
—
-----Original Message-----
From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Diego Zuccato <diego....@unibo.it>
Reply to: Slurm User Community List <slurm...@lists.schedmd.com>
Date: Wednesday, 12 May 2021 at 11:57 am
To: Slurm User Community List <slurm...@lists.schedmd.com>, "Renfro, Michael" <Ren...@tntech.edu>
Subject: [EXTERNAL] Re: [slurm-users] Cluster usage, filtered by partition
Renfro, Michael
Diego Zuccato
> Not sure which raw numbers you’re looking for, but I’m also getting a
> CSV export from XDMoD to calculate the total number of jobs and CPU
> hours we’ve completed. Doing that through the API as well.
-8<--
Open XDMoD is not compatible with PHP 7 at this time.
-8<--
It seems strange, since PHP5 it's been "dead" for quite some time. Maybe
just usual docrot? :) Anyone using it on PHP7.4 or even 8.0?
PHP5 would be a real showstopper for a public-facing website :(
Anyway, at a first glance, it uses a bit too many technologies for my
taste (php, java, js...) and could be a problem integrating it in a
vhost managed by one of our ISPConfig instances. But I'll try it.
Somehow I'll make it work :)
Diego Zuccato
> Anyway, at a first glance, it uses a bit too many technologies for my
> taste (php, java, js...) and could be a problem integrating it in a
> vhost managed by one of our ISPConfig instances. But I'll try it.
> Somehow I'll make it work :)
abadonware. Too many things that can go wrong and can't be patched, IMVHO.
Renfro, Michael
By the strictest definition of abandonware, not really, they released 9.5rc4 last week [1]. Won't argue any of the other points, since that's out of my depth, but there's a very low-volume mailing list at CCR-XDM...@LISTSERV.BUFFALO.EDU you could inquire at.
From:
Diego Zuccato <diego....@unibo.it>
Date: Wednesday, May 12, 2021 at 8:37 AM
To: Renfro, Michael <Ren...@tntech.edu>
Cc: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Cluster usage, filtered by partition
Diego Zuccato
PhantomJS is abadonware, at least according to its author.
I've been bitten too many times to ignore that a sw I'm evaluating is
based on something already abandoned. But I'll try asking on XDMoD list
to see if my fears are correct. Hope it's just a docrot issue (maybe
they already added compatibility with PHP7.4+ and replaced PhantomJS
with another headless browser). Tks.
> By the strictest definition of abandonware, not really, they released
> 9.5rc4 last week [1]. Won't argue any of the other points, since that's
> out of my depth, but there's a very low-volume mailing list at
> CCR-XDM...@LISTSERV.BUFFALO.EDU
>
> [1] https://github.com/ubccr/xdmod/releases/tag/v9.5.0-rc.4
> <https://github.com/ubccr/xdmod/releases/tag/v9.5.0-rc.4>
>
> *From: *Diego Zuccato <diego....@unibo.it>
> *Date: *Wednesday, May 12, 2021 at 8:37 AM
> *To: *Renfro, Michael <Ren...@tntech.edu>
> *Cc: *Slurm User Community List <slurm...@lists.schedmd.com>
> *Subject: *Re: [slurm-users] Cluster usage, filtered by partition
