[slurm-users] bufferoverflow in slurmd with acct_gather_energy plugin

[Hagdorn, Magnus Karl Moritz]

Hi all,
we recently enabled the energy gathering plugin on using the IPMI
gatherer with libfreeipmi. We are running the latest slurm 23.02.4 on
rocky 8.5. We are getting sporadic buffer overflows in slurmd when it
is trying to query the IPMI interface. We have the feeling this occurs
when a lot of jobs are getting started on the node. Has anybody come
across this issue and even better found a solution?
Regards
Magnus


--
Magnus Hagdorn
Charité – Universitätsmedizin Berlin
Geschäftsbereich IT | Scientific Computing
 
Campus Charité Virchow Klinikum
Forum 4 | Ebene 02 | Raum 2.020
Augustenburger Platz 1
13353 Berlin
 
magnus....@charite.de
https://www.charite.de
HPC Helpdesk: sc-hpc-...@charite.de

[Ole Holm Nielsen]

Hi Magnus,

On 8/28/23 10:16, Hagdorn, Magnus Karl Moritz wrote:
> we recently enabled the energy gathering plugin on using the IPMI
> gatherer with libfreeipmi. We are running the latest slurm 23.02.4 on
> rocky 8.5. We are getting sporadic buffer overflows in slurmd when it
> is trying to query the IPMI interface. We have the feeling this occurs
> when a lot of jobs are getting started on the node. Has anybody come
> across this issue and even better found a solution?

I'm curious to learn about your energy gathering method: How do you
extract node power using IPMI using FreeIMPI (or some other toolset), and
how do you configure Slurm for this?

In our cluster I select a Dell node where I obtain this IPMI power reading
from the BMC using a FreeIMPI tool:

> $ ipmi-dcmi -D LAN_2_0 --username=root --password=<secret> --hostname=c190b --get-system-power-statistics
> Current Power : 151 Watts
> Minimum Power over sampling duration : 6 watts
> Maximum Power over sampling duration : 293 watts
> Average Power over sampling duration : 153 watts
> Time Stamp : 08/29/2023 - 08:54:03
> Statistics reporting time period : 1000 milliseconds
> Power Measurement : Active

However, the node's iDRAC BMC web GUI presents a somewhat different
reading, which I assume must be reliable: 168 W.

I'm also using the Slurm with
AcctGatherEnergyType=acct_gather_energy/rapl, see [1]. With RAPL and
"scontrol show node c190" Slurm reports CurrentWatts=177 which just
measures CPU+DIMM power.

Thanks for sharing any insights.

Best regards,
Ole

[1]
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#power-monitoring-and-management

[Hagdorn, Magnus Karl Moritz]

Hi Ole,

On Tue, 2023-08-29 at 11:08 +0200, Ole Holm Nielsen wrote:
>
> I'm curious to learn about your energy gathering method:  How do you
> extract node power using IPMI using FreeIMPI (or some other toolset),
> and
> how do you configure Slurm for this?
>

We are using the SLURM plugin which is enabled using
AcctGatherEnergyType=acct_gather_energy/ipmi

https://slurm.schedmd.com/acct_gather.conf.html#SECTION_acct_gather_energy/IPMI


Cheers
magnus

[Ole Holm Nielsen]

Hi Magnus,

On 29-08-2023 13:56, Hagdorn, Magnus Karl Moritz wrote:
>> I'm curious to learn about your energy gathering method:  How do you
>> extract node power using IPMI using FreeIMPI (or some other toolset),
>> and
>> how do you configure Slurm for this?
>>
>
> We are using the SLURM plugin which is enabled using
> AcctGatherEnergyType=acct_gather_energy/ipmi
>
> https://slurm.schedmd.com/acct_gather.conf.html#SECTION_acct_gather_energy/IPMI

Yes, but can you share the details of which parameters you configure in
this plugin so that you can extract node power? This doesn't seem
obvious to me.

Thanks,
Ole

[Hagdorn, Magnus Karl Moritz]

Hi Ole,

On Tue, 2023-08-29 at 20:56 +0200, Ole Holm Nielsen wrote:
> Yes, but can you share the details of which parameters you configure
> in
> this plugin so that you can extract node power?  This doesn't seem
> obvious to me.

not much needs configuring. We have

EnergyIPMIFrequency=10
EnergyIPMICalcAdjustment=yes
EnergyIPMIPowerSensors=Node=DCMI_ENHANCED

in the acct_gather.conf.
Regards

[Ole Holm Nielsen]

Hi Magnus,

On 8/30/23 10:12, Hagdorn, Magnus Karl Moritz wrote:
>> Yes, but can you share the details of which parameters you configure
>> in
>> this plugin so that you can extract node power?  This doesn't seem
>> obvious to me.
>
> not much needs configuring. We have
>
> EnergyIPMIFrequency=10
> EnergyIPMICalcAdjustment=yes
> EnergyIPMIPowerSensors=Node=DCMI_ENHANCED
>
> in the acct_gather.conf.

This is a very useful example! I guess that you have also defined
EnergyIPMIUsername and EnergyIPMIPassword in acct_gather.conf? How is the
EnergyIPMIPassword protected from normal users if the
/etc/slurm/acct_gather.conf file exists?

An EnergyIPMIFrequency of 10 seconds sounds like it could put a high load
on the BMC and the server?

I have never tested IPMI DCMI_ENHANCED commands. Do you have some
FreeIMPI commands which can be used to verify the basic IPMI DCMI_ENHANCED
functionality?

Thanks,
Ole

[Hagdorn, Magnus Karl Moritz]

On Wed, 2023-08-30 at 10:38 +0200, Ole Holm Nielsen wrote:
> This is a very useful example!  I guess that you have also defined
> EnergyIPMIUsername and EnergyIPMIPassword in acct_gather.conf?  How
> is the
> EnergyIPMIPassword protected from normal users if the
> /etc/slurm/acct_gather.conf file exists?
>

it talks to the BMC via to OS, so no password/user required.

> An EnergyIPMIFrequency of 10 seconds sounds like it could put a high
> load
> on the BMC and the server?
>

that might be my problem - I haven't checked that.

> I have never tested IPMI DCMI_ENHANCED commands.  Do you have some
> FreeIMPI commands which can be used to verify the basic IPMI
> DCMI_ENHANCED
> functionality?
>

I checked the spec sheet of our BMC which suggested that it should be
able to do DCMI_ENHANCED

[Ole Holm Nielsen]

Hi Magnus,

On 8/30/23 11:17, Hagdorn, Magnus Karl Moritz wrote:
> On Wed, 2023-08-30 at 10:38 +0200, Ole Holm Nielsen wrote:
>> This is a very useful example!  I guess that you have also defined
>> EnergyIPMIUsername and EnergyIPMIPassword in acct_gather.conf?  How
>> is the
>> EnergyIPMIPassword protected from normal users if the
>> /etc/slurm/acct_gather.conf file exists?
>>
> it talks to the BMC via to OS, so no password/user required.

Ah, of course, the slurmd on your nodes can do local IPMI commands :-)

>> An EnergyIPMIFrequency of 10 seconds sounds like it could put a high
>> load
>> on the BMC and the server?
>>
> that might be my problem - I haven't checked that.

Maybe this could be a problem. It's anyway better not to have "OS jitter"
in HPC compute nodes by having system tasks executing too frequently.

>> I have never tested IPMI DCMI_ENHANCED commands.  Do you have some
>> FreeIMPI commands which can be used to verify the basic IPMI
>> DCMI_ENHANCED
>> functionality?
>>
> I checked the spec sheet of our BMC which suggested that it should be
> able to do DCMI_ENHANCED

That's good to know. Our servers from Huawei don't seem to support
DCMI_ENHANCED.

The following ipmitool command works locally on a node, but I can't figure
out the corresponding command to use with FreeIPMI.

# ipmitool dcmi power reading

Instantaneous power reading: 689 Watts
Minimum during sampling period: 19 Watts
Maximum during sampling period: 905 Watts
Average power reading over sample period: 682 Watts
IPMI timestamp: Wed Aug 30 09:35:28 2023
Sampling period: 00000001 Seconds.
Power reading state is: activated


Best regards,
Ole