Question about RAPL CPU Measures

[Khaled Z Mahmoud]

Hi,

I am using PAPI to read energy values periodically. As I understand, it basically reads the MSR energy counter. 

This values should be increasing since it is accumulating. However, I am seeing values at time t2 less than values in time t1.

Is this normal?

--

Regards,

Khaled Mahmoud

[Vince Weaver]

On Tue, 17 Apr 2018, Khaled Z Mahmoud wrote:

> I am using PAPI to read energy values periodically. As I understand, it
> basically reads the MSR energy counter.
>
> This values should be increasing since it is accumulating. However, I am

> seeing values at time *t2 *less than values in time *t1*.

how periodically are you reading things? I think the RAPL counters will
overflow and wrap around. It depends on how much energy you are using,
but I think it can happen fairly quickly, on the order of a minute or so.

Vince

[John McCalpin]

The wraparound time for the RAPL counters depends on the particular processor.

The counters are only 32 bits wide, and unlike the performance counters, there are no provisions for generating interrupts on overflows, so wraparound is the normal mode of operation.

On a Xeon Platinum 8160, the package energy counter increments once per 1/16384 Joules (~0.00006104 Joules), so the maximum value (2^32-1) corresponds to 262144 Joules.
At a maximum average power of 150 Watts, this counter can overflow in about 1748 seconds.
Other processor models may use different energy units, and they certainly have different maximum average power, so the wraparound time needs to be computed for each processor model.

I have not seen any processors with a wraparound time of less than about 500 seconds, but I have only reviewed the ones that are installed at my site.

--
John D. McCalpin, Ph.D.

Texas Advanced Computing Center
University of Texas at Austin
https://www.tacc.utexas.edu/about/directory/john-mccalpin

--
You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
To post to this group, send email to ptools-...@icl.utk.edu.
Visit this group at https://groups.google.com/a/icl.utk.edu/group/ptools-perfapi/.

[Khaled Z Mahmoud]

I have tried measuring the energy every millisecond and every 10 microseconds as well. Execution does not exceed 30 seconds.

My problem gets worse when adding more threads .... I am running a very simple program with each thread calculating counts (Just Looping)

I am speculating the following could have caused this: The code reads time timestamp and energy in two difference instructions. Although these instructions are consecutive, time could pass after reading the energy. For example, a thread is scheduled out of the core and then comes back. Thus, the energy reading will be related to an earlier time instant.  Many threads are working on the core at the same time.

I am not sure if that could be the case from the following sample

Here is a sample of the energy readings with time stamps.

Time       Value          Difference with previous Reading

376860   17.819000  0.028000
2376861  17.846000  0.027000
2376862  17.835000  -0.011000
2376863  17.859000  0.024000
2376864  17.882000  0.023000
2376865  17.907000  0.025000
2376866  17.931000  0.024000
2376867  17.953000  0.022000
2376868  17.978000  0.025000
2376869  18.001000  0.023000
2376870  18.025000  0.024000
2376871  18.051000  0.026000
2376872  18.051000  0.000000
2376873  18.039000  -0.012000
2376874  18.062000  0.023000
2376875  18.083000  0.021000
2376876  18.083000  0.000000
2376877  18.104000  0.021000
2376878  18.126000  0.022000
2376879  18.147000  0.021000
2376880  18.170000  0.023000

    To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfapi+unsubscribe@icl.utk.edu.

    To post to this group, send email to ptools-...@icl.utk.edu.
    Visit this group at https://groups.google.com/a/icl.utk.edu/group/ptools-perfapi/.

--

Regards,

Khaled Mahmoud

[Vince Weaver]

On Tue, 17 Apr 2018, Khaled Z Mahmoud wrote:

> I have tried measuring the energy every millisecond and every 10
> microseconds as well. Execution does not exceed 30 seconds.
> My problem gets worse when adding more threads .... I am running a very
> simple program with each thread calculating counts (Just Looping)

RAPL readings only update around every 1ms so there's usually no need to
read them much more often than that.

Also, the measurements are per socket so unless you are on a multi-socket
system there's no reason to have more than one thread reading the values.

Vince

[小光]

I tried rapl_plot.c(https://github.com/tpatki/papi-rapl/blob/master/src/components/rapl/utils/rapl_plot.c)

and I can get PACKAGE_ENERGY_PACKAGE0, PACKAGE_ENERGY_PACKAGE1, DRAM_ENERGY_PACKAGE0,DRAM_ENERGY_PACKAGE1.

My question is, I have 2 sockets, 24 cores.For example, how to use papi to measure socket 0 on CPU 0-11?

The rapl_plot.c measures both the 2 sockets ? or just measure the socket 0?  The result of PACKAGE_ENERGY_PACKAGE0 include of the 2 sockets? 

[Vince Weaver]

On Thu, 24 May 2018, 小光 wrote:

> I tried
> rapl_plot.c(https://github.com/tpatki/papi-rapl/blob/master/src/components/rapl/utils/rapl_plot.c)
> and I can get PACKAGE_ENERGY_PACKAGE0, PACKAGE_ENERGY_PACKAGE1,
> DRAM_ENERGY_PACKAGE0,DRAM_ENERGY_PACKAGE1.
>
> My question is, I have 2 sockets, 24 cores.For example, how to use papi to
> measure socket 0 on CPU 0-11?
>
> The rapl_plot.c measures both the 2 sockets ? or just measure the socket 0?
> The result of PACKAGE_ENERGY_PACKAGE0 include of the 2 sockets?

The PACKAGE_ENERGY_PACKAGE0 event measures package (socket) 0, and
the PACKAGE_ENERGY_PACKAGE1 event measures package (socket) 1.

Vince

[Allen]

That makes sense.

Thanks.

[Allen]

I tried the code rapl_plot.c(https://github.com/tpatki/papi-rapl/blob/master/src/components/rapl/utils/rapl_plot.c) 

why there are many "0.0000,0.0"

 

...

1530581602244,total_time:23.3958,values[i]:41076660.0000,37.1,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602246,total_time:23.3970,values[i]:50720214.0000,45.8,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602247,total_time:23.3982,values[i]:0.0000,0.0,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602248,total_time:23.3994,values[i]:34362792.0000,31.0,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602249,total_time:23.4007,values[i]:42907714.0000,39.2,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602250,total_time:23.4018,values[i]:50903320.0000,46.1,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602252,total_time:23.4031,values[i]:40588378.0000,36.7,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602253,total_time:23.4043,values[i]:0.0000,0.0,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602254,total_time:23.4055,values[i]:38574218.0000,34.9,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602255,total_time:23.4067,values[i]:41015625.0000,37.5,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602256,total_time:23.4079,values[i]:43334960.0000,39.2,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602258,total_time:23.4091,values[i]:44067382.0000,39.8,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602259,total_time:23.4103,values[i]:39367675.0000,35.6,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602260,total_time:23.4115,values[i]:34301757.0000,31.0,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602261,total_time:23.4127,values[i]:42297363.0000,38.6,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602262,total_time:23.4139,values[i]:41625976.0000,37.6,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602264,total_time:23.4151,values[i]:42724609.0000,38.6,(Average Power for PACKAGE_ENERGY:PACKAGE0)

1530581602265,total_time:23.4163,values[i]:0.0000,0.0,(Average Power for PACKAGE_ENERGY:PACKAGE0) 

...

[Allen]

between PAPI_start and PAPI_stop

if usleep(1000),1ms, there are some "0.0000,0.0".

if usleep(500),there are more "0.0000,0.0".

if usleep(2000),there are no "0.0000,0.0".

[John McCalpin]

According to Section 14.9.3 of Volume 3 of the Intel Architectures Software Developer’s Manual (document 325384, revision 067, May 2018) the RAPL counters are updated at intervals of approximately 1 millisecond.

While we don’t know anything about how this “approximately 1 millisecond” is determined internally, is not surprising that measurements with an intended spacing of approximately 1 millisecond will read the same value twice on occasion.   One would expect measurements with intervals much shorter than 1 millisecond to have lots of repeats, and measurements noticeably larger than 1 millisecond to have no repeats.  This seems to match your observations.

 

-- 

John D. McCalpin, Ph.D.

Texas Advanced Computing Center

University of Texas at Austin

https://www.tacc.utexas.edu/about/directory/john-mccalpin