PAPI_add_event error in SGI Altix

[Isaac Sánchez Barrera]

Hi,

I was trying to access some counters using the Extrae library but I got
an error when adding the counters. After getting in touch with the
developers of Extrae, it seems to be an issue with PAPI.

The system is an SGI Altix UV100 with 12 Intel Xeon E7-8837 sockets. It
runs SUSE Enterprise Linux 11 with Linux kernel 2.6.32.36-0.5-default.

The compiler is GCC 5.1.0. I have compiled PAPI 5.5.1.0 configured with
the following command line (I attach you the config.log):
./configure --prefix=${HOME}/compiled/papi CC=gcc F77=gfortran

I would like to get PAPI_TOT_INS and PAPI_TOT_CYC, which are available
on the system as the output of papi_avail shows (see attached
papi_avail.txt). However, a simple test application (test.c) gives the
following output:

PAPI_add_event error -1: Invalid argument


I have later tried doing `make test` in PAPI (should have done it
before), which fails with an error like the one shown in thread
<https://lists.eecs.utk.edu/pipermail/ptools-perfapi/2016-March/003919.html>.
To my surprise, however, the output of running `perf stat -v -e cycles
/bin/ls` (as proposed in the follow-up
<https://lists.eecs.utk.edu/pipermail/ptools-perfapi/2016-March/003923.html>)
does not give a message saying "Old kernel, cannot exclude guest or host
samples.", unlike in that case.

Do you have any indication of what could be happening or what I should
try next?


Thanks in advance.
Regards,
--
Isaac Sánchez Barrera


http://bsc.es/disclaimer

[Philip Mucci]

HI Isaac,

Sometimes the machines have their own libpfm installed. This very well could be what’s happening here.

Send us the ‘ldd’ of ctests/zero. Also, please try and run the ‘self’ test program from the libpfm that is included in the library.

Regards

> --
> You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
> To post to this group, send email to ptools-...@icl.utk.edu.
> Visit this group at https://groups.google.com/a/icl.utk.edu/group/ptools-perfapi/.
> <config.log><papi_avail.txt><test.c>

[Isaac Sánchez Barrera]

Hi Philip,

El 13/03/17 a las 23:37, Philip Mucci escribió:

> HI Isaac,
>
> Sometimes the machines have their own libpfm installed. This very well could be what’s happening here.

Not sure about that, `whereis libpfm` shows nothing by default (and no
path in LD_LIBRARY_PATH and LIBRARY_PATH contains any libpfm binary)

>
> Send us the ‘ldd’ of ctests/zero.

Here it is:
linux-vdso.so.1 => (0x00007fff86bff000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe389678000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe3899d6000)

> Also, please try and run the ‘self’ test program from the libpfm that is included in the library.

I see there are both libpfm4 and libpfm-3.y. The output of
`libpfm4/perf_examples/self` (after compiling it, because the example
binaries are not built doing the make for PAPI) is

self: cannot open event 0: Invalid argument


For `libpfm-3.y/examples_v3.x/self` (after compiling both the library
and the examples) the output is

sycall base 286
major version 3
minor version 0
Intel architected PMU: version=3 num_gen=4 num_fixed=3 pmc=5 pmd=7
cannot create session Bad address


Thanks for your help!

[Philip Mucci]

>
> On Mar 14, 2017, at 3:08 AM, Isaac Sánchez Barrera <isaac....@bsc.es> wrote:
>
> Hi Philip,
>
> El 13/03/17 a las 23:37, Philip Mucci escribió:
>> HI Isaac,
>>
>> Sometimes the machines have their own libpfm installed. This very well could be what’s happening here.
>
> Not sure about that, `whereis libpfm` shows nothing by default (and no path in LD_LIBRARY_PATH and LIBRARY_PATH contains any libpfm binary)

Libpfm is installed by papi under the prefix you specify.

>
>>
>> Send us the ‘ldd’ of ctests/zero.
>
> Here it is:
> linux-vdso.so.1 => (0x00007fff86bff000)
> libc.so.6 => /lib64/libc.so.6 (0x00007fe389678000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fe3899d6000)

Ok, so you are linking with the static version of papi, libpapi.a.

>
>
>> Also, please try and run the ‘self’ test program from the libpfm that is included in the library.
>
> I see there are both libpfm4 and libpfm-3.y. The output of `libpfm4/perf_examples/self` (after compiling it, because the example binaries are not built doing the make for PAPI) is
>
> self: cannot open event 0: Invalid argument

This is getting closer to finding the problem. Please do the following:

git clone https://git.code.sf.net/p/perfmon2/libpfm4 perfmon2-libpfm4

cd perfmon2-libpfm4
make all

cd examples
./check_events

cd perf_examples
./self

If that fails, then we have an event encoding issue with the version of kernel on the system. Then run the above commands but first do the following and send us the output.

export LIBPFM_DEBUG_STDOUT=yes
export LIBPFM_DEBUG=yes
export LIBPFM_VERBOSE=yes

I would have thought your distribution would come with a libpfm and libpapi provided by the package manager. PAPI team, could this be the dreaded ‘exclude_guest’ bug?

Phil

[Vince Weaver]

A few other things to try.

What does "utils/papi_component_avail" say?

Also, since you are using a vendor kernel, you probably want to pass
--with-assumed-kernel=2.6.34
or a higher version because of the backported fixes.

The problem could very well be the host/guest bit issue even though perf
isn't warning about it. I forget if we ever put a fix into that for PAPI.
The way perf works around it is to probe for support each time you run it.

Vince

[Isaac Sánchez Barrera]

Hi Philip,

>>
>> Not sure about that, `whereis libpfm` shows nothing by default (and no path in LD_LIBRARY_PATH and LIBRARY_PATH contains any libpfm binary)
>
>
> Libpfm is installed by papi under the prefix you specify.
>

I meant without considering the one installed by papi. Sorry for not
being precise.

>
> This is getting closer to finding the problem. Please do the following:
>
> git clone https://git.code.sf.net/p/perfmon2/libpfm4 perfmon2-libpfm4
>
> cd perfmon2-libpfm4
> make all
>
> cd examples
> ./check_events
>
> cd perf_examples
> ./self
>
> If that fails, then we have an event encoding issue with the version of kernel on the system. Then run the above commands but first do the following and send us the output.
>
> export LIBPFM_DEBUG_STDOUT=yes
> export LIBPFM_DEBUG=yes
> export LIBPFM_VERBOSE=yes
>

The `check_events` binary runs without problem (or so I think), the
`self` fails as in the previous email. I attach the outputs of both (I
had to change the "yes" to 1, otherwise the verbose/debug outputs were
not printing).

> I would have thought your distribution would come with a libpfm and libpapi provided by the package manager. PAPI team, could this be the dreaded ‘exclude_guest’ bug?
>

It's quite possible that both libraries are available from the package
manager. If I cannot make it work, I will ask the sysadmins to install
them, but I might have to wait for some (long) time until they can help me.


I will reply to Vince in different message.

Thanks a lot for your help!

[Isaac Sánchez Barrera]

Hi Vince,

>
> A few other things to try.
>
> What does "utils/papi_component_avail" say?

Available components and hardware information.
--------------------------------------------------------------------------------
PAPI Version : 5.5.1.0
Vendor string and code : GenuineIntel (1)
Model string and code : Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (47)
CPU Revision : 2.000000
CPUID Info : Family: 6 Model: 47 Stepping: 2
CPU Max Megahertz : 2666
CPU Min Megahertz : 2666
Hdw Threads per core : 1
Cores per Socket : 8
Sockets : 12
NUMA Nodes : 12
CPUs per Node : 8
Total CPUs : 96
Running in a VM : no
Number Hardware Counters : 7
Max Multiplex Counters : 32
--------------------------------------------------------------------------------

Compiled-in components:
Name: perf_event Linux perf_event CPU counters
Name: perf_event_uncore Linux perf_event CPU uncore and northbridge
\-> Disabled: Insufficient permissions for uncore access. Set
/proc/sys/kernel/perf_event_paranoid to 0 or run as root.

Active components:
Name: perf_event Linux perf_event CPU counters
Native: 179, Preset: 58, Counters: 7
PMU's supported: ix86arch, perf,
wsm_dp, perf_raw


--------------------------------------------------------------------------------
component.c PASSED

>
> Also, since you are using a vendor kernel, you probably want to pass
> --with-assumed-kernel=2.6.34
> or a higher version because of the backported fixes.

No luck with that specific assumed kernel. Tried also with 2.6.36 just
in case, but `make test` has failed equally. Maybe I should ask the
sysadmins to try and install the vendor-provided versions of libpfm and
libpapi, as Philip says.

>
> The problem could very well be the host/guest bit issue even though perf
> isn't warning about it. I forget if we ever put a fix into that for PAPI.
> The way perf works around it is to probe for support each time you run it.
>
> Vince
>

Thanks for your help.

[Isaac Sánchez Barrera]

Hi Vince & Philip,

TL;DR: After many trial-and-error steps, I got PAPI to work. I am using
version 4.4.0, and it seems to be the only one of all I have tried to
compile and work correctly.

The sysadmins installed libpapi from the SLES 11 repositories
(libpapi-3.7.2-0.10.1, papi-devel-3.7.2-0.10.1, papi-3.7.2-0.10.1), but
it didn't work corretly. It only detected 3 counters (including
PAPI_TOT_INS and PAPI_TOT_CYC), but they all returned zero and had an
output saying

PAPI Error:
pfm_find_full_event(RETIRED_MISPREDICTED_BRANCH_INSTRUCTIONS,0x7fffa09bd3f0):
event not found.


According to the software releases on the PAPI website, 4.1.0 is the
first version to have support for Westmere, but it does not compile
fine. I tried with 4.1.1, but had to add -Wno-unused-but-set-variable to
DBG in libpfm-3.y/config.mk to make it compile. The output was the same
as for 3.7.2 from the repo (which I could not compile manually myself).
I later tried with 4.1.2, same problems. Then I went straight to the
latest 4.x release (4.4.0), and voilà! Everything worked fine with no
hassle. Maybe the reason is that the processors are Westmere EX?

The output of `ctests/zero`:

Test case 0: start, stop.
-----------------------------------------------
Default domain is: 1 (PAPI_DOM_USER)
Default granularity is: 1 (PAPI_GRN_THR)
Using 20000000 iterations of c += a*b
-------------------------------------------------------------------------
Test type : 1
PAPI_FP_INS : 60000028
PAPI_TOT_CYC : 200020238
Real usec : 73533
Real cycles : 196041287
Virt usec : 75182
Virt cycles : 200437878
-------------------------------------------------------------------------
Verification: PAPI_TOT_CYC should be roughly real_cycles
zero.c PASSED


The output of `libpfm4/perf_examples/self`:

INITIAL: 15,773 PERF_COUNT_HW_CPU_CYCLES (0.00% scaling,
raw=15,773, ena=11,538, run=11,538)
INITIAL: 21,688 PERF_COUNT_HW_INSTRUCTIONS (0.00%
scaling, raw=21,688, ena=62,870, run=62,870)
Final counts:
FINAL: 26,654,801,768 PERF_COUNT_HW_CPU_CYCLES (0.00% scaling,
raw=26,654,801,768, ena=9,997,788,016, run=9,997,788,016)
FINAL: 39,948,683,352 PERF_COUNT_HW_INSTRUCTIONS (0.00% scaling,
raw=39,948,683,352, ena=9,997,783,658, run=9,997,783,658)


For `utils/papi_component_avail`:

Available components and hardware information.
--------------------------------------------------------------------------------

PAPI Version : 4.4.0.0

Vendor string and code : GenuineIntel (1)
Model string and code : Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz (47)
CPU Revision : 2.000000
CPUID Info : Family: 6 Model: 47 Stepping: 2

CPU Megahertz : 2666.887939
CPU Clock Megahertz : 2666

Hdw Threads per core : 1
Cores per Socket : 8

NUMA Nodes : 12
CPUs per Node : 8
Total CPUs : 96

Number Hardware Counters : 7
Max Multiplex Counters : 32
--------------------------------------------------------------------------------

Name: perf_events.c

--------------------------------------------------------------------------------
component.c PASSED


If you want the outputs for `papi_avail` or for the tests using version
4.1.1/4.1.2 you can ask for them and I will attach them here.

[Vince Weaver]

On Wed, 15 Mar 2017, Isaac Sánchez Barrera wrote:

> TL;DR: After many trial-and-error steps, I got PAPI to work. I am using
> version 4.4.0, and it seems to be the only one of all I have tried to compile
> and work correctly.

This sounds very much like it still could be the :mg/:mh problem, as I
think support for it was added to libpfm around release 4.4.0 (which is
coincidentally around when PAPI 4.4.0 came out)

As a test, you can try configuring with
--with-pe-incdir=/usr/include/linux
and see if that makes a difference, but I'm not sure how well supported
that is with recent PAPI.

I'll look into getting a proper fix for this committed.

Vince

[Vince Weaver]

I've committed a change that might fix this problem to papi-git.
Try checking out the most recent PAPI tree and see if it works (no
special configure options necessary).

Vince

[Isaac Sánchez Barrera]

Thanks, Vince. I'll try that tomorrow and tell you the outcome, the
machine is down for scheduled filesystem maintenance until 17:00 UTC.

El 16/03/17 a las 06:34, Vince Weaver escribió:

[Isaac Sánchez Barrera]

Hi Vince,

El 16/03/17 a las 06:34, Vince Weaver escribió:
>

> I've committed a change that might fix this problem to papi-git.
> Try checking out the most recent PAPI tree and see if it works (no
> special configure options necessary).

It seems to work OK:

ctests/zero
Test case 0: start, stop.
-----------------------------------------------
Default domain is: 1 (PAPI_DOM_USER)
Default granularity is: 1 (PAPI_GRN_THR)
Using 20000000 iterations of c += a*b
-------------------------------------------------------------------------
Test type : 1

PAPI_FP_INS : 60000007
PAPI_TOT_CYC : 200014952
Real usec : 73265
Real cycles : 195323934
Virt usec : 75115
Virt cycles : 200253924

-------------------------------------------------------------------------
Verification: PAPI_TOT_CYC should be roughly real_cycles

NOTE: Not true if dynamic frequency scaling is enabled.
Verification: PAPI_FP_INS should be roughly 40000000
PAPI_FP_INS Error of 50.00%
zero.c WARNING
Line # 134
Warning: FLOPS validation

zero.c PASSED with WARNING


I think I remember reading somewhere (the PAPI wiki maybe?) that the FP
counters were "broken" in Intel (and disabled since Haswell), so this
should be no problem. Also, the readings match those from papi-4.4.0.

Thanks a lot for fixing this issue!