Accessing native events for top-down level 2 on Sapphire Rapids

[Ian Lumsden]

Hello,

I am trying to update the top-down service in LLNL's Caliper tool to support Sapphire Rapids. Currently, this service uses PAPI to access counters and native events. I've managed to work out how to use PAPI to access the necessary native events for the first level top-down metrics (i.e., frontend bound, backend bound, bad speculation, and retiring). However, I cannot figure out how to access the following native events, which I need for the second level top-down metrics:

• topdown-fetch-lat
• topdown-br-mispredict
• topdown-mem-bound
• topdown-heavy-ops

How can I access these events through PAPI?

In case it helps, I've confirmed that the filesystem entries for these events exist in /sys/devices/cpu/events, and I've confirmed that these events do not show up in papi_native_avail. I've also found the event and umask values for these events in the Linux kernel: https://github.com/torvalds/linux/blob/b1bc554e009e3aeed7e4cfd2e717c7a34a98c683/arch/x86/events/intel/core.c#L472

Thanks,

Ian Lumsden

Global Computing Laboratory

Department of Electrical Engineering and Computer Science

University of Tennessee, Knoxville

[Ian Lumsden]

I have a bit of an update on this.

I managed to access these counters by using their raw codes, which I got from this file in the Linux kernel. However, I've now run into a pair of other issues. First, I've found that I cannot access any of the counters I need when using PAPI's multiplexing. Second, the values I'm getting for these counters are all gigantic numbers (on the order of billions or larger) with only the least significant digits (e.g., through the low thousands) differing. As a result, my top-down calculations are ending up being roughly 0.25 for each of the level 1 metrics.

How can I work around these issues?

In case it helps, these are the specific metrics (in PAPI) format that I'm using:

• perf::slots
  
• perf::topdown-retiring
• perf::topdown-bad-spec
• perf::topdown-be-bound
• INT_MISC:UOP_DROPPING
• perf_raw::r8400 (raw version of topdown-heavy-ops)
• perf_raw::r8500 (raw version of topdown-br-mispredict)
• perf_raw::r8600 (raw version of topdown-fetch-lat)
• perf_raw::r8700 (raw version of topdown-mem-bound)

Also, here are two examples of the values I'm getting. The first image is the output of `papi_command_line`. The second image is a printout of a Caliper trace showing the values we get for some of these metrics.

Thanks,

Ian Lumsden

[Vince Weaver]

you seem to have a variety of different issues here.

For one, PAPI gets its event values from libpfm4, so if the events aren't
appearing in PAPI then we need to get that fixed upstream in libpfm4.
I don't have access to a Sapphire Rapids machine, but it looks like a few
topdown events should show up in papi_native_avail, for example
TOPDOWN:SLOTS
TOPDOWN:BACKEND_BOUND_SLOTS
TOPDOWN:MEMORY_BOUND_SLOTS
and similar. Do those events appear for you, and can you add them?

What do you mean that you can't access the values when multiplexing? It
won't let you add the events, or do you get some sort of error?

The oddly large values from papi_command_line shouldn't be happening,
that's a bit odd. What version of PAPI are you using? Does this only
happen when you are multiplexing by having lots of events, or does it
happen on the topdown events if you run them one at a time?

Vince Weaver
vincent...@maine.edu

On Tue, 23 Jul 2024, Ian Lumsden wrote:

> I have a bit of an update on this.
>
> I managed to access these counters by using their raw codes, which I got
> from this file

> <https://github.com/torvalds/linux/blob/e9680017b2dc8686a908ea1b51941a91b6da9f1d/arch/x86/events/intel/core.c#L480>

> in the Linux kernel. However, I've now run into a pair of other issues.
> First, I've found that I cannot access any of the counters I need when
> using PAPI's multiplexing. Second, the values I'm getting for these
> counters are all gigantic numbers (on the order of billions or larger) with
> only the least significant digits (e.g., through the low thousands)
> differing. As a result, my top-down calculations are ending up being
> roughly 0.25 for each of the level 1 metrics.
>
> How can I work around these issues?
>
> In case it helps, these are the specific metrics (in PAPI) format that I'm
> using:
>

> - perf::slots
> - perf::topdown-retiring
> - perf::topdown-bad-spec
> - perf::topdown-be-bound
> - INT_MISC:UOP_DROPPING
> - perf_raw::r8400 (raw version of topdown-heavy-ops)
> - perf_raw::r8500 (raw version of topdown-br-mispredict)
> - perf_raw::r8600 (raw version of topdown-fetch-lat)
> - perf_raw::r8700 (raw version of topdown-mem-bound)

>
> Also, here are two examples of the values I'm getting. The first image is
> the output of `papi_command_line`. The second image is a printout of a
> Caliper trace showing the values we get for some of these metrics.
>
> Thanks,
>
> Ian Lumsden
>
> On Thursday, July 18, 2024 at 12:35:11 PM UTC-7 Ian Lumsden wrote:
>
> > Hello,
> >
> > I am trying to update the top-down service in LLNL's Caliper tool to
> > support Sapphire Rapids. Currently, this service uses PAPI to access
> > counters and native events. I've managed to work out how to use PAPI to
> > access the necessary native events for the first level top-down metrics
> > (i.e., frontend bound, backend bound, bad speculation, and retiring).
> > However, I cannot figure out how to access the following native events,
> > which I need for the second level top-down metrics:
> >

> > - topdown-fetch-lat
> > - topdown-br-mispredict
> > - topdown-mem-bound
> > - topdown-heavy-ops

> >
> > How can I access these events through PAPI?
> >
> > In case it helps, I've confirmed that the filesystem entries for these
> > events exist in /sys/devices/cpu/events, and I've confirmed that these

> > events do *not* show up in papi_native_avail. I've also found the event

> > and umask values for these events in the Linux kernel:
> > https://github.com/torvalds/linux/blob/b1bc554e009e3aeed7e4cfd2e717c7a34a98c683/arch/x86/events/intel/core.c#L472
> >
> > Thanks,
> >
> > Ian Lumsden
> >

> > *Global Computing Laboratory <https://globalcomputing.group/>*

> > Department of Electrical Engineering and Computer Science
> > University of Tennessee, Knoxville
> >
>

> --
> You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
> To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/6100b8a1-f196-4eea-a186-17d2a8f42c50n%40icl.utk.edu.
>

Vince Weaver
vincent...@maine.edu
Associate Professor, Electrical and Computer Engineering
http://web.eece.maine.edu/~vweaver/

[Ian Lumsden]

Hi Vince,

I can certainly forward the discussion about the named metrics to libpfm, but I can get around that for the time being.

Regarding the higher level metrics starting with `TOPDOWN:`, I can access those perfectly fine. However, the equations for the derived top-down calculations (from Intel's perfmon repo) are all in terms of the lower-level events (e.g., topdown-be-bound, and topdown-mem-bound).

Regarding the multiplexing, I'm not entirely sure what the error is because I'm deferring all the use of PAPI to Caliper's PAPI service. From what I understand, the multiplexing is enabled on an EventSet here. I currently have this block of code commented out for the purposes of testing. After enabling multiplexing, the events I listed in my last message are added to the EventSet using `papi_add_events`. When I run this, I get the error from these lines, saying that only 1 metric was successfully added to the EventSet (which I assume to be `perf::slots`). I do not get this error with the multiplexing commented out.

I am using PAPI 5.6.0.0, which is the system-wide install on the Dane system at LLNL. Dane also has an LMOD module for PAPI 6.0.0.1, but I find that version cannot recognize any of the "topdown" counters.

Finally, I was wondering if the version of the Linux kernel could cause issues. I just found out that the kernel on this system is 4.18, and these counters weren't added until 5.10 (for L1 top-down counters) and 5.12 (for L2 top-down counters) respectively (based on this file in the kernel). Despite this, I do see filesystem entries matching these names in `/sys/devices/cpu/events`.

Thanks for the help,

Ian Lumsden

[Heike Jagode]

Both PAPI versions you are using are quite outdated: PAPI 6.5 (released December 2017) and PAPI 6.0.0.1 (released Spring 2020). It might be worth installing the latest PAPI version in your home directory.

With the latest PAPI version running on a Sapphire Rapids, in addition to the perf topdown events (slots, topdown-retiring, topdown-bad-spec, topdown-fe-bound, topdown-be-bound), I am able to monitor two additional events from your list of events that you can't access:
+++ topdown-mem-bound
+++ topdown-br-mispredict

See output of papi_command_line utility:

$> ./papi_command_line TOPDOWN:MEMORY_BOUND_SLOTS TOPDOWN:BR_MISPREDICT_SLOTS TOPDOWN:BACKEND_BOUND_SLOTS TOPDOWN:BAD_SPEC_SLOTS TOPDOWN:SLOTS   TOPDOWN:SLOTS_P

Successfully added: TOPDOWN:MEMORY_BOUND_SLOTS
Successfully added: TOPDOWN:BR_MISPREDICT_SLOTS
Successfully added: TOPDOWN:BACKEND_BOUND_SLOTS
Successfully added: TOPDOWN:BAD_SPEC_SLOTS
Successfully added: TOPDOWN:SLOTS
Successfully added: TOPDOWN:SLOTS_P

TOPDOWN:MEMORY_BOUND_SLOTS : 0
TOPDOWN:BR_MISPREDICT_SLOTS : 0
TOPDOWN:BACKEND_BOUND_SLOTS : 0
TOPDOWN:BAD_SPEC_SLOTS : 36576
TOPDOWN:SLOTS : 962187834
TOPDOWN:SLOTS_P : 962187918


See snippet of papi_native_avail for details about these topdown events:

| TOPDOWN                                                                      |
|            Topdown events.                                                   |
|     :BACKEND_BOUND_SLOTS                                                     |
|            TMA slots where no uops were being issued due to lack of back-end |
|            resources.                                                        |
|     :BAD_SPEC_SLOTS                                                          |
|            TMA slots wasted due to incorrect speculations.                   |
|     :BR_MISPREDICT_SLOTS                                                     |
|            TMA slots wasted due to incorrect speculation by branch mispredict|
|            ions                                                              |
|     :MEMORY_BOUND_SLOTS                                                      |
|            TBD                                                               |
|     :SLOTS                                                                   |
|            TMA slots available for an unhalted logical processor. Fixed count|
|            er - architectural event                                          |
|     :SLOTS_P                                                                 |
|            TMA slots available for an unhalted logical processor. General cou|
|            nter - architectural event                                        |


Thanks,
Heike

To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/40ae5b37-1c0c-44ba-90c2-b1c5cc1c9a3an%40icl.utk.edu.

--

__________________________________________
Heike Jagode, Ph.D., Research Associate Professor

Innovative Computing Laboratory (ICL)

University of Tennessee Knoxville

http://icl.utk.edu/~jagode/

[Ian Lumsden]

Hi Heike,

I tried with a Spack-installed PAPI 7.1.0, and it didn't help. I also tried on two different Sapphire Rapids systems in case it was something with the specific system I was testing on. I got the same general results on both systems.

Do you have any other suggestions?

Thanks,

Ian Lumsden

[Heike Jagode]

Can you be more specific, please? What exactly didn't help? Were you unable to install PAPI at all, and if so, what error did you encounter?

If you followed the steps on the "How to download and install PAPI" wiki page, which part is not working for you? (See steps also below):

git clone https://github.com/icl-utk-edu/papi.git
./configure --prefix=$PWD/install
make && make install
cd install/bin
./papi_component_avail

Thanks,
Heike

[Ian Lumsden]

First, I tried to install PAPI 7.1.0 using Spack. Then, I ran two tests:

1. The RAJA Performance Suite instrumented with a custom build of Caliper that provides correct top-down calculations for SPR
2. `papi_command_line perf::slots perf::topdown-be-bound perf::topdown-fe-bound perf::topdown-retiring perf::topdown-bad-spec`

When I did this, I got the results shown in the attached image.

To clarify, this does run. It's just producing seemingly incorrect values. In particular, because the values returned by PAPI for these counters are so similar, the equations for top-down are producing roughly 0.25 for all top-down derived metrics. That's because the general equation for each of these derived metrics looks like:

Backend bound = topdown-be-bound / (topdown-be-bound + topdown-fe-bound + topdown-retiring + topdown-bad-spec)

Does this help?

- Ian Lumsden

[Heike Jagode]

Instead of using the "perf::" events, can you use the topdown events I listed in my previous email?

With the latest PAPI version, you should see them in your papi_native_avail output as well.

Again, see snippet of papi_native_avail for details about these topdown events:

| TOPDOWN                                                                      |
|            Topdown events.                                                   |
|     :BACKEND_BOUND_SLOTS                                                     |
|            TMA slots where no uops were being issued due to lack of back-end |
|            resources.                                                        |
|     :BAD_SPEC_SLOTS                                                          |
|            TMA slots wasted due to incorrect speculations.                   |
|     :BR_MISPREDICT_SLOTS                                                     |
|            TMA slots wasted due to incorrect speculation by branch mispredict|
|            ions                                                              |
|     :MEMORY_BOUND_SLOTS                                                      |
|            TBD                                                               |
|     :SLOTS                                                                   |
|            TMA slots available for an unhalted logical processor. Fixed count|
|            er - architectural event                                          |
|     :SLOTS_P                                                                 |
|            TMA slots available for an unhalted logical processor. General cou|
|            nter - architectural event                                        |

Thanks,
Heike

[Heike Jagode]

OK, great! Thanks for confirming that you can monitor the TOPDOWN events successfully.

I don't need to know all the details of what you are trying to do, but could you please clarify what exactly is preventing you from replacing the PAPI perf:: topdown events with the other PAPI TOPDOWN events?

Thanks,
Heike

On Wed, Jul 24, 2024 at 9:24 PM Lumsden, Ian <ilum...@vols.utk.edu> wrote:

To clarify, when I say "I can't use those events," I mean I can't use them for what I'm trying to do. I can successfully get their values from PAPI.

Ian Lumsden

PhD Student in High Performance Computing

Global Computing Laboratory

Department of Electrical Engineering and Computer Science

University of Tennessee, Knoxville

On Wed, Jul 24, 2024 at 6:22 PM Lumsden, Ian <ilum...@vols.utk.edu> wrote:

Unfortunately, I can't use those events. The equations for the first two levels of top-down analysis for SPR are all in terms of the "perf::" events.

Ian Lumsden

PhD Student in High Performance Computing

Global Computing Laboratory

[Lumsden, Ian]

Unfortunately, I can't use those events. The equations for the first two levels of top-down analysis for SPR are all in terms of the "perf::" events.

Ian Lumsden

PhD Student in High Performance Computing

Global Computing Laboratory

Department of Electrical Engineering and Computer Science

University of Tennessee, Knoxville

On Wed, Jul 24, 2024 at 5:18 PM Heike Jagode <jag...@icl.utk.edu> wrote:

[Lumsden, Ian]

To clarify, when I say "I can't use those events," I mean I can't use them for what I'm trying to do. I can successfully get their values from PAPI.

Ian Lumsden

PhD Student in High Performance Computing

Global Computing Laboratory

Department of Electrical Engineering and Computer Science

University of Tennessee, Knoxville

On Wed, Jul 24, 2024 at 6:22 PM Lumsden, Ian <ilum...@vols.utk.edu> wrote:

[Ian Lumsden]

The main goal of what I'm trying to do is to add support for Intel's top-down analysis (i.e., the approach from this paper) to LLNL's Caliper profiler for SPR. However, the equations for the top-down calculations can differ between processors. I found documentation for these calculations for SPR in this file from Intel's perfmon repo. These calculations for the first two levels of top-down on SPR are defined in terms of the following events:

• perf::slots
  
• perf::topdown-retiring
• perf::topdown-bad-spec
• perf::topdown-be-bound
• INT_MISC:UOP_DROPPING

• perf_raw::r8400 (raw version of topdown-heavy-ops)

• perf_raw::r8500 (raw version of topdown-br-mispredict)

• perf_raw::r8600 (raw version of topdown-fetch-lat)

• perf_raw::r8700 (raw version of topdown-mem-bound)

If I can confirm that the `TOPDOWN:` events you shared correspond to some of the events above, I'd be happy to substitute those events out. But, there are still several others that would require me to use the "perf::" events.

Thanks,

Ian

[Heike Jagode]

We had a discussion in another thread where Vince suggested disabling direct reading from the registers. I tried that, and it looks like this fixes the problem. See details below:

On Wed, Jul 24, 2024 at 7:57 PM Ian Lumsden <lumsd...@gmail.com> wrote:

To clarify, this does run. It's just producing seemingly incorrect values. In particular, because the values returned by PAPI for these counters are so similar, the equations for top-down are producing roughly 0.25 for all top-down derived metrics.

If you reconfigure and rebuild PAPI with --disable-perfevent-rdpmc, it seems to resolve the issue with the perf::slots counts getting messed up. Can you try this and see if it works for you as well?

$./papi_command_line perf::slots perf::topdown-retiring perf::topdown-bad-spec perf::topdown-fe-bound perf::topdown-be-bound

This utility lets you add events from the command line interface to see if they work.

Successfully added: perf::slots
Successfully added: perf::topdown-retiring
Successfully added: perf::topdown-bad-spec
Successfully added: perf::topdown-fe-bound
Successfully added: perf::topdown-be-bound

perf::slots : 962011164
perf::topdown-retiring : 199947470
perf::topdown-bad-spec : 0
perf::topdown-fe-bound : 0
perf::topdown-be-bound : 762063941

[Ian Lumsden]

That seems to have worked!

I'm now getting reasonable values from everything except the second level calculations under Frontend Bound. Given that everything else is looking reasonable, I suspect that's a bug in my code within Caliper. I'll look into that and let you know if my assumption is right or if there seems to be something else from the PAPI side.

Thank you so much, Heike and Vince!

- Ian Lumsden