No bandwidth STAT of likwid-perfctr on AMD Milan

[Long Qu]

Dear Likwid team,

I am using likwid for measuring the memory bandwidth and data volume of my code on the AMD Milan. I am not able to get measurement results.

Here is an example of the output:

+-------------------------------------------------+-------------+-----------+-----------+-----------+

|                      Metric                     |     Sum     |    Min    |    Max    |    Avg    |

+-------------------------------------------------+-------------+-----------+-----------+-----------+

|             Runtime (RDTSC) [s] STAT            |   8441.7297 |   65.9441 |   65.9599 |   65.9510 |

|            Runtime unhalted [s] STAT            |   9593.9686 |   71.9808 |   79.3500 |   74.9529 |

|                 Clock [MHz] STAT                | 298269.7417 | 2257.9428 | 2407.0698 | 2330.2324 |

|                     CPI STAT                    |           0 |    inf    |         0 |         0 |

| Memory bandwidth (channels 0-3) [MBytes/s] STAT |           0 |         0 |         0 |         0 |

| Memory data volume (channels 0-3) [GBytes] STAT |           0 |         0 |         0 |         0 |

+-------------------------------------------------+-------------+-----------+-----------+-----------+

When running with -V 3, I see the following:

DEBUG - [perfgroup_readGroup:871] Reading group MEM1 from /home/longq0a/test_girih/likwid_install/share/likwid/perfgroups/zen3/MEM1.txt

DEBUG - [perfmon_addEventSet:2370] Added event ACTUAL_CPU_CLOCK for counter FIXC1 to group 0

DEBUG - [perfmon_addEventSet:2370] Added event MAX_CPU_CLOCK for counter FIXC2 to group 0

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register PMC0

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register PMC1

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register DFC0

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register DFC1

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register DFC2

DEBUG - [perfmon_addEventSet:2306] Cannot access counter register DFC3

The /proc/sys/kernel/perf_event_paranoid is set to 0 already.

Please advise and help.

Best regards,

Long

[Thomas Gruber]

There might be multiple reasons for this. The performance monitoring can be disabled by the system (single bit in a register). Can you use perf or any other tool? Is is a bare-metal system or some virtualized environment? Is a system monitoring daemon like PCP running? What are the contents of /sys/devices ?

Best,
Thomas

[Nichols Romero]

Hello Thomas and other likwid-users,

My apologies for replying to this old e-mail. I am in a similar vote with LIKWID on an AMD Zen3. It is a bare metal and not virtualized. I have perf installed but I cannot tell if its running 100% correctly. I ran some sanity tests for perf and there were definitely a handful of failures.

I do have /proc/sys/kernel/perf_event_paranoid set to 0 already.

Here are the contents of /sys/devices

amd_iommu_0  amd_iommu_2  amd_iommu_4  amd_iommu_6  breakpoint  ibs_fetch  isa     LNXSYSTM:00  pci0000:00  pci0000:40  pci0000:80  pci0000:c0  platform  power     system      uprobe
amd_iommu_1  amd_iommu_3  amd_iommu_5  amd_iommu_7  cpu         ibs_op     kprobe  msr          pci0000:20  pci0000:60  pci0000:a0  pci0000:e0  pnp0      software  tracepoint  virtual

Some other performance tools seem to be working at least in part, e.g. PAPI gives me total flop counts but I don't know how to use it to measurement bandwidth and data volume.

[Thomas Gruber]

Hi Nichols,

From your list of perf_event devices, I can see that your system does not provides the required amd_df unit to measure the data fabric traffic. Flop counts are performed through the cpu unit, so the flop rate is countable with any tool relying on perf_event (PAPI, perf itself of course, LIKWID with perf_event backend, ...). Is it a desktop/mobile version of the Zen3? Which kernel version are you running?

Best,
Thomas

P.S.

I saw your request on the PAPI mailing list regarding arithmetic intensity. There is no such preset because it would require data from sources with different topological specifity. While flops can often be counted per hardware thread, the memory controllers are commonly socket-specific, so you cannot measure the memory traffic of a single hardware thread to derive the arithmetic intensity. Even if you find the proper way to program the memory events in PAPI, it will not work out. If the unit is there but simply not exposed by the system/perf, you could try LIKWID with the accessdaemon backend as it does not rely on perf_event.

[Nichols A. Romero]

Hi Thomas,

This is what I am getting from: uname -r

5.15.0-72-generic

Its an AMD EPYC 7313. 

Thank you for your comments w.r.t e-mail to the PAPI list. I realized later that its a bit more complicated than I first thought.

I did try the LIKWID with the accessdaemon backend (built via Spack), but unfortunately I get a permissions error:

CPU name:       AMD EPYC 7313 16-Core Processor
CPU type:       AMD K19 (Zen3) architecture
CPU clock:      2.99 GHz
Cannot get access to MSRs. Please check permissions to the MSRs

Is this something that a sysadmin could easily fix?

--

---
You received this message because you are subscribed to the Google Groups "likwid-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to likwid-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/likwid-users/f87cb21e-2ad6-49b0-b580-1d6c9c7ed685n%40googlegroups.com.

--

Nichols A. Romero, Ph.D.

[Thomas Gruber]

Hi Nichols,

Since you wrote bare-metal and not virtualized, is the system at some cloud provider? I already heard that some cloud provider restrict hardware performance counting by patching the Linux kernel. The vanilla Linux 5.15.0 knows about the units (https://github.com/torvalds/linux/blob/v5.15/arch/x86/events/amd/uncore.c#L574-L585).

You probably need the msr kernel module, so a modprobe msr by a sysadmin should do the trick. Moreover the spack installation is a two step process. The user installs the package in his/her package tree but the accessdaemon needs special treatment by a sysadmin (chown root:root path/to/likwid-accessD; chmod u+s path/to/likwid-accessD). There was an attempt at spack to generate shell scripts during the build process that need to be executed by a sysadmin. Not sure whether this is already merged and active.

Best,
Thomas

[Nichols A. Romero]

Hi Thomas,

No, it's not a cloud provider. The system is just sitting at company HQ.

So, if I understand correctly I need to have the amd_df unit only if I am using the perf_event backend but not the accessdaemon backend? For the accessdaemon, I will have our sysadmin do the modprobe msr and see if I can get the correct permissions for likwid-accessD. (not sure who they will feel about elevated permissions).

The perf_event backend seem the preferable route, any other things that I should try to get this missing amd_df unit to appear? Scouring the internet with a google search turned up nothing for me :)

To view this discussion on the web visit https://groups.google.com/d/msgid/likwid-users/96e60893-36d7-43f0-ab83-aa281bdad8f6n%40googlegroups.com.

[Thomas Gruber]

Hi Nichols,

Then maybe ask the sysadmins whether they applied any patches to Linux. If not, get a system boot log (like dmesg) to see whether there is some message about amd_df. Based on the kernel sources, the kernel get's its info from the CPUID instruction. So, you could check CPUID (extended AMD flags: CPUID level 0x80000001, ECX, word 6, https://github.com/torvalds/linux/blob/v5.15/arch/x86/include/asm/cpufeatures.h#L182) yourself. I never heard of a BIOS switch for that but you never know. You could check the official documentation whether these is a possibility to deactivate the DataFabric unit, then you know how to turn it on again.

You are correct, the amd_df unit is only required with the perf_event backend. The accessdaemon/direct backend accesses the counter differently but requires higher privileges to do so.

Not sure what you can do. Your system misses both Uncore units (amd_df and amd_l3). Here are the paths LIKWID uses: https://github.com/RRZE-HPC/likwid/blob/master/src/includes/perfmon_zen3_counters.h#L88-L94

Best,
Thomas

[Nichols A. Romero]

Hi Thomas,

I was staring at /boot/config-* and noticed that there are some PERF variables that are not set... and others that have odd values.

Do you have a "good" /boot/config-* that is known to be working?

Here is a snippet of some things that make me take pause and wonder where the Linux kernel might be set-up correctly for perf.

# Performance monitoring
#
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=m
CONFIG_PERF_EVENTS_INTEL_CSTATE=m
# CONFIG_PERF_EVENTS_AMD_POWER is not set
CONFIG_PERF_EVENTS_AMD_UNCORE=m
# end of Performance monitoring

 

You received this message because you are subscribed to a topic in the Google Groups "likwid-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/likwid-users/21GICNvXYb8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to likwid-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/likwid-users/a80b0b2d-3a6a-4fc2-9ad6-0e1c2df92012n%40googlegroups.com.

[Thomas Gruber]

Hi Nichols,

I never looked at kernel configs for perf stuff. My assumption would be that modules are automatically loaded as soon as AMD is detected but I don't know. Therefore... you could try to find out the module name for CONFIG_PERF_EVENTS_AMD_UNCORE and get it loaded by a sysadmin. Then you check the output of /sys/devices again. If there is a amd_df and amd_l3, you should be able to use LIKWID directly (with perf_event backend).

Best,
Thomas

[Nichols A. Romero]

Hi Thomas,

I finally got this working on the Ubuntu AMD system with the Zen3. This is what I had to do:

insmod /lib/modules/5.15.0-72-generic/kernel/arch/x86/events/amd/amd-uncore.ko

If you want me to, I can document it on your LIKWID wiki. Just let me know where it should go and I can make a pull request.

Thanks for your help! I can finally measure memory accesses on our Zen3 via perf_events backend with LIKWID.

To view this discussion on the web visit https://groups.google.com/d/msgid/likwid-users/51b9c010-9c58-41af-b342-2ccc65279601n%40googlegroups.com.

[Thomas Gruber]

Hi,

Glad to read that it works now.

Thanks for the offer. You could add a seperate section above https://github.com/RRZE-HPC/likwid/wiki/TutorialLikwidPerf#feature-limitations . Please use a kernel-independent path like /lib/modules/$(uname -r)/...

For the wiki, you cannot open a PR. If you are not allowed to change the wiki directly, I see two possibilites.

- I add your Github Account to the LIKWID repo as contributor

- You post the text here and I add it

[Nichols A. Romero]

My github username is naromero77. 

Thanks, 

To view this discussion on the web visit https://groups.google.com/d/msgid/likwid-users/682d4a79-9c4e-40d8-b3ab-a2fb9ae4bc3dn%40googlegroups.com.