Number of available counters decreased with Linux kernel update.

[Pavel]

Hello,

We have upgraded the Linux kernel version on our machine and noticed that our monitoring system (that uses PAPI 5.6.0.0) is not able to collect all processor counters anymore with error "Event exists, but cannot be counted due to hardware resource limits".

Do you know what could cause that? Any help would be appreciated.

Enabling multiplexing helps to solve it, but counters sometimes report negative speed (e.g. PAPI_LD_INS reading without resetting: 100 -> 200 -> 300 -> 290 -> 400 -> 500 ...)

OLD kernel:

/usr/sbin/sysctl -w kernel.nmi_watchdog=0
./papi_event_chooser PRESET PAPI_LD_INS PAPI_SR_INS

Event Chooser: Available events which can be added with given events.
--------------------------------------------------------------------------------
PAPI version             : 5.6.0.0
Operating system         : Linux 3.10.0-229.el7.x86_64
Vendor string and code   : GenuineIntel (1, 0x1)
Model string and code    : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz (63, 0x3f)
CPU revision             : 2.000000
CPUID                    : Family/Model/Stepping 6/63/2, 0x06/0x3f/0x02
CPU Max MHz              : 3600
CPU Min MHz              : 1200
Total cores              : 28
SMT threads per core     : 2
Cores per socket         : 14
Sockets                  : 1
Cores per NUMA region    : 28
NUMA regions             : 1
Running in a VM          : no
Number Hardware Counters : 11
Max Multiplex Counters   : 384
Fast counter read (rdpmc): no
--------------------------------------------------------------------------------
    Name        Code    Deriv Description (Note)
PAPI_L1_DCM  0x80000000  No   Level 1 data cache misses
PAPI_L1_ICM  0x80000001  No   Level 1 instruction cache misses
PAPI_L2_DCM  0x80000002  Yes  Level 2 data cache misses
...

PAPI_L3_TCW  0x80000060  No   Level 3 total cache writes
PAPI_REF_CYC 0x8000006b  No   Reference clock cycles
-------------------------------------------------------------------------
Total events reported: 54

NEW kernel:

Event Chooser: Available events which can be added with given events.
--------------------------------------------------------------------------------
PAPI version             : 5.6.0.0
Operating system         : Linux 3.10.0-957.1.3.el7.x86_64
Vendor string and code   : GenuineIntel (1, 0x1)
Model string and code    : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz (63, 0x3f)
CPU revision             : 2.000000
CPUID                    : Family/Model/Stepping 6/63/2, 0x06/0x3f/0x02
CPU Max MHz              : 3600
CPU Min MHz              : 1200
Total cores              : 28
SMT threads per core     : 2
Cores per socket         : 14
Sockets                  : 1
Cores per NUMA region    : 28
NUMA regions             : 1
Running in a VM          : no
Number Hardware Counters : 11
Max Multiplex Counters   : 384
Fast counter read (rdpmc): no
--------------------------------------------------------------------------------

    Name        Code    Deriv Description (Note)
PAPI_TOT_INS 0x80000032  No   Instructions completed
PAPI_TOT_CYC 0x8000003b  No   Total cycles
PAPI_LST_INS 0x8000003c  Yes  Load/store instructions completed
PAPI_REF_CYC 0x8000006b  No   Reference clock cycles
-------------------------------------------------------------------------
Total events reported: 4

[Phil Mucci]

Hi Pavel.

It’s possible the kernel may have blacklisted the presets used in those events due to errors when used in conjunction with Hyperthreading. A number of them could suffer corruption when HT was enabled. 

I would try and check the derivations of the original presets, like L1_DCM and check that event specifically with papi_native_avail.

Phil

--
You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/19e383f8-df6a-40f2-9e70-ca69a3887385n%40icl.utk.edu.

[Phil Mucci]

Hi Pavel,

That is the kernel enforcing rules on simultaneously measuring events that corrupt each others counters when scheduled on the same core’s hyper threads. If you want this to work on the new kernel, boot the kernel with hyperthreading disabled. That should do it. The option varies on kernel version, you probably need ’nosmt’ on your boot line. 

If you do end up using multiplexing in the HT case, make sure your multiplexing interval is long enough to capture enough samples… 

TL/DR. 

If you’d like gory details of the kernel’s event scheduler and failings w.r.t HT and errata, I recommend this post. https://hadibrais.wordpress.com/2019/09/06/the-linux-perf-event-scheduling-algorithm/

Phil

On Apr 7, 2021, at 10:55, Pavel <shvets...@gmail.com> wrote:

Hello Phil, thanks alot for the answer!

I checked papi_native_avail and papi_decode - everything looks the same on different kernels.

Also I was able to reproduce such behavior with perf util. On the old kernel 3 counters work together and on the new one the multiplexing is required. So it seems to be some issues in the kernel.. (

[Pavel]

Hello Phil,

Thank you very much! We will try it and will study the article you provided.

среда, 7 апреля 2021 г. в 12:59:38 UTC+3, mucci: