AMD Zen3 ENERGY measurement
34 views
Skip to first unread message
Demirhan Sevim
Oct 9, 2023, 6:57:32 AM10/9/23
to likwid-users
Hello,
When I run likwid-powermeter I get this error:
ERROR - [./src/access_client.c:access_client_startDaemon:138] No such file or directory.
Failed to find the daemon '/cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.1-x6krdpo47eaam4zob7ubyq33ky6affbb/sbin/likwid-accessD'
Is there something wrong with my installation or are there any flags or events that I need to enable?
Thanks,
I want to use LIKWID Marker API to measure energy on a AMD EPYC 7763 CPU. I can successfully measure energy on a Intel(R) Xeon(R) Gold 5318Y CPU and I can measure other metrics like CLOCK for example on the EPYC device, but when I try to
measure the ENERGY metric on the EPYC device I encounter some errors.
When I use the Marker API: likwid-perfctr -C S0:0 -g ENERGY -m ./run.sh, I get this error:
CPU name: AMD EPYC 7763 64-Core Processor
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:869] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
CPU name: AMD EPYC 7763 64-Core Processor
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:869] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
When I run likwid-powermeter I get this error:
ERROR - [./src/access_client.c:access_client_startDaemon:138] No such file or directory.
Failed to find the daemon '/cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.1-x6krdpo47eaam4zob7ubyq33ky6affbb/sbin/likwid-accessD'
Is there something wrong with my installation or are there any flags or events that I need to enable?
Thanks,
Demirhan
Thomas Gruber
Oct 9, 2023, 7:26:19 AM10/9/23
to likwid-users
Hi,
It might be that your ZEN system does not provide the required units. Does a folder /sys/devices/power exist? If not, you cannot use perf_event event for energy measurements.
likwid-powermeter is only usable with accessdaemon/direct mode but not with perf_event.
Please try a later version of LIKWID. 5.2.1 is years old
Best,
Thomas
Thomas
Demirhan Sevim
Oct 11, 2023, 10:55:26 AM10/11/23
to likwid-users
Hello,
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
I updated to the latest version but I am still getting the same errors.
CPU name: AMD EPYC 7763 64-Core Processor
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:891] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
The line that I get the error changed to 891 but I'm guessing thats because of the code changes in the new version.
Here are the files under my /sys/devices/power
cpumask events format perf_event_mux_interval_ms power subsystem type uevent
Is there anything else that I could check?
Thanks,
Is there anything else that I could check?
Thanks,
Demirhan
Thomas Gruber
Oct 11, 2023, 11:02:59 AM10/11/23
to likwid-users
Hi,
please check that your perf_event_paranoid value is 0: cat /proc/sys/kernel/perf_event_paranoid
If so, please list the files and their content from the /sys/devices/power/events folder.
And run your command again with likwid-perfctr -V 3 <your options> and attach the whole output.
I have a suspicion what could wrong but not sure yet.
Best,
Thomas
Thomas
Demirhan Sevim
Oct 11, 2023, 11:54:32 AM10/11/23
to likwid-users
Hello,
CPU name: AMD EPYC 7763 64-Core Processor
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:891] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:891] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
perf_event_paranoid is set to 0. Here are the files under
/sys/devices/power/events
energy-pkg energy-pkg.scale energy-pkg.unit
Inside energy-pkg -> event=0x02
Inside energy-pkg.scale -> 2.3283064365386962890625e-10
Inside energy-pkg.unit -> Joules
Here is my output from: likwid-perfctr -V 3 -C S0:0 -g ENERGY -m ./run.sh
DEBUG - [hwloc_init_cpuInfo:373] HWLOC CpuInfo Family 25 Model 1 Stepping 1 Vendor 0x0 Part 0x0 isIntel 0 numHWThreads 128 activeHWThreads 128
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 0 Thread 0 Core 0 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 64 Thread 1 Core 0 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 1 Thread 0 Core 1 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 65 Thread 1 Core 1 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 2 Thread 0 Core 2 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 66 Thread 1 Core 2 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 3 Thread 0 Core 3 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 67 Thread 1 Core 3 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 4 Thread 0 Core 4 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 68 Thread 1 Core 4 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 5 Thread 0 Core 5 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 69 Thread 1 Core 5 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 6 Thread 0 Core 6 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 70 Thread 1 Core 6 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 7 Thread 0 Core 7 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 71 Thread 1 Core 7 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 8 Thread 0 Core 8 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 72 Thread 1 Core 8 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 9 Thread 0 Core 9 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 73 Thread 1 Core 9 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 10 Thread 0 Core 10 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 74 Thread 1 Core 10 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 11 Thread 0 Core 11 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 75 Thread 1 Core 11 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 12 Thread 0 Core 12 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 76 Thread 1 Core 12 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 13 Thread 0 Core 13 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 77 Thread 1 Core 13 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 14 Thread 0 Core 14 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 78 Thread 1 Core 14 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 15 Thread 0 Core 15 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 79 Thread 1 Core 15 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 16 Thread 0 Core 16 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 80 Thread 1 Core 16 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 17 Thread 0 Core 17 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 81 Thread 1 Core 17 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 18 Thread 0 Core 18 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 82 Thread 1 Core 18 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 19 Thread 0 Core 19 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 83 Thread 1 Core 19 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 20 Thread 0 Core 20 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 84 Thread 1 Core 20 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 21 Thread 0 Core 21 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 85 Thread 1 Core 21 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 22 Thread 0 Core 22 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 86 Thread 1 Core 22 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 23 Thread 0 Core 23 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 87 Thread 1 Core 23 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 24 Thread 0 Core 24 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 88 Thread 1 Core 24 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 25 Thread 0 Core 25 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 89 Thread 1 Core 25 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 26 Thread 0 Core 26 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 90 Thread 1 Core 26 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 27 Thread 0 Core 27 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 91 Thread 1 Core 27 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 28 Thread 0 Core 28 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 92 Thread 1 Core 28 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 29 Thread 0 Core 29 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 93 Thread 1 Core 29 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 30 Thread 0 Core 30 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 94 Thread 1 Core 30 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 31 Thread 0 Core 31 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 95 Thread 1 Core 31 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 32 Thread 0 Core 32 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 96 Thread 1 Core 32 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 33 Thread 0 Core 33 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 97 Thread 1 Core 33 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 34 Thread 0 Core 34 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 98 Thread 1 Core 34 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 35 Thread 0 Core 35 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 99 Thread 1 Core 35 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 36 Thread 0 Core 36 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 100 Thread 1 Core 36 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 37 Thread 0 Core 37 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 101 Thread 1 Core 37 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 38 Thread 0 Core 38 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 102 Thread 1 Core 38 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 39 Thread 0 Core 39 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 103 Thread 1 Core 39 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 40 Thread 0 Core 40 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 104 Thread 1 Core 40 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 41 Thread 0 Core 41 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 105 Thread 1 Core 41 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 42 Thread 0 Core 42 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 106 Thread 1 Core 42 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 43 Thread 0 Core 43 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 107 Thread 1 Core 43 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 44 Thread 0 Core 44 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 108 Thread 1 Core 44 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 45 Thread 0 Core 45 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 109 Thread 1 Core 45 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 46 Thread 0 Core 46 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 110 Thread 1 Core 46 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 47 Thread 0 Core 47 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 111 Thread 1 Core 47 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 48 Thread 0 Core 48 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 112 Thread 1 Core 48 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 49 Thread 0 Core 49 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 113 Thread 1 Core 49 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 50 Thread 0 Core 50 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 114 Thread 1 Core 50 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 51 Thread 0 Core 51 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 115 Thread 1 Core 51 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 52 Thread 0 Core 52 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 116 Thread 1 Core 52 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 53 Thread 0 Core 53 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 117 Thread 1 Core 53 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 54 Thread 0 Core 54 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 118 Thread 1 Core 54 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 55 Thread 0 Core 55 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 119 Thread 1 Core 55 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 56 Thread 0 Core 56 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 120 Thread 1 Core 56 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 57 Thread 0 Core 57 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 121 Thread 1 Core 57 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 58 Thread 0 Core 58 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 122 Thread 1 Core 58 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 59 Thread 0 Core 59 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 123 Thread 1 Core 59 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 60 Thread 0 Core 60 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 124 Thread 1 Core 60 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 61 Thread 0 Core 61 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 125 Thread 1 Core 61 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 62 Thread 0 Core 62 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 126 Thread 1 Core 62 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 63 Thread 0 Core 63 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 127 Thread 1 Core 63 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 0 Level 1 Size 32768 Threads 2
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 1 Level 2 Size 524288 Threads 2
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 2 Level 3 Size 33554432 Threads 16
DEBUG - [affinity_init:539] Affinity: Socket domains 1
DEBUG - [affinity_init:541] Affinity: CPU die domains 1
DEBUG - [affinity_init:546] Affinity: CPU cores per LLC 8
DEBUG - [affinity_init:549] Affinity: Cache domains 8
DEBUG - [affinity_init:553] Affinity: NUMA domains 1
DEBUG - [affinity_init:554] Affinity: All domains 12
DEBUG - [affinity_addNodeDomain:370] Affinity domain N: 128 HW threads on 64 cores
DEBUG - [affinity_addSocketDomain:401] Affinity domain S0: 128 HW threads on 64 cores
DEBUG - [affinity_addDieDomain:438] Affinity domain D0: 128 HW threads on 64 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C0: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C1: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C2: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C3: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C4: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C5: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C6: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C7: 16 HW threads on 8 cores
DEBUG - [affinity_addMemoryDomain:504] Affinity domain M0: 128 HW threads on 64 cores
DEBUG - [create_lookups:290] T 0 T2C 0 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 1 T2C 1 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 2 T2C 2 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 3 T2C 3 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 4 T2C 4 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 5 T2C 5 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 6 T2C 6 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 7 T2C 7 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 8 T2C 8 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 9 T2C 9 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 10 T2C 10 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 11 T2C 11 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 12 T2C 12 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 13 T2C 13 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 14 T2C 14 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 15 T2C 15 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 16 T2C 16 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 17 T2C 17 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 18 T2C 18 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 19 T2C 19 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 20 T2C 20 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 21 T2C 21 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 22 T2C 22 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 23 T2C 23 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 24 T2C 24 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 25 T2C 25 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 26 T2C 26 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 27 T2C 27 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 28 T2C 28 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 29 T2C 29 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 30 T2C 30 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 31 T2C 31 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 32 T2C 32 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 33 T2C 33 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 34 T2C 34 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 35 T2C 35 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 36 T2C 36 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 37 T2C 37 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 38 T2C 38 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 39 T2C 39 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 40 T2C 40 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 41 T2C 41 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 42 T2C 42 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 43 T2C 43 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 44 T2C 44 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 45 T2C 45 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 46 T2C 46 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 47 T2C 47 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 48 T2C 48 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 49 T2C 49 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 50 T2C 50 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 51 T2C 51 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 52 T2C 52 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 53 T2C 53 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 54 T2C 54 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 55 T2C 55 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 56 T2C 56 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 57 T2C 57 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 58 T2C 58 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 59 T2C 59 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 60 T2C 60 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 61 T2C 61 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 62 T2C 62 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 63 T2C 63 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 64 T2C 0 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 65 T2C 1 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 66 T2C 2 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 67 T2C 3 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 68 T2C 4 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 69 T2C 5 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 70 T2C 6 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 71 T2C 7 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 72 T2C 8 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 73 T2C 9 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 74 T2C 10 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 75 T2C 11 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 76 T2C 12 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 77 T2C 13 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 78 T2C 14 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 79 T2C 15 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 80 T2C 16 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 81 T2C 17 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 82 T2C 18 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 83 T2C 19 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 84 T2C 20 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 85 T2C 21 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 86 T2C 22 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 87 T2C 23 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 88 T2C 24 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 89 T2C 25 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 90 T2C 26 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 91 T2C 27 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 92 T2C 28 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 93 T2C 29 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 94 T2C 30 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 95 T2C 31 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 96 T2C 32 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 97 T2C 33 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 98 T2C 34 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 99 T2C 35 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 100 T2C 36 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 101 T2C 37 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 102 T2C 38 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 103 T2C 39 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 104 T2C 40 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 105 T2C 41 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 106 T2C 42 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 107 T2C 43 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 108 T2C 44 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 109 T2C 45 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 110 T2C 46 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 111 T2C 47 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 112 T2C 48 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 113 T2C 49 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 114 T2C 50 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 115 T2C 51 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 116 T2C 52 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 117 T2C 53 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 118 T2C 54 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 119 T2C 55 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 120 T2C 56 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 121 T2C 57 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 122 T2C 58 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 123 T2C 59 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 124 T2C 60 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 125 T2C 61 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 126 T2C 62 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 127 T2C 63 T2S 0 T2D 0 T2LLC 7 T2M 0
--------------------------------------------------------------------------------
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 0 Thread 0 Core 0 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 64 Thread 1 Core 0 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 1 Thread 0 Core 1 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 65 Thread 1 Core 1 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 2 Thread 0 Core 2 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 66 Thread 1 Core 2 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 3 Thread 0 Core 3 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 67 Thread 1 Core 3 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 4 Thread 0 Core 4 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 68 Thread 1 Core 4 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 5 Thread 0 Core 5 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 69 Thread 1 Core 5 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 6 Thread 0 Core 6 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 70 Thread 1 Core 6 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 7 Thread 0 Core 7 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 71 Thread 1 Core 7 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 8 Thread 0 Core 8 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 72 Thread 1 Core 8 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 9 Thread 0 Core 9 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 73 Thread 1 Core 9 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 10 Thread 0 Core 10 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 74 Thread 1 Core 10 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 11 Thread 0 Core 11 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 75 Thread 1 Core 11 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 12 Thread 0 Core 12 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 76 Thread 1 Core 12 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 13 Thread 0 Core 13 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 77 Thread 1 Core 13 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 14 Thread 0 Core 14 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 78 Thread 1 Core 14 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 15 Thread 0 Core 15 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 79 Thread 1 Core 15 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 16 Thread 0 Core 16 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 80 Thread 1 Core 16 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 17 Thread 0 Core 17 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 81 Thread 1 Core 17 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 18 Thread 0 Core 18 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 82 Thread 1 Core 18 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 19 Thread 0 Core 19 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 83 Thread 1 Core 19 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 20 Thread 0 Core 20 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 84 Thread 1 Core 20 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 21 Thread 0 Core 21 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 85 Thread 1 Core 21 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 22 Thread 0 Core 22 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 86 Thread 1 Core 22 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 23 Thread 0 Core 23 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 87 Thread 1 Core 23 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 24 Thread 0 Core 24 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 88 Thread 1 Core 24 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 25 Thread 0 Core 25 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 89 Thread 1 Core 25 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 26 Thread 0 Core 26 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 90 Thread 1 Core 26 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 27 Thread 0 Core 27 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 91 Thread 1 Core 27 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 28 Thread 0 Core 28 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 92 Thread 1 Core 28 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 29 Thread 0 Core 29 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 93 Thread 1 Core 29 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 30 Thread 0 Core 30 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 94 Thread 1 Core 30 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 31 Thread 0 Core 31 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 95 Thread 1 Core 31 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 32 Thread 0 Core 32 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 96 Thread 1 Core 32 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 33 Thread 0 Core 33 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 97 Thread 1 Core 33 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 34 Thread 0 Core 34 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 98 Thread 1 Core 34 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 35 Thread 0 Core 35 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 99 Thread 1 Core 35 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 36 Thread 0 Core 36 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 100 Thread 1 Core 36 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 37 Thread 0 Core 37 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 101 Thread 1 Core 37 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 38 Thread 0 Core 38 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 102 Thread 1 Core 38 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 39 Thread 0 Core 39 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 103 Thread 1 Core 39 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 40 Thread 0 Core 40 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 104 Thread 1 Core 40 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 41 Thread 0 Core 41 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 105 Thread 1 Core 41 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 42 Thread 0 Core 42 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 106 Thread 1 Core 42 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 43 Thread 0 Core 43 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 107 Thread 1 Core 43 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 44 Thread 0 Core 44 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 108 Thread 1 Core 44 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 45 Thread 0 Core 45 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 109 Thread 1 Core 45 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 46 Thread 0 Core 46 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 110 Thread 1 Core 46 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 47 Thread 0 Core 47 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 111 Thread 1 Core 47 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 48 Thread 0 Core 48 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 112 Thread 1 Core 48 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 49 Thread 0 Core 49 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 113 Thread 1 Core 49 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 50 Thread 0 Core 50 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 114 Thread 1 Core 50 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 51 Thread 0 Core 51 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 115 Thread 1 Core 51 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 52 Thread 0 Core 52 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 116 Thread 1 Core 52 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 53 Thread 0 Core 53 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 117 Thread 1 Core 53 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 54 Thread 0 Core 54 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 118 Thread 1 Core 54 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 55 Thread 0 Core 55 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 119 Thread 1 Core 55 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 56 Thread 0 Core 56 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 120 Thread 1 Core 56 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 57 Thread 0 Core 57 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 121 Thread 1 Core 57 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 58 Thread 0 Core 58 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 122 Thread 1 Core 58 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 59 Thread 0 Core 59 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 123 Thread 1 Core 59 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 60 Thread 0 Core 60 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 124 Thread 1 Core 60 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 61 Thread 0 Core 61 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 125 Thread 1 Core 61 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 62 Thread 0 Core 62 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 126 Thread 1 Core 62 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 63 Thread 0 Core 63 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_nodeTopology:582] HWLOC Thread Pool PU 127 Thread 1 Core 63 Die 0 Socket 0 inCpuSet 1
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 0 Level 1 Size 32768 Threads 2
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 1 Level 2 Size 524288 Threads 2
DEBUG - [hwloc_init_cacheTopology:803] HWLOC Cache Pool ID 2 Level 3 Size 33554432 Threads 16
DEBUG - [affinity_init:539] Affinity: Socket domains 1
DEBUG - [affinity_init:541] Affinity: CPU die domains 1
DEBUG - [affinity_init:546] Affinity: CPU cores per LLC 8
DEBUG - [affinity_init:549] Affinity: Cache domains 8
DEBUG - [affinity_init:553] Affinity: NUMA domains 1
DEBUG - [affinity_init:554] Affinity: All domains 12
DEBUG - [affinity_addNodeDomain:370] Affinity domain N: 128 HW threads on 64 cores
DEBUG - [affinity_addSocketDomain:401] Affinity domain S0: 128 HW threads on 64 cores
DEBUG - [affinity_addDieDomain:438] Affinity domain D0: 128 HW threads on 64 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C0: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C1: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C2: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C3: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C4: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C5: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C6: 16 HW threads on 8 cores
DEBUG - [affinity_addCacheDomain:474] Affinity domain C7: 16 HW threads on 8 cores
DEBUG - [affinity_addMemoryDomain:504] Affinity domain M0: 128 HW threads on 64 cores
DEBUG - [create_lookups:290] T 0 T2C 0 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 1 T2C 1 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 2 T2C 2 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 3 T2C 3 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 4 T2C 4 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 5 T2C 5 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 6 T2C 6 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 7 T2C 7 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 8 T2C 8 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 9 T2C 9 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 10 T2C 10 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 11 T2C 11 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 12 T2C 12 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 13 T2C 13 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 14 T2C 14 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 15 T2C 15 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 16 T2C 16 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 17 T2C 17 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 18 T2C 18 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 19 T2C 19 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 20 T2C 20 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 21 T2C 21 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 22 T2C 22 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 23 T2C 23 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 24 T2C 24 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 25 T2C 25 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 26 T2C 26 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 27 T2C 27 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 28 T2C 28 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 29 T2C 29 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 30 T2C 30 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 31 T2C 31 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 32 T2C 32 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 33 T2C 33 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 34 T2C 34 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 35 T2C 35 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 36 T2C 36 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 37 T2C 37 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 38 T2C 38 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 39 T2C 39 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 40 T2C 40 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 41 T2C 41 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 42 T2C 42 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 43 T2C 43 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 44 T2C 44 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 45 T2C 45 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 46 T2C 46 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 47 T2C 47 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 48 T2C 48 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 49 T2C 49 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 50 T2C 50 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 51 T2C 51 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 52 T2C 52 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 53 T2C 53 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 54 T2C 54 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 55 T2C 55 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 56 T2C 56 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 57 T2C 57 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 58 T2C 58 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 59 T2C 59 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 60 T2C 60 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 61 T2C 61 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 62 T2C 62 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 63 T2C 63 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 64 T2C 0 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 65 T2C 1 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 66 T2C 2 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 67 T2C 3 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 68 T2C 4 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 69 T2C 5 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 70 T2C 6 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 71 T2C 7 T2S 0 T2D 0 T2LLC 0 T2M 0
DEBUG - [create_lookups:290] T 72 T2C 8 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 73 T2C 9 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 74 T2C 10 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 75 T2C 11 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 76 T2C 12 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 77 T2C 13 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 78 T2C 14 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 79 T2C 15 T2S 0 T2D 0 T2LLC 1 T2M 0
DEBUG - [create_lookups:290] T 80 T2C 16 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 81 T2C 17 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 82 T2C 18 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 83 T2C 19 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 84 T2C 20 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 85 T2C 21 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 86 T2C 22 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 87 T2C 23 T2S 0 T2D 0 T2LLC 2 T2M 0
DEBUG - [create_lookups:290] T 88 T2C 24 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 89 T2C 25 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 90 T2C 26 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 91 T2C 27 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 92 T2C 28 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 93 T2C 29 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 94 T2C 30 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 95 T2C 31 T2S 0 T2D 0 T2LLC 3 T2M 0
DEBUG - [create_lookups:290] T 96 T2C 32 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 97 T2C 33 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 98 T2C 34 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 99 T2C 35 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 100 T2C 36 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 101 T2C 37 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 102 T2C 38 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 103 T2C 39 T2S 0 T2D 0 T2LLC 4 T2M 0
DEBUG - [create_lookups:290] T 104 T2C 40 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 105 T2C 41 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 106 T2C 42 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 107 T2C 43 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 108 T2C 44 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 109 T2C 45 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 110 T2C 46 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 111 T2C 47 T2S 0 T2D 0 T2LLC 5 T2M 0
DEBUG - [create_lookups:290] T 112 T2C 48 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 113 T2C 49 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 114 T2C 50 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 115 T2C 51 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 116 T2C 52 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 117 T2C 53 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 118 T2C 54 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 119 T2C 55 T2S 0 T2D 0 T2LLC 6 T2M 0
DEBUG - [create_lookups:290] T 120 T2C 56 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 121 T2C 57 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 122 T2C 58 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 123 T2C 59 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 124 T2C 60 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 125 T2C 61 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 126 T2C 62 T2S 0 T2D 0 T2LLC 7 T2M 0
DEBUG - [create_lookups:290] T 127 T2C 63 T2S 0 T2D 0 T2LLC 7 T2M 0
--------------------------------------------------------------------------------
CPU name: AMD EPYC 7763 64-Core Processor
CPU type: AMD K19 (Zen3) architecture
CPU clock: 2.45 GHz
CPU family: 25
CPU model: 1
CPU short: zen3
CPU stepping: 1
CPU features: FP MMX SSE SSE2 HTT MMX RDTSCP MONITOR SSSE FMA SSE4.1 SSE4.2 AES AVX RDRAND AVX2 RDSEED SSE3
CPU arch: x86_64
--------------------------------------------------------------------------------
[likwid-pin] Main PID -> hwthread 0 - OK
Executing: ./run.sh blackscholes native
DEBUG - [perfmon_addEventSet:2187] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:873] Reading group ENERGY from /cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.2-xatndb6pdfiz3z62pwpf2lqmchds2whd/share/likwid/perfgroups/zen3/ENERGY.txt
DEBUG - [perfmon_addEventSet:2366] Added event ACTUAL_CPU_CLOCK for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event MAX_CPU_CLOCK for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RETIRED_INSTRUCTIONS for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event CPU_CLOCKS_UNHALTED for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_CORE_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_PKG_ENERGY for counter PWR1 to group 0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x1 , Flags: 0x1
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x2 , Flags: 0x2
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x3 , Flags: 0xC0
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x4 , Flags: 0x76
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0xF , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
CPU model: 1
CPU short: zen3
CPU stepping: 1
CPU features: FP MMX SSE SSE2 HTT MMX RDTSCP MONITOR SSSE FMA SSE4.1 SSE4.2 AES AVX RDRAND AVX2 RDSEED SSE3
CPU arch: x86_64
--------------------------------------------------------------------------------
[likwid-pin] Main PID -> hwthread 0 - OK
Executing: ./run.sh blackscholes native
DEBUG - [perfmon_addEventSet:2187] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:873] Reading group ENERGY from /cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.2-xatndb6pdfiz3z62pwpf2lqmchds2whd/share/likwid/perfgroups/zen3/ENERGY.txt
DEBUG - [perfmon_addEventSet:2366] Added event ACTUAL_CPU_CLOCK for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event MAX_CPU_CLOCK for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RETIRED_INSTRUCTIONS for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event CPU_CLOCKS_UNHALTED for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_CORE_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_PKG_ENERGY for counter PWR1 to group 0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x1 , Flags: 0x1
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x2 , Flags: 0x2
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x3 , Flags: 0xC0
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x4 , Flags: 0x76
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0xF , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:891] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
DEBUG - [perfmon_setupCountersThread_perfevent:892] open error: cpu_id=0 pid=-1 flags=0 type=15 config=0x1 disabled=1 inherit=1 exclusive=0 config2=0x7F39EFA7EA38
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0x10 , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
--------------------------------------------------------------------------------
[PARSEC] Benchmarks to run: parsec.blackscholes
[PARSEC] [========== Running benchmark parsec.blackscholes [1] ==========]
[PARSEC] Deleting old run directory.
[PARSEC] Setting up run directory.
[PARSEC] Unpacking benchmark input 'native'.
in_10M.txt
[PARSEC] Running 'time /cta/users/dsevim/Documents/parsec-3.0/pkgs/apps/blackscholes/inst/amd64-linux.gcc-hooks/bin/blackscholes 1 in_10M.txt prices.txt 1 0 -t cpu -d 0 -p 1':
[PARSEC] [---------- Beginning of output ----------]
DEBUG - [perfmon_addEventSet:2187] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:873] Reading group ENERGY from /cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.2-xatndb6pdfiz3z62pwpf2lqmchds2whd/share/likwid/perfgroups/zen3/ENERGY.txt
DEBUG - [perfmon_addEventSet:2366] Added event ACTUAL_CPU_CLOCK for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event MAX_CPU_CLOCK for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RETIRED_INSTRUCTIONS for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event CPU_CLOCKS_UNHALTED for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_CORE_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_PKG_ENERGY for counter PWR1 to group 0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x1 , Flags: 0x1
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x2 , Flags: 0x2
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x3 , Flags: 0xC0
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x4 , Flags: 0x76
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0xF , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0x10 , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
--------------------------------------------------------------------------------
[PARSEC] Benchmarks to run: parsec.blackscholes
[PARSEC] [========== Running benchmark parsec.blackscholes [1] ==========]
[PARSEC] Deleting old run directory.
[PARSEC] Setting up run directory.
[PARSEC] Unpacking benchmark input 'native'.
in_10M.txt
[PARSEC] Running 'time /cta/users/dsevim/Documents/parsec-3.0/pkgs/apps/blackscholes/inst/amd64-linux.gcc-hooks/bin/blackscholes 1 in_10M.txt prices.txt 1 0 -t cpu -d 0 -p 1':
[PARSEC] [---------- Beginning of output ----------]
DEBUG - [perfmon_addEventSet:2187] Currently 1 groups of 2 active
DEBUG - [perfgroup_readGroup:873] Reading group ENERGY from /cta/apps/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/likwid-5.2.2-xatndb6pdfiz3z62pwpf2lqmchds2whd/share/likwid/perfgroups/zen3/ENERGY.txt
DEBUG - [perfmon_addEventSet:2366] Added event ACTUAL_CPU_CLOCK for counter FIXC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event MAX_CPU_CLOCK for counter FIXC2 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RETIRED_INSTRUCTIONS for counter PMC0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event CPU_CLOCKS_UNHALTED for counter PMC1 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_CORE_ENERGY for counter PWR0 to group 0
DEBUG - [perfmon_addEventSet:2366] Added event RAPL_PKG_ENERGY for counter PWR1 to group 0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x1 , Flags: 0x1
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:710] SETUP_FIXED [0] Register 0x2 , Flags: 0x2
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x3 , Flags: 0xC0
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:741] SETUP_PMC [0] Register 0x4 , Flags: 0x76
DEBUG - [perfmon_setupCountersThread_perfevent:872] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0xF , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
ERROR - [./src/includes/perfmon_perfevent.h:perfmon_setupCountersThread_perfevent:891] Invalid argument.
Setup of event RAPL_CORE_ENERGY on CPU 0 failed: Invalid argument
DEBUG - [perfmon_setupCountersThread_perfevent:892] open error: cpu_id=0 pid=-1 flags=0 type=15 config=0x1 disabled=1 inherit=1 exclusive=0 config2=0x7FA911502A38
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0x10 , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x5 , Flags: 0x4CF96
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x6 , Flags: 0x877F7
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x7 , Flags: 0xB26D
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x8 , Flags: 0xBF9C
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x9 , Flags: 0xF330000
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 0 VALUE 315286
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 1 VALUE 554999
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 2 VALUE 45677
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 3 VALUE 49052
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 4 VALUE 0
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 5 VALUE 255000576
PARSEC Benchmark Suite Version 3.0-beta-20150206
[HOOKS] PARSEC Hooks Version 1.2
Num of Options: 10000000
Num of Runs: 1
Size of data: 400000000
[HOOKS] Entering ROI
[HOOKS] Leaving ROI
[HOOKS] Total time spent in ROI: 0.432s
[HOOKS] Terminating
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x5 , Flags: 0x85A03A663
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x6 , Flags: 0x602780854
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x7 , Flags: 0x1E2180B8D8
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x8 , Flags: 0x7ECD245F2
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x9 , Flags: 0x3DC694F0000
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 0 VALUE 35869927011 DIFF 35869611725.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 1 VALUE 25811224660 DIFF 25810669661.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 2 VALUE 129411102936 DIFF 129411057259.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 3 VALUE 34037974514 DIFF 34037925462.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 4 VALUE 0 DIFF 0.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 5 VALUE 4245194473472 DIFF 0.000000
DEBUG - [likwid_markerClose:421] Creating Marker file /tmp/likwid_369376.txt with 1 regions 1 groups and 1 threads
DEBUG - [likwid_markerClose:426] 1 1 1
DEBUG - [likwid_markerClose:435] 0:all-0
DEBUG - [likwid_markerClose:461] 0 0 0 1 1.424055e+01 6 3.586961e+10 2.581067e+10 1.294111e+11 3.403793e+10 0.000000e+00 0.000000e+00
real 0m15,327s
user 0m9,989s
sys 0m0,515s
[PARSEC] [---------- End of output ----------]
[PARSEC]
[PARSEC] BIBLIOGRAPHY
[PARSEC]
[PARSEC] [1] Bienia. Benchmarking Modern Multiprocessors. Ph.D. Thesis, 2011.
[PARSEC]
[PARSEC] Done.
--------------------------------------------------------------------------------
Region all, Group 1: ENERGY
+-------------------+------------+
| Region Info | HWThread 0 |
+-------------------+------------+
| RDTSC Runtime [s] | 14.240550 |
| call count | 1 |
+-------------------+------------+
+----------------------+---------+--------------+
| Event | Counter | HWThread 0 |
+----------------------+---------+--------------+
| ACTUAL_CPU_CLOCK | FIXC1 | 35869610000 |
| MAX_CPU_CLOCK | FIXC2 | 25810670000 |
| RETIRED_INSTRUCTIONS | PMC0 | 129411100000 |
| CPU_CLOCKS_UNHALTED | PMC1 | 34037930000 |
| RAPL_CORE_ENERGY | PWR0 | 0 |
| RAPL_PKG_ENERGY | PWR1 | 0 |
+----------------------+---------+--------------+
+----------------------+------------+
| Metric | HWThread 0 |
+----------------------+------------+
| Runtime (RDTSC) [s] | 14.2406 |
| Runtime unhalted [s] | 14.6411 |
| Clock [MHz] | 3404.7171 |
| CPI | 0.2630 |
| Energy Core [J] | 0 |
| Power Core [W] | 0 |
| Energy PKG [J] | 0 |
| Power PKG [W] | 0 |
+----------------------+------------+
DEBUG - [perfmon_setupCountersThread_perfevent:748] SETUP_POWER [0] Register 0x10 , Flags: 0x0
DEBUG - [perf_uncore_setup:527] Get information for uncore counters from folder /sys/bus/event_source/devices/power
DEBUG - [perfmon_setupCountersThread_perfevent:882] perf_event_open: cpu_id=0 pid=-1 flags=0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:928] RESET_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_startCountersThread_perfevent:939] START_COUNTER [0] Register 0x0 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x5 , Flags: 0x4CF96
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x6 , Flags: 0x877F7
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x7 , Flags: 0xB26D
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x8 , Flags: 0xBF9C
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x9 , Flags: 0xF330000
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 0 VALUE 315286
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 1 VALUE 554999
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 2 VALUE 45677
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 3 VALUE 49052
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 4 VALUE 0
DEBUG - [likwid_markerStartRegion:561] START [all] READ EVENT [0=0] EVENT 5 VALUE 255000576
PARSEC Benchmark Suite Version 3.0-beta-20150206
[HOOKS] PARSEC Hooks Version 1.2
Num of Options: 10000000
Num of Runs: 1
Size of data: 400000000
[HOOKS] Entering ROI
[HOOKS] Leaving ROI
[HOOKS] Total time spent in ROI: 0.432s
[HOOKS] Terminating
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x5 , Flags: 0x85A03A663
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x5 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x6 , Flags: 0x602780854
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x6 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x7 , Flags: 0x1E2180B8D8
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x7 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x8 , Flags: 0x7ECD245F2
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x8 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1015] FREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [perfmon_readCountersThread_perfevent:1019] READ_COUNTER [0] Register 0x9 , Flags: 0x3DC694F0000
DEBUG - [perfmon_readCountersThread_perfevent:1042] UNFREEZE_COUNTER [0] Register 0x9 , Flags: 0x0
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 0 VALUE 35869927011 DIFF 35869611725.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 1 VALUE 25811224660 DIFF 25810669661.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 2 VALUE 129411102936 DIFF 129411057259.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 3 VALUE 34037974514 DIFF 34037925462.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 4 VALUE 0 DIFF 0.000000
DEBUG - [likwid_markerStopRegion:638] STOP [all] READ EVENT [0=0] EVENT 5 VALUE 4245194473472 DIFF 0.000000
DEBUG - [likwid_markerClose:421] Creating Marker file /tmp/likwid_369376.txt with 1 regions 1 groups and 1 threads
DEBUG - [likwid_markerClose:426] 1 1 1
DEBUG - [likwid_markerClose:435] 0:all-0
DEBUG - [likwid_markerClose:461] 0 0 0 1 1.424055e+01 6 3.586961e+10 2.581067e+10 1.294111e+11 3.403793e+10 0.000000e+00 0.000000e+00
real 0m15,327s
user 0m9,989s
sys 0m0,515s
[PARSEC] [---------- End of output ----------]
[PARSEC]
[PARSEC] BIBLIOGRAPHY
[PARSEC]
[PARSEC] [1] Bienia. Benchmarking Modern Multiprocessors. Ph.D. Thesis, 2011.
[PARSEC]
[PARSEC] Done.
--------------------------------------------------------------------------------
Region all, Group 1: ENERGY
+-------------------+------------+
| Region Info | HWThread 0 |
+-------------------+------------+
| RDTSC Runtime [s] | 14.240550 |
| call count | 1 |
+-------------------+------------+
+----------------------+---------+--------------+
| Event | Counter | HWThread 0 |
+----------------------+---------+--------------+
| ACTUAL_CPU_CLOCK | FIXC1 | 35869610000 |
| MAX_CPU_CLOCK | FIXC2 | 25810670000 |
| RETIRED_INSTRUCTIONS | PMC0 | 129411100000 |
| CPU_CLOCKS_UNHALTED | PMC1 | 34037930000 |
| RAPL_CORE_ENERGY | PWR0 | 0 |
| RAPL_PKG_ENERGY | PWR1 | 0 |
+----------------------+---------+--------------+
+----------------------+------------+
| Metric | HWThread 0 |
+----------------------+------------+
| Runtime (RDTSC) [s] | 14.2406 |
| Runtime unhalted [s] | 14.6411 |
| Clock [MHz] | 3404.7171 |
| CPI | 0.2630 |
| Energy Core [J] | 0 |
| Power Core [W] | 0 |
| Energy PKG [J] | 0 |
| Power PKG [W] | 0 |
+----------------------+------------+
Thomas Gruber
Oct 11, 2023, 12:03:43 PM10/11/23
to likwid-users
Your system does not provide the RAPL_CORE_ENERGY event through perf_event. Go to your installation folder and remove the event RAPL_CORE_ENERGY from the eventlist and remove all lines containing PWR0 from the metric list. But you should be able to get energy data from the socket (RAPL_PKG_ENERGY:PWR1). I will note on my todo list to check why RAPL_CORE_ENERGY is not simply skipped.
Before changing the file, you can test the socket event using likwid-perfctr -g RAPL_PKG_ENERGY:PWR1 ...
Best,
Thomas
Thomas
Demirhan Sevim
Oct 11, 2023, 12:28:39 PM10/11/23
to likwid-users
Thanks a lot for the help. When I run: likwid-perfctr -C S0:0 -g RAPL_PKG_ENERGY:PWR1 -m ./run.sh I don't get any errors now but the reading is 0:
Region all, Group 1: Custom
+-------------------+------------+
| Region Info | HWThread 0 |
+-------------------+------------+
| RDTSC Runtime [s] | 12.514800 |
| call count | 1 |
+-------------------+------------+
+---------------------+---------+--------------+
| Event | Counter | HWThread 0 |
+---------------------+---------+--------------+
| Runtime (RDTSC) [s] | TSC | 1.251480e+01 |
| RAPL_PKG_ENERGY | PWR1 | 0 |
+---------------------+---------+--------------+
| call count | 1 |
+-------------------+------------+
+---------------------+---------+--------------+
| Event | Counter | HWThread 0 |
+---------------------+---------+--------------+
| Runtime (RDTSC) [s] | TSC | 1.251480e+01 |
| RAPL_PKG_ENERGY | PWR1 | 0 |
+---------------------+---------+--------------+
I'm guessing it doesn't provide that as well. Do you know if there is any other way or another API that I can use to read power/energy from an AMD CPU?
Thanks a lot again,
Demirhan
Thomas Gruber
Oct 11, 2023, 4:39:39 PM10/11/23
to likwid-users
So this means, the configuration of this event works. Why it does not count is out-of-scope of LIKWID. In perf_event mode, it relies on perf_event providing reasonable data.
You could try perf itself whether it generally works (I use perf rarely, so I hope this works):
perf stat -e power/event=0x02/ <executable>
Not sure whether perf scales the raw value directly or whether you have to multiply with energy-pkg.scale to get Joules.
Since most tools out there rely on perf_event, there is not much choice when perf_event does not provide the unit/event or does not count. Many tools use the PAPI library for measurements, which uses itself perf_event. The only tool, except LIKWID, which does not fully rely on perf_event is Intel's PCM but it probably does not support your AMD system.
If you rebuild LIKWID with accessdaemon (requires root privileges once at installation) or direct mode (requires root privileges at runtime), it might work. It does not use perf_event but access the counters directly.
Best,
Thomas
Thomas
Reply all
Reply to author
Forward
0 new messages