Sampling Performance Counters (hpm counters)

[Gururaj Saileshwar]

Hi All, 

I am new to firesim, so apologies if this is a naïve question. I am trying to get performance-counter statistics for SPEC-17 benchmarks on Firesim. I am following the methodology for single-node simulations in the Firesim Doc, and using the default config: a 4-core Rocket-core RISC-V design. I am interested in measuring execution-cycles, executed-instructions, L2-accesses, L2-misses and DRAM-accesses for each benchmark.

I tried to use the hpm_counters and memory_stats.csv to measure the core and memory statistics as described in this UC Berkeley Lab Assignment – however, I am having trouble with both aspects:

1. I tried adding the commands “start_counters” and “stop_counters” before and after running a SPEC-17 benchmark inside the image, but I am getting the error “command not found” for both start_counters and stop_counters. I added    "—counter" in the common_args field in spec17-intrate.json file, to ensure these commands are issued before and after running each benchmark.
2. The memory_stats.csv pulled from the FPGA to the manager-instance after the benchmark completes, does not contain any statistics. It has only one line with headers - “brespError,llc_misses,llc_peakMSHRsUsed …” and no values. Is there something additional to be done inside the FPGA-image to be able to ensure the statistics are recorded in this csv file?

Could someone with more experience help me understand how I can get the core and cache statistics for a benchmark in Firesim? 

Thanks,

Gururaj

 

--------------------

Some additional details:

• I am following the default methodology for running SPEC-17 benchmarks (with the “test” input) in the Firesim Doc, after setting up the infrastructure as outlined in Getting-Started guide. The benchmarks themselves are running successfully in rate-mode, as per the .out and .err files pulled from the FPGA at the end.
• I  observe the image “firesim/sw/firesim-software/images/br_base.img” (used to create the RootFS containing the benchmarks), does not have the hpm_counters binary under /usr/bin/. Is there a particular way to modify the image to be able to support the performance counters?

 

[austin.da...@gmail.com]

Have you tried compiling hpm_counters from (https://github.com/ccelio/riscv-hpmcounters) and putting it in the image yourself? I wasn't too sure from your message if you didn't have the binary or you were just having trouble getting it on the disk image.

Additionally, it looks like the default configs don't have the performance counters enabled, but perhaps a firesim developer can confirm. You might need to rebuild the image with something like:

class WithPerfCounters extends Config((site, here, up) => {

  case RocketTilesKey => up(RocketTilesKey) map (tile => tile.copy(

    core = tile.core.copy(nPerfCounters = XXX)

  ))

})

Austin

[Alon Amid]

As for the memory_stats.csv, you need to make sure you change the profileinterval parameter in your config_runtime.ini to a value that’s different than -1. If you set that value to X, then the memory_stats values will be sampled every X cycles.

 

I’m not familiar enough with the usage of hpm_counters, so I will leave that for someone else to answer, but I would point out that FireSim also has the AutoCounter feature which can possibly be more fine-grained and might be worth looking at (https://docs.fires.im/en/latest/Advanced-Usage/Debugging-and-Profiling-on-FPGA/AutoCounter.html).

 

Finally, I believe there was a recent thread on the Chipyard mailing list that might be relevant: https://groups.google.com/u/2/g/chipyard/c/NBFasBCiR8w

 

 

Shana Tova,

Alon

--
You received this message because you are subscribed to the Google Groups "FireSim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firesim+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/861b93e3-b49e-471e-9180-3e8020ef1dbcn%40googlegroups.com.

[Gururaj S]

Thanks Austin, Alon !

Regarding memory_stats,csv:  I managed to get statistics values in this file by setting the "profileinterval" variable as suggested.

Regarding hpm_counters: I added the hpm_counters binary to the disk image. But when I ran hpm_counters from the FPGA after booting the image, I got the following invalid-instruction error as shown below. 

hpm_counters
[    8.749103] hpm_counters[250]: unhandled signal 4 code 0x1 at 0x0000000000010b42 in hpm_counters[10000+2000]
[    8.749120] CPU: 1 PID: 250 Comm: hpm_counters Tainted: G           O      5.7.0-rc3-58539-g5f5fd87b36e2 #1
[    8.749130] epc: 0000000000010b42 ra : 0000000000010afa sp : 0000003fffc12af0
[    8.749247]  gp : 0000000000013800 tp : 0000003fed3e6b40 t0 : 0000003fed3ebd58
[    8.749473]  t1 : 0000003fed415d54 t2 : 0000000000012860 s0 : 0000000000018000
[    8.749698]  s1 : 0000000000011000 a0 : 00000000008bd6fd a1 : 0000003fffc12b30
[    8.749923]  a2 : 0000000000000000 a3 : 0000000000000008 a4 : 0000000000000000
[    8.750149]  a5 : 0000000000010f10 a6 : 0000000000000000 a7 : 0000000000000087
[    8.750374]  s2 : 0000000000000000 s3 : 00000000000130b8 s4 : 0000000000000000
[    8.750600]  s5 : 00000000000ff7b0 s6 : 0000000000000000 s7 : ffffffffffffffff
[    8.750825]  s8 : 0000000000000000 s9 : 00000000000fe3b0 s10: 00000000000ff730
[    8.751066]  s11: 0000000000000000 t3 : 0000000000000000 t4 : 000000000002dd54
[    8.751276]  t5 : 0000000000000006 t6 : 0000000000000000
[    8.751442] status: 8000000200006020 badaddr: 00000000c0302573 cause: 0000000000000002
Illegal instruction

Perhaps this is because I didn't correctly update the bootloader as suggested by ccelio/riscv-hpmcounters. Should I be updating the /home/centos/firesim/sw/firesim-software/riscv-pk/machine/minit.c in the default set-up? It already appears to have the following lines that conflict with the instructions in the git-repo:

// Enable user/supervisor use of perf counters

  if (supports_extension('S'))
    write_csr(scounteren, -1);

  write_csr(mcounteren, -1);"

Also after building a boot-loader, where should it be placed for the toolchain to use this new bootloader? 

I would really appreciate any instructions on how to get this set up to get to work with the defaulthwconfig=firesim-rocket-quadcore-no-nic-l2-llc4mb-ddr3 . 

Regards,

Gururaj

To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/0cca01d68d3c%242f460340%248dd209c0%24%40berkeley.edu.

[Austin Harris]

I believe this is the error I would expect from the performance counters not being enabled in the core. You should be able to look at the instruction at the faulting PC and double-check.

There isn't a way to work around this with the default AGFI, you have to rebuild with nPerfCounters set in rocket-chip since Howie confirmed (https://groups.google.com/g/chipyard/c/NBFasBCiR8w/m/3YyjjtU0AwAJ) that the default configs don't have the perf counters. This parameter is what creates those registers in the register file, etc. in the RTL.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/CAHg8tWrS%2Bmss4g5daQ6gqvCRGOGS-5oj%3DnxCfnxZD7G9B82eyQ%40mail.gmail.com.

[Gururaj S]

Thanks Austin. Your suggestions were quite helpful. 

A quick update to how I got this working: 

- The perf-counters were indeed not enabled in the default FPGA image. 

- I had to rebuild the FPGA-image (as per this guide), while adding  "WithPerfCounters_" to the config-variable TARGET_CONFIG in config_build_recipes.ini. 

- Now the hpm_counters binary executes and measures the statistics successfully (One caveat - currently, it only measures Execution-Time, Cycles & Instructions. It looks like some additional steps might be needed for the other counters.)

Regards,

Gururaj

You received this message because you are subscribed to a topic in the Google Groups "FireSim" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firesim/hWoqInJK4ME/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firesim+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/CAGW4o-6v_4_9LFAAhq62M%3Dpb5Y3nwyF2HS44HGRYg1sJ5L7Y%2Bg%40mail.gmail.com.