Measuring L1/L2 cache hits and misses

[Siyi Hu]

Hi all,

Is there any way to measure how many L1 and L2 cache hits/misses has the system made?

I am now using RoCC as our accelerator and it seems the RoCC interface does not support L1 cache hit/miss measurement.

Besides, SiFive inclusive cache module neither does not set up ports or signals for counting L2 cache hits and misses.

Any help would be greatly appreciated.

Best regards,

Siyi

[Howard Mao]

Hi Siyi,

I'm not sure how to go about measuring L1 cache hits from RoCC specifically, but if you want to count all the L1D hits/misses on a core, you can set up an event counter. To do this, you'll need to set one of the HPM event CSRS (mhpmevent3-mhpmevent31) and then read from the corresponding HPM counter CSR (mhpmcounter3-mhpmcounter31). If you're using rocket, the list of events is listed here:

https://github.com/chipsalliance/rocket-chip/blob/1872f5d501221f13950aa2293939634a1e0d1735/src/main/scala/rocket/RocketCore.scala#L122

The different events are grouped into event sets. To specify which events a counter should track, you write the mphmevent CSR with a selector. The eight least significant bits of the selector form an integer specifying which event set to use. The upper bits are a bitmask selecting events in that set. So to count all of the memory requests to the L1D, you set it to 0xe00 (0 is the event set, 0xe selects the load, store, and AMO counters). To count the number of L1D misses, you'd set it to 0x202.

For L2 misses, there's no architectural way of doing it. But you could approximate it pretty closely by counting the number of AXI4 read requests that get sent during simulation.

-- Howie

--
You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/41a2b40f-f68e-49f2-8295-e563442cf532o%40googlegroups.com.

[Siyi Hu]

Hi Howie,

Thank you very much for your advice and sorry for replying this late.

It really took me some time on trying to get mhpmcounter CSRs work, however I still got a problem.

I've tried to activate the performance monitors by writing CSRs in my baremetal software workload as follows:

    start = read_csr(mcycle);                                   /* csrr  rd, mcycle        */

    write_csr(mhpmevent3, 0xe00);                               /* csrwi mhpmevent3, 0xe00 */

    printf("- mhpmevent3: 0x%lx \n", read_csr(mhpmevent3));     /* csrr  rd, mhpmevent3    */

    computation();

    end = read_csr(mcycle);

    printf("- Total cylces: %lu\n", end - start);

    printf("- instret: %lu\n", read_csr(minstret));             /* csrr  rd, minstret      */

    printf("- mhpmcounter3: 0x%lx \n", read_csr(mhpmcounter3)); /* csrr  rd, mhpmcounter3  */

The result somehow looks like:

- mhpmevent3: 0x0
- Total cylces: 32571

- instret: 30387

- mhpmcounter3: 0x0

I am wondering whether the core's running mode caused this problem. Is it necessary to manually set the Rocket core to run on M-mode in order to enable these HPM event CSRs?

With best regards,

Siyi

2020年7月2日木曜日 12時55分16秒 UTC+9 zhemao:

[Howard Mao]

Sorry, I forgot that the default core configurations don't have performance counters enabled. Try adding the chipyard.config.WithNPerfCounters config fragment to your configuration. That should make your code work. 

--

You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/648a9fe5-65c7-456a-bf1c-50ed1cd407a8o%40googlegroups.com.

[Siyi Hu]

Hi Howie,

I think the code works exactly fine now by adding this config thanks to your help.

Much appreciated again to you for these advice. They really helped me a lot.

Best regards,

Siyi

2020年7月9日木曜日 9時20分57秒 UTC+9 zhemao:

To unsubscribe from this group and stop receiving emails from it, send an email to chip...@googlegroups.com.

[husi...@gmail.com]

Hi Howie,

Terribly sorry for reopening this question after it has been posted for 2 months. (I left this work because of some mental problems and just came back on track)

I tried to figure out how to measure AXI4 read requests on a Rocket Chip system this week, and it seems there is few existing profiling tools that may be able to do this.

I wonder if I need to go checking the source under https://github.com/chipsalliance/rocket-chip/tree/master/src/main/scala/amba/axi4 and implement the request counter by myself.

It would be greatly appreciated if you could give me some hints on this.

With best regards,

Siyi

2020年7月2日木曜日 12:55:16 UTC+9 zhemao:

[Howard Mao]

Are you simulating using FireSim or VCS? In FireSim, we already have a DRAM profiling tool that can get the counts for you. In VCS, I would suggest adding a hardware printf statement that outputs every time a request is sent. You can then post-process the simulation log with a script to count the number of requests. Since you likely want to see the L2 miss rate and not just the number of misses, what you probably want to do is have one kind of print statement for the inner interface and one for the outer interface. So in your Chipyard repo, open up this file in generators/sifive-cache https://github.com/sifive/block-inclusivecache-sifive/blob/master/design/craft/inclusivecache/src/InclusiveCache.scala#L188

Then add the following lines

when (in.a.fire()) {

    printf("inner acquire")

}

when (out.a.fire()) {

    printf("outer acquire")

}

The first printf statement will trigger on every L2 request, and the second printf statement will only trigger on misses.

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/cc8009f9-46a4-421e-8141-a2f6a6f4b746n%40googlegroups.com.

[husi...@gmail.com]

Hi Howie,

Thanks for the reply. I am currently using Verilator, which has the same simulation environment as VCS. 

It is very helpful to know how to modify sifive-cache module. I'll give it a try as soon as possible.

Actually we are moving our simulation onto FireSim within this month, so it is great to know that DRAM profiling tool is available on this platform.

Thank you so much for your advices!

With best regards,

Siyi

2020年9月16日水曜日 6:13:26 UTC+9 zhemao:

[Ayushi Agarwal]

Hi Siyi/Howie,

I was trying to replicate this into my config file. To enable HPM counters, I add

chipyard.config.WithNPerfCounters ++

in the .scala config file. But when I build the config, it says that the chipyard config does not have any config fragment by this name.

Is there something else that I am missing?

[Howard Mao]

Well that's very strange. Which commit of chipyard are you using? Can you see the WithNPerfCounters class in generators/chipyard/src/main/scala/ConfigFragments.scala?

To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/b53defbf-8162-4605-b80a-7f78d96bed54n%40googlegroups.com.

[Ayushi Agarwal]

I could not find the WithNPerfCounters class on the config fragments file (at the path you've mentioned). It seemed strange to me too and hence I dropped an email.

I am using the master branch of chipyard. The output of git log is:

commit 19152d3b73f1c2255ce378501f6c01f9363b75e9 (HEAD -> master, origin/master, origin/HEAD)
Author: alonamid <alon...@eecs.berkeley.edu>
Date:   Mon Jul 6 20:29:34 2020 -0700

Thanks and Regards, 

Ayushi Agarwal

PhD Student, Khosla School of IT, 

IIT, Delhi

http://www.cse.iitd.ernet.in/~anz188503/

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/CAHCr_CBxZKi1fjDzKBjpstDVk%3DkeYUZV0x-md1dSbJfAN_-ftA%40mail.gmail.com.

[husi...@gmail.com]

Hi Ayushi

I guess all you need to do is to add the following code to your generators/chipyard/src/main/scala/ConfigFragments.scala.

class WithNPerfCounters(n: Int) extends Config((site, here, up) => {

  case RocketTilesKey => up(RocketTilesKey) map (tile => tile.copy(

    core = tile.core.copy(nPerfCounters = n)

  ))

  case BoomTilesKey => up(BoomTilesKey) map (tile => tile.copy(

    core = tile.core.copy(nPerfCounters = n)

  ))

})

BTW, I am using chipyard 1.2.0 and above code is copied from generators/boom/src/main/scala/common/config-mixins.scala.

It seems performance counters are only configured in BOOM but not in Rocket. Well it's not a big problem :).

Hope this would help.

With best regards,

Siyi

2020年10月10日土曜日 14:49:42 UTC+9 ayushi...@gmail.com:

[Ayushi Agarwal]

Hi Siyi,

Actually I tried this. But I think it demands the value of 'n' from the top config. So,

If I did boom.common.WithNPerfCounters ++ in the config right above the boom, then I get an error saying that the value of n is missing. Now I was not able to figure out, what would be the value of 'n', it would be predetermined right?

I just started using this framework (and frankly I just want a quick assessment on stats for SW RTL simulation before diving into the depth), so pardon me for my naivety.

Thanks and Regards, 

Ayushi Agarwal

PhD Student, Khosla School of IT, 

IIT, Delhi

http://www.cse.iitd.ernet.in/~anz188503/

--
You received this message because you are subscribed to a topic in the Google Groups "Chipyard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chipyard/NBFasBCiR8w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/47520295-229d-4898-a57f-5ed05a7bb8d5n%40googlegroups.com.

[husi...@gmail.com]

Hi Ayushi,

I think the error message is telling that you missed the parameter 'n' to construct this class. 

So instead of "boom.common.WithNPerfCounters ++", you may try writing "chipyard.config.WithNPerfCounters(29) ++", where 29 indicates you are planning to use all 29 performance counters defined in RISC-V spec.

With best regards,

Siyi

2020年10月10日土曜日 21:40:47 UTC+9 ayushi...@gmail.com:

[husi...@gmail.com]

Hi Howie,

I have moved our design onto FireSim and I'm currently trying to figure out how to get the DRAM profiling tool counts.

I attached a uartlog generated by an example Gemmini test in this mail. It seems that there is little information about DRAM.

I followed the instructions on Chapter 9 (Debugging and Profiling on the FPGA) in the FireSim doc and have already turned the tracing option on.

Is there any other switches I need to get in order to use the DRAM profiling tool?

With best regards,

Siyi

2020年9月16日水曜日 6:13:26 UTC+9 zhemao:

Are you simulating using FireSim or VCS? In FireSim, we already have a DRAM profiling tool that can get the counts for you. In VCS, I would suggest adding a hardware printf statement that outputs every time a request is sent. You can then post-process the simulation log with a script to count the number of requests. Since you likely want to see the L2 miss rate and not just the number of misses, what you probably want to do is have one kind of print statement for the inner interface and one for the outer interface. So in your Chipyard repo, open up this file in generators/sifive-cache https://github.com/sifive/block-inclusivecache-sifive/blob/master/design/craft/inclusivecache/src/InclusiveCache.scala#L188

[zheha...@gmail.com]

Hi Siyi,

The DRAM profiling information will be put in a memory_stats.csv file. If you ssh onto the F1 instance, you should be able to find it in the sim_slot_0 directory. 

-- Howie

[husi...@gmail.com]

Hi Howie,

Thanks very much for the reply. I think memory_stats.csv file just works fine for me at present.

Much appreciates to your helps.

With best regards,

Siyi

2020年10月20日火曜日 1:11:14 UTC+9 zheha...@gmail.com: