How to perform performance testing

[Nicholas McDonald US]

What is the prescribed way to perform performance testing in Chisel? I have a Chisel project where there exists a base class that defines a device's interface and many derived classes that define implementations. Each implementation may have a different efficiency. I'd like to create a benchmark for the base class and feed it any derived instance to get back a performance evaluation. This should be like a unit test where I poke the inputs and peek on the outputs, but I don't think this should be defined as a unit test as it may take much longer that a unit test and should write the results to a file for later processing. 

Any thoughts?

[Øyvind Harboe]

You could elaborate Verilog and use Verilator directly?

You would have to create a C++ program to wiggle the inputs of your design and read the output. You can then mock your dependencies of the DUT in C++.

[Martin Schoeberl]

First define what your performance measure is. Speed in fmax? Chip area? Number of clock cycles? You can only measure the last one directly with a Chisel test. For the others you need to synthesize your design. Be aware that designs with low number of clock cycles might result in a bad fmax. These are (some of) the design tradeoffs in digital design.

cheers,

Martin

On 18 Nov, 2021, at 22:18, 'Nicholas McDonald US' via chisel-users <chisel...@googlegroups.com> wrote:

What is the prescribed way to perform performance testing in Chisel? I have a Chisel project where there exists a base class that defines a device's interface and many derived classes that define implementations. Each implementation may have a different efficiency. I'd like to create a benchmark for the base class and feed it any derived instance to get back a performance evaluation. This should be like a unit test where I poke the inputs and peek on the outputs, but I don't think this should be defined as a unit test as it may take much longer that a unit test and should write the results to a file for later processing. 

Any thoughts?

--
You received this message because you are subscribed to the Google Groups "chisel-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chisel-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chisel-users/299a3710-49d6-4afc-b169-f9157ae60102n%40googlegroups.com.

[Bruno Ferres]

Just to give some insights about my work (as it could help here), we are currently working on an estimation/exploration framework integrated in Chisel which is built to enable users to define custom metrics of interests in the flow, including synthesis, simulation and/or FIRRTL based estimations.

It won't automatically provide a solution for such use case, but should help you to implement whatever you call "efficiency" here

It is not yet available, but I'll let the community know once it is - it should not be long as I'm supposed to defend my thesis in a few weeks.

Regards,

Bruno

To view this discussion on the web visit https://groups.google.com/d/msgid/chisel-users/FB59955F-5F4C-4658-BC4B-F70791A6A159%40jopdesign.com.

[Øyvind Harboe]

Very interesting! Good luck!

You received this message because you are subscribed to a topic in the Google Groups "chisel-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chisel-users/Iw9nAPXPgkE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chisel-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chisel-users/CAFW_6JULwj-qBfKPs02ogOnt1sSoSWUWORZ587TqHuy-ChuArQ%40mail.gmail.com.

[Nicholas McDonald US]

In terms of performance, I do not mean to measure clock frequency or chip area. I can find these out using synthesis and place-n-route. For performance, it is efficiency of the design. For example, let's assume I'm designing a processor. Performance would be giving it a binary to execute and discover the "efficiency" of the design. 

I see 2 options:

1. Export verilog, use a verilog simulator to benchmark design efficiency. This means I need to write my benchmark suite in verilog. If the verilog simulator supports another language (e.g., C++), I could also use that language.

2. Use Chisel test to benchmark the efficiency. This is preferred for productivity but I can't figure out how to get it into the flow. 

My question was aimed at #2. 

[Øyvind Harboe]

Verilator grew out of this usecase. I think Verilator originated oit of the Alpha CPU simulation.

For a CPU, you can write some C++ that mocks the top level signals of your CPU: clock, reset, memory and memory mapped IO. then you link it with the C++ library that Verilator generates.

I have done this: works very well. Verilator has the best performance available, commercial or not, for this usecase.

You can also do dependency injection (using Chisel BoringUtils) to decide which parts of your CPU you want to run in simulated verilog and which you want to do in c++.

So if you dont have an MMU you can mock it(emulate it in c++) while you work on the verilog/chisel implementation.

To view this discussion on the web visit https://groups.google.com/d/msgid/chisel-users/234ebbc4-cf49-477a-aea9-bf3c685965b3n%40googlegroups.com.

[Nicholas McDonald US]

Seems like the ideal set up would be Chisel integration with Verilator where the Chisel code could drive the Verilator I/Os. 

[Øyvind Harboe]

It depends on your use-case.

I've used both actually. 

If you are able to articulate some machine code for your CPU using Chiseltest, then that's a nice fast turnaround integration test in Chiseltest.

If you have more stuff you need to do, like read in .elf files handle symbols, run larger programs, etc. C++ is probably a better match from a performance and integration point of view.

[Bruno Ferres]

I've been working on this to extract some data from simulations, i've used wrappers around the PeekPokeTester construct to specify in the test benches some variable of interests (eg. the difference between hw results and a given golden reference), then I can access to those variables and use it in my framework through simulation runs (either with verilator or treadle)

I don't know if this can apply to tour use case though

Regards,

Bruno

To view this discussion on the web visit https://groups.google.com/d/msgid/chisel-users/db92d1a3-9cb0-4367-ba34-b4861382f22en%40googlegroups.com.