Benchmarking riscv against arm on qemu environment

[Pintu Kumar]

Hi All,

Since we don't have real RISC-V hardware available here in India, we
are preparing QEMU setup for our bench-marking.
We have done basic arm-32 bring up on QEMU, using minimal rootfs.
We have also done basic riscv-64 bring up on QEMU, using minimal buildroot.

Now we wanted to do some basic bench-marking test to compare results
on both architecture.
Currently, I am not sure what kind of tools can be used for
bench-marking with arm on qemu.
Can we bench-mark using:
- Cyclictest
- lm-bench
- LTP
- etc.

I ran some simple pthread latency measurement tests on both qemu arm/riscv.
The results are horrible.
For 100 us sleep duration for each iteration,
Both qemu and arm took ~9000 us, for each sample to complete.

QEMU-RISCV-64:
------------------------
# ./simple_pthread.out 10
main: creating thread name: thread0, priority: 99
Thread: 0, Iteration: 10, Sleep duration: 100 us
Thread[0] - Avg Latency: 9203.689 us
Thread[0] - Max Latency: 9890.700 us
Thread[0] - Min Latency: 3695.400 us
1 3795.400 3695.400
2 9637.200 9537.200
3 9938.800 9838.800
4 9960.800 9860.800
5 9938.200 9838.200
6 9895.100 9795.100
7 9990.700 9890.700
8 9953.800 9853.800
9 9965.300 9865.300
10 9961.600 9861.600
ALL FINISHED.......!!!


QEMU-ARM32:
--------------------------------
# ./simple_pthread.out 10
main: creating thread name: thread0, priority: 99
Thread: 0, Iteration: 10, Sleep duration: 100 us

Thread[0] - Avg Latency: 9201.200 us
Thread[0] - Max Latency: 9426.000 us
Thread[0] - Min Latency: 7852.000 us
1 7952.000 7852.000
2 9113.000 9013.000
3 9526.000 9426.000
4 9434.000 9334.000
5 9481.000 9381.000
6 9504.000 9404.000
7 9491.000 9391.000
8 9497.000 9397.000
9 9506.000 9406.000
10 9508.000 9408.000
ALL FINISHED.......!!!


So, I am wondering what type of tests can be done to compare both
architecture results.

If there is already some bench-marking done, please let me know how
can I re-sue it for my environment.


Thanks,
Pintu

[Sathya Narayanan N]

Hi Pintu,

We have RISCV hardware at IIT Madras. We have taped out 64 bit single core.

And booted linux successfully on it.

Lots of things are happening here. You check at look at our new website: http://shakti.org.in/

Interesting things are there.

regards,

Sathya

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAOuPNLjru1Yr5wnspzFTQbZ33jT0jBu-6Ztdw9E-iXWy%2BnKgWA%40mail.gmail.com.

--

 regards,
Sathya  

 

[Bruce Hoult]

I'm curious what you expect to learn from benchmarking different iSAs on qemu. What decision and different course of action depends on the result?

Certainly, performance in qemu has virtually nothing to do with the performance of real hardware. For exmaple things that are easy in hardware and can be done in parallel often happen in serial in an emulator. An extreme case is the emulation of an MMU. Another is emulation of condition codes ... this is something that slows down ARM emulation a lot, while RISC-V comparing and conditional branching is much easier to emulate quickly.

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.

[Pintu Kumar]

Dear Satya,

Thank you so much for your reply.
Yes, I am already aware about Shakti processor. Its indeed a great job!

Just curious to know, do you also sale RISCV boards (with Linux
support), for hobby or experimental purposes?
Or, do you know any place in India where we can buy RISCV boards?

Since, we don't have real hardware, it will be difficult to run our
bench-marking test and kernel features on QEMU environment.


Thanks,
Pintu

[Pintu Kumar]

On Fri, Aug 24, 2018 at 7:45 PM Bruce Hoult <bruce...@sifive.com> wrote:
>
> I'm curious what you expect to learn from benchmarking different iSAs on qemu. What decision and different course of action depends on the result?
>

Since, we don't have real hardware, we thought to choose qemu for our
initial bring up.
Initially, we just wanted to compare, how each one behaves
differently, for different scenarios.
Like: sleep latency, multi-threading performance, etc.
We have some internal arm Linux kernel specific patches, which we want
to port for riscv.
So, we wanted to early evaluate, if it is worth to do it.

> Certainly, performance in qemu has virtually nothing to do with the performance of real hardware. For exmaple things that are easy in hardware and can be done in parallel often happen in serial in an emulator. An extreme case is the emulation of an MMU. Another is emulation of condition codes ... this is something that slows down ARM emulation a lot, while RISC-V comparing and conditional branching is much easier to emulate quickly.
>

Yes, I know, its a pain to do performance measure on qemu. But until
real hardware is available to us, we need to start with something.
That is the reason we are using qemu on both side (qemu-arm and
qemu-riscv, with same kernel and rootfs).
Then we will only consider the delta latency difference on both.
But, the main point is that, we don't know what kind of test/tools can
be used for basic bench-marking for both arch.
Can you list time some of the existing tools used for bench-marking ?

>> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

[Bruce Hoult]

On Mon, Aug 27, 2018 at 12:58 AM, Pintu Kumar <pintu...@gmail.com> wrote:

On Fri, Aug 24, 2018 at 7:45 PM Bruce Hoult <bruce...@sifive.com> wrote:
>
> I'm curious what you expect to learn from benchmarking different iSAs on qemu. What decision and different course of action depends on the result?
>
Since, we don't have real hardware, we thought to choose qemu for our
initial bring up.
Initially, we just wanted to compare, how each one behaves
differently, for different scenarios.
Like: sleep latency, multi-threading performance, etc.
We have some internal arm Linux kernel specific patches, which we want
to port for riscv.
So, we wanted to early evaluate, if it is worth to do it.

qemu is useful to get software working on.

It is very unlikely you'll learn anything useful about the performance of actual hardware from it. It doesn't even try to emulate performance -- qemu just runs as fast as it can.

If you want to learn something about performance you need a cycle-accurate emulator. And you'll need to have a model of whatever microarchitecture you want to find the performance of. I think some people are marketing those already, but lacking anything else (or money to buy them) you could use verilator. It's much slower than qemu, but so are all emulators.

As for Shakti, I believe they only have a prototype chip at the moment, not anything for sale. I don't know of anyone other than SiFive who have real Linux-capable RISC-V silicon for sale to the general public.