qemu-riscv32

[David H. Lynch Jr.]

What is the current state of riscv32 support in the risc-v qemu
repository ?

Will it execute 32 bit Risc-v code ?
Does it have linux userspace support ?

i.e. can I compile hello.c and expect it to be executed under qemu-
riscv32 without booting a linux under qemu ?

[Stefan O'Rear]

Yes. It may be a bit bugger than 64-bit, but it works for me right now.

-s

[David H. Lynch Jr.]

On Sun, 2018-02-11 at 21:11 -0800, Stefan O'Rear wrote:
> On Sun, Feb 11, 2018 at 9:02 PM, David H. Lynch Jr. <dhlynch2@gmail.c

> om> wrote:
> > What is the current state of riscv32 support in the risc-v qemu
> > repository ?
> >
> > Will it execute 32 bit Risc-v code ?
> > Does it have linux userspace support ?
> >
> > i.e. can I compile hello.c and expect it to be executed under qemu-
> > riscv32 without booting a linux under qemu ?
>
> Yes.  It may be a bit bugger than 64-bit, but it works for me right
> now.
>
> -s
>

Thanks - I have been able to get riscv32 toolchain, spike and gemu all
working and running "hello world".

I am now trying to get a minimal riscv32 Linux kernel to boot under
either spike or qemu, thus far no joy.

[Michael Clark]

There is at least one gotcha that I am aware of. You will need to re-enable HTIF for 32-bit hosts. See this piece of code:

https://github.com/riscv/riscv-pk/blob/e5846a2bc707eaa58dc8ab6a8d20a090c6ee8570/machine/htif.h#L6-L13

As far as I can tell, QEMU should be able to handle split 32-bit writes to the 64-bit HTIF registers however I haven’t tested it.

Essentially you would remove the #ifdef and use the same definition of TOHOST_CMD. As I said, I have not tested it. I am not sure if the compiler makes any guarantees with respect to the order of writes however one would imagine the low word, which is the least significant word on a little-endian host, meaning the device and command would be stored afterwards. Of course there is no barriers (memory fences) so this is based on an assumption of in-order execution. Really HTIF needs special consideration for 32-bit hosts with appropriate memory barriers. From reading the qemu/hw/riscv/riscv_htif.c, it seems that the driver is written with respect for split writes for double words.

It would be nice to document and harden the HTIF two-register communication protocol, including specifying 32-bit and 64-bit operation…

You might want to start with a simple program to test. I just uploaded riscv-probe which proves that the QEMU HTIF emulation works with 32-bit, so theoretically we could submit a PR to remove the ifdef from riscv-pk. See results here (note the same binaries will run in spike):

- https://github.com/michaeljclark/riscv-probe

$ qemu-system-riscv64 -nographic -machine spike_v1.10 -kernel bin/riscv64/probe-htif
isa: rv64imafdcsu
csr: fflags (not supported) cause=2
csr: frm (not supported) cause=2
csr: fcsr (not supported) cause=2
csr: cycle 0x0000000000009236
csr: time 0x0000000000009cf4
csr: instret 0x000000000000a41a
csr: cycleh (not supported) cause=2
csr: timeh 0x0000000000000000
csr: instreth (not supported) cause=2
csr: mvendorid 0x0000000000000000
csr: marchid 0x0000000000000000
csr: mimpid 0x0000000000000000
csr: mhartid 0x0000000000000000
csr: mstatus 0x0000000000000000
csr: misa 0x800000000014112d
csr: medeleg 0x0000000000000000
csr: mideleg 0x0000000000000000
csr: mie 0x0000000000000000
csr: mtvec 0x0000000080000004
csr: mcounteren 0x0000000000000000
csr: mscratch 0x0000000000000000
csr: mepc 0x000000008000081e
csr: mcause 0x0000000000000002
csr: mip 0x0000000000000000
csr: sstatus 0x0000000000000000
csr: sie 0x0000000000000000
csr: stvec 0x0000000000000000
csr: scounteren 0x0000000000000000
csr: sscratch 0x0000000000000000
csr: sepc 0x0000000000000000
csr: scause 0x0000000000000000
csr: sip 0x0000000000000000

$ qemu-system-riscv32 -nographic -machine spike_v1.10 -kernel bin/riscv32/probe-htif
isa: rv32imafdcsu
csr: fflags (not supported) cause=2
csr: frm (not supported) cause=2
csr: fcsr (not supported) cause=2
csr: cycle 0x00008458
csr: time 0x00008d36
csr: instret 0x000093b2
csr: cycleh (not supported) cause=2
csr: timeh 0x00000000
csr: instreth (not supported) cause=2
csr: mvendorid 0x00000000
csr: marchid 0x00000000
csr: mimpid 0x00000000
csr: mhartid 0x00000000
csr: mstatus 0x00000000
csr: misa 0x4014112d
csr: medeleg 0x00000000
csr: mideleg 0x00000000
csr: mie 0x00000000
csr: mtvec 0x80000004
csr: mcounteren 0x00000000
csr: mscratch 0x00000000
csr: mepc 0x80000a14
csr: mcause 0x00000002
csr: mip 0x00000000
csr: sstatus 0x00000000
csr: sie 0x00000000
csr: stvec 0x00000000
csr: scounteren 0x00000000
csr: sscratch 0x00000000
csr: sepc 0x00000000
csr: scause 0x00000000
csr: sip 0x00000000

[Michael Clark]

> On 13/02/2018, at 12:44 PM, Michael Clark <michae...@mac.com> wrote:
>
>
>
>> On 13/02/2018, at 10:41 AM, David H. Lynch Jr. <dhly...@gmail.com> wrote:
>>
>> On Sun, 2018-02-11 at 21:11 -0800, Stefan O'Rear wrote:
>>> On Sun, Feb 11, 2018 at 9:02 PM, David H. Lynch Jr. <dhlynch2@gmail.c
>>> om> wrote:
>>>> What is the current state of riscv32 support in the risc-v qemu
>>>> repository ?
>>>>
>>>> Will it execute 32 bit Risc-v code ?
>>>> Does it have linux userspace support ?
>>>>
>>>> i.e. can I compile hello.c and expect it to be executed under qemu-
>>>> riscv32 without booting a linux under qemu ?
>>>
>>> Yes. It may be a bit bugger than 64-bit, but it works for me right
>>> now.
>>>
>>> -s
>>>
>>
>> Thanks - I have been able to get riscv32 toolchain, spike and gemu all
>> working and running "hello world".
>>
>> I am now trying to get a minimal riscv32 Linux kernel to boot under
>> either spike or qemu, thus far no joy.
>
> There is at least one gotcha that I am aware of. You will need to re-enable HTIF for 32-bit hosts. See this piece of code:
>
> https://github.com/riscv/riscv-pk/blob/e5846a2bc707eaa58dc8ab6a8d20a090c6ee8570/machine/htif.h#L6-L13
>
> As far as I can tell, QEMU should be able to handle split 32-bit writes to the 64-bit HTIF registers however I haven’t tested it.

I have tested it. See below. I hadn’t tested it until I got until nearly finishing writing this email :-D

> --
> You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
> To post to this group, send email to sw-...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/A6988721-774E-460E-90C8-C8A35EC86298%40mac.com.

[Michael Clark]

> On 13/02/2018, at 12:45 PM, Michael Clark <michae...@mac.com> wrote:
>
>
>
>> On 13/02/2018, at 12:44 PM, Michael Clark <michae...@mac.com> wrote:
>>
>>
>>
>>> On 13/02/2018, at 10:41 AM, David H. Lynch Jr. <dhly...@gmail.com> wrote:
>>>
>>> On Sun, 2018-02-11 at 21:11 -0800, Stefan O'Rear wrote:
>>>> On Sun, Feb 11, 2018 at 9:02 PM, David H. Lynch Jr. <dhlynch2@gmail.c
>>>> om> wrote:
>>>>> What is the current state of riscv32 support in the risc-v qemu
>>>>> repository ?
>>>>>
>>>>> Will it execute 32 bit Risc-v code ?
>>>>> Does it have linux userspace support ?
>>>>>
>>>>> i.e. can I compile hello.c and expect it to be executed under qemu-
>>>>> riscv32 without booting a linux under qemu ?
>>>>
>>>> Yes. It may be a bit bugger than 64-bit, but it works for me right
>>>> now.
>>>>
>>>> -s
>>>>
>>>
>>> Thanks - I have been able to get riscv32 toolchain, spike and gemu all
>>> working and running "hello world".
>>>
>>> I am now trying to get a minimal riscv32 Linux kernel to boot under
>>> either spike or qemu, thus far no joy.
>>
>> There is at least one gotcha that I am aware of. You will need to re-enable HTIF for 32-bit hosts. See this piece of code:
>>
>> https://github.com/riscv/riscv-pk/blob/e5846a2bc707eaa58dc8ab6a8d20a090c6ee8570/machine/htif.h#L6-L13

This is the riscv-pk/bbl patch:

diff --git a/machine/htif.h b/machine/htif.h
index a96bf60..802ed60 100644
--- a/machine/htif.h
+++ b/machine/htif.h
@@ -3,14 +3,8 @@

#include <stdint.h>

-#if __riscv_xlen == 64
-# define TOHOST_CMD(dev, cmd, payload) \
+#define TOHOST_CMD(dev, cmd, payload) \
(((uint64_t)(dev) << 56) | ((uint64_t)(cmd) << 48) | (uint64_t)(payload))
-#else
-# define TOHOST_CMD(dev, cmd, payload) ({ \
- if ((dev) || (cmd)) __builtin_trap(); \
- (payload); })
-#endif
#define FROMHOST_DEV(fromhost_value) ((uint64_t)(fromhost_value) >> 56)
#define FROMHOST_CMD(fromhost_value) ((uint64_t)(fromhost_value) << 8 >> 56)
#define FROMHOST_DATA(fromhost_value) ((uint64_t)(fromhost_value) << 16 >> 16)

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/8A67737F-7980-4854-AF21-03628B68B7FA%40mac.com.

[Stefan O'Rear]

On Mon, Feb 12, 2018 at 3:44 PM, Michael Clark <michae...@mac.com> wrote:
>
>
>> On 13/02/2018, at 10:41 AM, David H. Lynch Jr. <dhly...@gmail.com> wrote:
>> I am now trying to get a minimal riscv32 Linux kernel to boot under
>> either spike or qemu, thus far no joy.
>
> There is at least one gotcha that I am aware of. You will need to re-enable HTIF for 32-bit hosts. See this piece of code:
>
> https://github.com/riscv/riscv-pk/blob/e5846a2bc707eaa58dc8ab6a8d20a090c6ee8570/machine/htif.h#L6-L13

How close is the htif-less virt board to working?

-s

[Michael Clark]

I haven’t tried booting linux in qemu-system-riscv32. The 16550a uses byte accesses, and bbl now supports the 16550a, so it should work unmodified.

It should work okay assuming sv32 is working fine. We should test it…

I’d first compile riscv-pk/bbl with --enable-print-device-tree to check that the console system is working.

[Michael Clark]

BTW I just found a bug that was present in QEMU and is present in Spike, likely because the CSR code in QEMU comes from Spike.

rdcycle is implemented but rdcycleh is not implemented, on rv32
rdinstret is implemented but rdinstreth is not implemented, on rv32

mcycle and mcycleh are both okay, on rv32
minstret and minstreth are both okay, on rv32

Notice the missing case statements for CSR_CYCLEH and CSR_INSTRETH

- https://github.com/riscv/riscv-isa-sim/blob/master/riscv/processor.cc#L560-L572

However, the definitions are present here:

- https://github.com/riscv/riscv-isa-sim/blob/master/riscv/encoding.h#L899-L901

I understand rdtime/rdtimeh can be emulated, but implementing rdcycle/rdinstret but not rdcycleh/rdinstreth seems like a bug.

I’ve updated riscv-probe to add a few more CSRs (currently missing are the performance counters):

- https://github.com/michaeljclark/riscv-probe

> csr: mvendorid 0x00000000
> csr: marchid 0x00000000
> csr: mimpid 0x00000000
> csr: mhartid 0x00000000
> csr: mstatus 0x00000000
> csr: misa 0x4014112d
> csr: medeleg 0x00000000
> csr: mideleg 0x00000000
> csr: mie 0x00000000
> csr: mtvec 0x80000004
> csr: mcounteren 0x00000000
> csr: mscratch 0x00000000
> csr: mepc 0x80000a14
> csr: mcause 0x00000002
> csr: mip 0x00000000
> csr: sstatus 0x00000000
> csr: sie 0x00000000
> csr: stvec 0x00000000
> csr: scounteren 0x00000000
> csr: sscratch 0x00000000
> csr: sepc 0x00000000
> csr: scause 0x00000000
> csr: sip 0x00000000
>

[Michael Clark]

>>
>> $ qemu-system-riscv32 -nographic -machine spike_v1.10 -kernel bin/riscv32/probe-htif
>> isa: rv32imafdcsu
>> csr: fflags (not supported) cause=2
>> csr: frm (not supported) cause=2
>> csr: fcsr (not supported) cause=2
>> csr: cycle 0x00008458
>> csr: time 0x00008d36
>> csr: instret 0x000093b2
>> csr: cycleh (not supported) cause=2
>> csr: timeh 0x00000000
>> csr: instreth (not supported) cause=2
>
> BTW I just found a bug that was present in QEMU and is present in Spike, likely because the CSR code in QEMU comes from Spike.
>
> rdcycle is implemented but rdcycleh is not implemented, on rv32
> rdinstret is implemented but rdinstreth is not implemented, on rv32
>
> mcycle and mcycleh are both okay, on rv32
> minstret and minstreth are both okay, on rv32
>
> Notice the missing case statements for CSR_CYCLEH and CSR_INSTRETH
>
> - https://github.com/riscv/riscv-isa-sim/blob/master/riscv/processor.cc#L560-L572
>
> However, the definitions are present here:
>
> - https://github.com/riscv/riscv-isa-sim/blob/master/riscv/encoding.h#L899-L901
>
> I understand rdtime/rdtimeh can be emulated, but implementing rdcycle/rdinstret but not rdcycleh/rdinstreth seems like a bug.
>
> I’ve updated riscv-probe to add a few more CSRs (currently missing are the performance counters):
>
> - https://github.com/michaeljclark/riscv-probe

cycleh and instreth are missing, but cycle and instret are present. I’ve just patched riscv-qemu to fix the same issue. see output below

I understand time/timeh are emulated by bbl.

$ spike --isa=rv32g bin/riscv32/probe-htif
isa: rv32imafdsu

csr: fflags (not supported) cause=2
csr: frm (not supported) cause=2
csr: fcsr (not supported) cause=2

csr: mcycle 0x000b4e08
csr: minstret 0x000dbf08
csr: mcycleh 0x00000000
csr: minstreth 0x00000000
csr: cycle 0x00151207
csr: time (not supported) cause=2
csr: instret 0x001adf4e

csr: cycleh (not supported) cause=2

csr: timeh (not supported) cause=2

csr: instreth (not supported) cause=2

csr: mvendorid 0x00000000
csr: marchid 0x00000000
csr: mimpid 0x00000000
csr: mhartid 0x00000000

csr: mstatus 0x00000080
csr: misa 0x40141129

csr: medeleg 0x00000000
csr: mideleg 0x00000000
csr: mie 0x00000000
csr: mtvec 0x80000004
csr: mcounteren 0x00000000
csr: mscratch 0x00000000

csr: mepc 0x80000a64
csr: mcause 0x00000002
csr: mtval 0x00000000
csr: mip 0x00000080
csr: sstatus 0x00000000
csr: sedeleg (not supported) cause=2
csr: sideleg (not supported) cause=2

csr: sie 0x00000000
csr: stvec 0x00000000
csr: scounteren 0x00000000
csr: sscratch 0x00000000
csr: sepc 0x00000000
csr: scause 0x00000000

csr: stval 0x00000000
csr: sip 0x00000000
csr: satp 0x00000000

$ qemu-system-riscv32 -nographic -machine spike_v1.10 -kernel bin/riscv32/probe-htif
isa: rv32imafdcsu
csr: fflags (not supported) cause=2
csr: frm (not supported) cause=2
csr: fcsr (not supported) cause=2

csr: mcycle 0xdf6ab15b
csr: minstret 0xdf74d431
csr: mcycleh 0x0004fa70
csr: minstreth 0x0004fa70
csr: cycle 0xdf867386
csr: time 0x0000aec4
csr: instret 0xdf922922
csr: cycleh 0x0004fa70
csr: timeh 0x00000000
csr: instreth 0x0004fa70

csr: mvendorid 0x00000000
csr: marchid 0x00000000
csr: mimpid 0x00000000
csr: mhartid 0x00000000
csr: mstatus 0x00000000
csr: misa 0x4014112d
csr: medeleg 0x00000000
csr: mideleg 0x00000000
csr: mie 0x00000000
csr: mtvec 0x80000004
csr: mcounteren 0x00000000
csr: mscratch 0x00000000

csr: mepc 0x8000099c
csr: mcause 0x00000002
csr: mtval 0x00000000

csr: mip 0x00000000
csr: sstatus 0x00000000

csr: sedeleg (not supported) cause=2
csr: sideleg (not supported) cause=2

csr: sie 0x00000000
csr: stvec 0x00000000
csr: scounteren 0x00000000
csr: sscratch 0x00000000
csr: sepc 0x00000000
csr: scause 0x00000000

csr: stval 0x00000000
csr: sip 0x00000000
csr: satp 0x00000000

[David H. Lynch Jr.]

Thank you all for the help.

On Tue, 2018-02-13 at 13:27 +1300, Michael Clark wrote:
> > On 13/02/2018, at 1:11 PM, Stefan O'Rear <sor...@gmail.com> wrote:
> >

> > On Mon, Feb 12, 2018 at 3:44 PM, Michael Clark <michaeljclark@mac.c
> > om> wrote:
> > >
> > >
> > > > On 13/02/2018, at 10:41 AM, David H. Lynch Jr. <dhlynch2@gmail.