RISC-V QEMU Testing on 'riscv-next' and upstream patch review feedback

[Michael Clark]

Hi Folks,

Just a quick note about RISC-V QEMU changes, active branches and upstream patch review feedback.

We've spent a reasonable amount of time forward porting RISC-V QEMU changes to upstream QEMU. Given the last sync was in September 2016, there were several relatively large breaking changes to various internal APIs that required a resonable amount of work to get back in sync. These were resolved and we now have a stable port against upstream QEMU. Each subsequent version of the patch series will be put on a qemu-upstream-v<n> branch until we are eventually merged upstream and we'll post patches to the RISC-V Patches mailing list. The first patch series was generated from this branch:

- https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v1

While we were still testing the upstream branch we spent some time back-porting changes to 'master' via 'riscv-next' to keep the two branches in sync as 'master' is the current stable version. The riscv-qemu 'master' branch is still based on the September 9, 2016 sync with upstream QEMU and the 'riscv-next' branch previously held changes that were destined for 'master'.

We have now had a reasonable amount of testing feedback from the upstream branch so we have promoted 'qemu-upstream-v1' to 'riscv-next' with the plan promote 'riscv-next' to master in the not too distant future. The 'riscv-next' branch will be where we stage changes that have undergone testing before merge or promotion to master. The 'riscv-next' branch is now pretty much in sync with upstream QEMU and we would appreciate folk try the 'riscv-next' branch and give feedback on "regressions" so that we can promote it to master as soon as possible. Note: we don't classify a missing feature on a new machine as a "regression". i.e. the "virt" machine does not yet support power-off and reset, however, given there was no RISC-V "virt" machine beforehand, this is not a regression.

- https://github.com/riscv/riscv-qemu/tree/riscv-next

Given the eventual promotion of 'riscv-next' to 'master' I have created archive branches to retain the project history, as during upstreaming we had to squash history into a series of logical commits that can be submitted upstream. The archive branch names are based on the date the branch was rebased against upstream. We currently have two archive branches:

- https://github.com/riscv/riscv-qemu/tree/qemu-20160927

- https://github.com/riscv/riscv-qemu/tree/qemu-20171222

We've already got a lot of high-quality feedback from submitting the patch series to upstream QEMU so the feedback from the v1 patch series has been highly useful and will aid in making the RISC-V QEMU port a high quality architecture in QEMU. I've started making a log of changes to incorporate into the next patch series that we will likely send out sometime in January or February. There is a lot of clean-up work that needs to be performed i.e. eradicating all instances of printf and exit, and instead using the appropriate assertions (g_assert_not_reached) or log APIS (qemu_log_mask). There are also many printfs that should be converted to use the trace API. We have to balance code cleanups with essential features so we might prioritorize some features above code cleanup. e.g. it is quite high on our priority list to implement power-off and reset for the "virt" machine as well as adding an emulated PCI host such as the QEMU GPEX PCI host or the XILINX PCI host, and it's unlikely that a non-standard hack for power-off and reset would pass upstream code review. i.e. we can't take short-cuts if we want to get merged upstream. We also have to make sure when performing a large number of cleanups, code changes and removals that we don't introduce any regressions, and this requires a lot of testing.

If anyone wants to volunteer to help with changes necessary to pass upstream review, please contact me as I could use the help. We definitely need to balance priorities in terms of adding new features and code cleanups. The next patch series will be posted to the RISC-V patches list so folk are aware of the necessary cleanups and can submit PR's to help out. Cleanups will likely be performed as a background task given we already have a stable branch based on recent upstream QEMU. i.e. it satisfies the usability and no regressions objective. The new "virt" machine features such as power-off and reset and VirtIO PCI may very well be prioritized ahead of cleanups although I'm not at liberty to pre-empt priorities until they have been set.

So in summary, if you are interested in helping with the RISC-V QEMU upstreaming process, which includes testing feedback and ideally PRs, please check out the 'riscv-next' branch and please base any RISC-V QEMU PRs against 'riscv-next' as we plan to use 'riscv-next' as the integration branch for generating the next patch series. 'riscv-next' is now in sync with QEMU master as of Dec 22, 2017 and it appears to be quite stable, and should be a little more usable given we now have basic network and block device suport working on the "virt" board. There is even nascent SMP support on the "virt" machine (limited to 8 cpus due to a hardcoded limit for thread stacks in BBL).

I started preparing a log of changes based on feedback, most generously from Richard Henderson, but given I'm going to be traveling for a few days and potentially working on other stuff, so don't expect a stream of commits from me for the next few days... Below is the list of changes I started on which we intend to include in the v2 patch series change log, along with other changes I'm still digesting from the responses on the qemu-devel mailing list. It was a bit of a sprint to get the patch series divided up and pass checkpatch which resulted in a substantial amount of format changes to the codebase.

- Make disassembler register name arrays static const char * const

- Remove redundant NULL terminators from disassembler register name arrays

- Remove redundant debug mode printf from exception handler code

- Remove redundant properties from sifive_prci

- Remove redundant #ifdef around cpu_has_work - handled in generic code

- Remove tlb_flush(cs); from riscv_cpu_reset - handled in generic code

- Remove unnecessary g_strdup from cpu_register

- Remove TARGET_HAS_ICE from cpu.h

- Remove duplicate typedef from struct CPURISCVState definition in cpu.h

- Add ULL to 64-bit constants in cpu_bits.h and everywhere else

- Move QEMUTimer *timer into RISCVCPU

- Remove mode fetch in get_physical_address. trust mmu_idx from cpu_mmu_index

- 64-bit constants must use ULL.  Otherwise builds from a 32-bit host will fail

e.g. #define MSTATUS64_UXL       0x0000000300000000

e.g. #define MSTATUS64_SXL       0x0000000C00000000

- Use qemu_log_mask with LOG_UNIMP or LOG_GUEST_ERROR instead of printf

- Remove occurrences of exit and replace with g_assert_not_reached

- once the VM has parsed args and initialised, there should be no exits

- illegal instruction exceptions should be raised for anything unimplemented

- Change instrumentation printfs to use QEMU trace API

- Many other changes mentioned in patch review feedback on the qemu-devel list

- still digesting

If anyone has the desire to tackle some of these over the next few days, you won't clash with me as I'll be traveling. Again, please base PRs on 'riscv-next'.

Regards,

Michael.

[Michael Clark]

To cut this long story short. We need to get enough testing feedback on 'riscv-next' to promote it to 'master'. That seems to be the logical next step.

Then it will be easier to work on new features and more cleanup work as we'll no longer need to back-porting to the 20160927 branch which is currently still 'master'.

[Liviu Ionescu]

> On 5 Jan 2018, at 01:47, Michael Clark <m...@sifive.com> wrote:
>
> ... a quick note about RISC-V QEMU changes

Thank you, Michael, that's good news.

I'm interested on the status of emulation for:

- HiFive1
- E31ARTY
- E51ARTY

My requirements are quite basic, in addition to the core & interrupts, I would need basic clock and GPIO support, aiming to run RTOS blinky apps on the existing SiFive boards.

The final goal is to integrate the RISC-V QEMU into GNU MCU Eclipse, similarly to the integration of Cortex-M QEMU (https://gnu-mcu-eclipse.github.io/tutorials/blinky-arm/#run-the-test-with-qemu), and be able to run the projects generated by the SiFive template on QEMU.


Thank you,

Liviu

[Richard W.M. Jones]

I've moved the Fedora riscv-qemu package to the current
riscv-next[1][2], so it'll get some testing doing builds locally.

Is this version supposed to support SMP? And if so are any
corresponding kernel patched needed?

Rich.

[1] https://github.com/rwmjones/fedora-riscv-bootstrap/commit/ddf8c3f9a16495c332d2832324590f9ba6a90663
[2] http://copr-fe.cloud.fedoraproject.org/coprs/rjones/riscv/build/695224/

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Michael Clark]

> On 5/01/2018, at 10:55 PM, Richard W.M. Jones <rjo...@redhat.com> wrote:
>
>
> I've moved the Fedora riscv-qemu package to the current
> riscv-next[1][2], so it'll get some testing doing builds locally.

Thanks! I’d really like to try one of the fedora images with glibc, coreutils, bash, openssh, etc. My testing so far has been limited to busybox and dropbear dynamically linked against glibc.

> Is this version supposed to support SMP? And if so are any
> corresponding kernel patched needed?

Yes. Alpha SMO support. I wouldn’t suggest the SMP support is “workload ready” but it boots. I enabled SMP on a Linux kernel built from the riscv-linux-4.14 branch and it booted (slowly) and I could cat /proc/cpuinfo and /proc/interrupts and could see all the cores and they were getting interrupts, but I noticed lag on VirtIO so there might be some issues with interrupt delivery. It needs some directed attention to find out why RISC-V QEMU is laggy with more than one core enabled.

[Michael Clark]

> On 6/01/2018, at 12:05 AM, Michael Clark <michae...@mac.com> wrote:
>
> Alpha “SMP” support

It’s this iPhone keyboard.

[Richard W.M. Jones]

On Sat, Jan 06, 2018 at 12:05:42AM +1300, Michael Clark wrote:
>
> > On 5/01/2018, at 10:55 PM, Richard W.M. Jones <rjo...@redhat.com> wrote:
> >
> >
> > I've moved the Fedora riscv-qemu package to the current
> > riscv-next[1][2], so it'll get some testing doing builds locally.
>
> Thanks! I’d really like to try one of the fedora images with glibc, coreutils, bash, openssh, etc. My testing so far has been limited to busybox and dropbear dynamically linked against glibc.

Sure, grab the current stage 3 disk image here:

http://oirase.annexia.org/riscv/

glibc and coreutils in that image are both cross-compiled (see my
notes in the other email about what "stage 3" and "stage 4" mean).

It also has some RPMs under /rpmbuild/RPMS inside the disk image so
you can install packages using:

rpm -Uvh --nodeps /rpmbuild/RPMS/riscv64/...

Since it's the stage 3 disk image (not stage 4) it's full of massive
hacks, and --nodeps is required because RPM doesn't know about the
files in the disk. This will all be fixed in stage 4. But it should
be enough for testing purposes.

All the sources are in:
https://github.com/rwmjones/fedora-riscv-bootstrap

> > Is this version supposed to support SMP? And if so are any
> > corresponding kernel patched needed?
>
> Yes. Alpha SMO support. I wouldn’t suggest the SMP support is
> “workload ready” but it boots. I enabled SMP on a Linux kernel built
> from the riscv-linux-4.14 branch and it booted (slowly) and I could
> cat /proc/cpuinfo and /proc/interrupts and could see all the cores
> and they were getting interrupts, but I noticed lag on VirtIO so
> there might be some issues with interrupt delivery. It needs some
> directed attention to find out why RISC-V QEMU is laggy with more
> than one core enabled.

OK I'll try again and see if I can make a proper bug report this time
(if it still doesn't work).

Is MTTCG working? Without MTTCG there's not really any point in us
using SMP at this time, since builds will just be slower.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

Yes it does work (slowly of course):

stage3:/# cat /proc/cpuinfo
hart : 0
isa : rv64imafdc
mmu : sv48

hart : 1
isa : rv64imafdc
mmu : sv48

hart : 2
isa : rv64imafdc
mmu : sv48

hart : 3
isa : rv64imafdc
mmu : sv48

stage3:/# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
7: 0 0 0 0 riscv,plic0,c000000 7 virtio1
8: 83 42 61 28 riscv,plic0,c000000 8 virtio0
13: 59 25 28 58 riscv,plic0,c000000 10 ttyS0

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard Henderson]

On 01/05/2018 03:32 AM, Richard W.M. Jones wrote:
> Is MTTCG working? Without MTTCG there's not really any point in us
> using SMP at this time, since builds will just be slower.

No.

The atomic instructions need to use atomic tcg ops and memory barriers need
filling in. Then add TCG_GUEST_DEFAULT_MO and flip the switch in the configure
file.


r~