Multicore RocketChip and SMP support on Risc-V Linux

[Stella]

I'm looking for a Risc-V Linux that supports SMP and at least successfully boots to ash (busybox). I'm aware that the Risc-V Linux Port is currently in the process of being upstreamed (see Palmer Dabbelt - RISC-V Linux Port v9). Reading the change highlights, SMP and atomics have undergone significant changes (multiple defect fixes).

I'm targeting the following versions of rocket chip. Ultimately, the goal is to have a multicore Rocket and a multicore Boom deployed to FPGA (Xilinx Zc706).

freechipsproject/rocket-chip@master (privileged specification 1.10)

ucb-bar/fpga-zynq@7581d21 -> rocket-chip@cf75c20 (privileged specification 1.10?)
https://github.com/donggyukim/fpga-zynq/tree/boom -> rocket-chip@d8379e2 (privileged specification 1.09?)

On rocket-chip@cf75c20 I have riscv/riscv...@riscv-4.12-v7 running without SMP on the FPGA (quadcore rocket). It works.
Compiling with SMP option and using spike -p4 also boots successfully. Unfortunately, the kernel hangs on the fpga when initializing the other CPUs:

[ 0.000000] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.000000] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.000000] smp: Bringing up secondary CPUs ...

Then I tried riscv/riscv-linux@riscv-for-submission-v9 with spike -p4. Despite configuring initramfs.txt, no shell or error ever appears?! It looks like some parts are missing, i.e., its very bare bones?

Beyond, I've recently come across sifive/freedom-u-sdk@new and riscv/riscv...@priv-1.10. I have yet to test these.

Thus, any suggestions for a working Risc-V Linux with SMP for Rocket? Thanks in advance!

[John Leidel]

You can indeed get the SMP linux builds up and running.  You can use the RISC-V Poky build that will do much of the leg work for you (including creating the disk images).  However, for SMP support in Poky, you'll need this patch: https://github.com/riscv/riscv-poky/pull/13.  (It may have been updated to support SMP work, but I haven't looked)

You can also follow the directions in the riscv-linux build here: https://github.com/riscv/riscv-linux/blob/riscv-next/Documentation/admin-guide/README.rst

You can manually configure the SMP kernel's limits during the kernel configuration process (I believe the max was previously 64).  I was only ever able to get 8 cores running w/ hanging the kernel during boot, but this may have changed given the recent flurry of development activity.  

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+unsubscribe@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/de49c2c6-3908-473e-bb6b-3c5ac9ec86b8%40groups.riscv.org.

[Sayath the Emperor]

I've now tested RISC-V Poky. I integrated your pull request, as seemed appropriate John, for priv 1.10.

Specifically, I tested riscv/riscv...@priv-1.10 with CONFIG_SMP=y and CONFIG_NR_CPUS=8. This leads to a successful SMP boot on spike (SPIKE_ARGS=-p4 runspike riscv64, check attachment 'spike').

Sadly, the FPGA begs to differ (rocket-chip@cf75c20, see attachment 'FPGA'). Ultimately, it yet again stops at "[    0.000000] smp: Bringing up secondary CPUs ...".

Maybe rocket-chip@cf75c20 is priv-1.09 and I'm mistaken in it being 1.10? I'll try the priv-1.09 branch and report back...

[Stella]

riscv/riscv-poky @ master / priv-1.09 both do not build? I assume maintenance has shifted towards priv-1.10.

Using the bbl generated for spike by sifive/freedom-u-sdk@new (~/freedom-u-sdk/work/riscv-pk/spike/bbl) and updating Fesvr accordingly, I once again run into the dreaded "smp: Bringing up secondary CPUs ...". Log is attached.

Obviously, SMP is working for Spike. Yet, it isn't working for my deployment of RocketChip(@cf75c20) on Fpga. So either a) I'm being impacted by the SMP defects discussed in the upstreaming process during v3/v4/v5 (likely, these are probably missing in the above versions) or b) a privileged specification mismatch between software and hardware or c) my rocket configuration is incorrect?

I guess another option instead of  Linux would be some kind of microkernel just supporting SMP, e.g., SEL4 (https://heshamelmatary.blogspot.de/2017/06/update-sel4risc-v-smp-support-sel4.html). I'd rather not go down that route though.

[Palmer Dabbelt]

freedom-u-sdk/new works on our FPGA builds. We're working on getting
everything synced up to a sane version so this is reproducible via
freedom/master and freedom-u-sdk/master.

[Martin Maas]

Hi,

 

riscv/riscv-poky @ master / priv-1.09 both do not build? I assume maintenance has shifted towards priv-1.10.

Maintenance has indeed shifted towards priv-1.10. As the other branches rely on repositories that are changing quite rapidly, I would use this branch as much as possible. I have now switched riscv-poky over to priv-1.10 as the default.

 

Obviously, SMP is working for Spike. Yet, it isn't working for my deployment of RocketChip(@cf75c20) on Fpga. So either a) I'm being impacted by the SMP defects discussed in the upstreaming process during v3/v4/v5 (likely, these are probably missing in the above versions) or b) a privileged specification mismatch between software and hardware or c) my rocket configuration is incorrect?

We haven't done any testing of riscv-poky on SMP systems, but I am happy to merge any pull-requests that fix these problems. My guess is that this is likely something that will stabilize as Linux is being upstreamed, since poky is currently using revisions of riscv-linux that worked in Spike but are not tested across a wider range of systems.

Thanks,

Martin