Linux support for a 32-core SMP Rocket Chip system

[morais.lucas.h]

Hi there,

I am having some difficulties to bring up all 32 cores of a Rocket Chip on Linux 5.10, using OpenSBI v0.9.

After Linux boots as described in this instrumented log, only 16 cores appear on /proc/cpuinfo. The ones that successfully boot are those with hart id in [0,15].

Right now, it seems to me that the most likely culprit is OpenSBI, since it mediates all hart start actions by intercepting the ecalls coming from Linux (arch/riscv/kernel/cpu_ops_sbi.c, function sbi_hsm_hart_start).

By inspecting FPGA nets during boot with Vivado probes, I could confirm that at least hart 1 (the first to be started by hart 0) is booting right after receiving a local interrupt from the latter, while hart 31 (which is not working) never receives any interrupt.

Also, I am inclined to believe that the problem is not in RC's implementation of CLINT (mainly defined in devices/tilelink/CLINT.scala), because (1) its comments briefly mention that it should support up to 4095 devices and (2) my own inspection of its memory mapped scheme ranging from 0x0 to 0x10000 suggests that it should be able to index 4096 hart-private MIP registers.

My question is finally this: are OpenSBI constants and other default parameters making any assumptions that could prevent more than 16 cores from being indexed by CLINT for booting purposes? In particular, is SBI_PLATFORM_DEFAULT_HART_STACK_SIZE (include/sbi/sbi_platform.h) default value of 8192 enough for allowing hart 0 to communicate with all other cores?

And just to confirm a final point, the fact that each core can only describe up to 16 PMP regions should not have anything to do with it, right?

Any insights on this would be of very welcome.

Thanks,

Lucas

[Tommy Murphy]

This suggests that it should be able to handle 128 harts...

https://github.com/riscv-software-src/opensbi/blob/48f91ee9c960f048c4a7d1da4447d31e04931e38/include/sbi/sbi_hartmask.h#L22

But maybe some other code elsewhere limits this further?

[Lucas]

Hi Tommy,

I bumped into that one as well, but I still feel that the system could be getting constrained by some ill-dimensioned memory structure.

Also, in platform/template/platform.c, a certain PLATFORM_HART_COUNT macro is set to 4 by default, but it seems that fw_platform_init dynamically detects the number of harts described by the device tree before this could be a problem (firmware/fw_base.S).

Right now, to validate the theory that OpenSBI might be the problem, I am planning to:

1) Build Linux without support for HSM-based hart initialization (modifying arch/riscv/kernel/smpboot.c), which should make hart boot depend on a spin-waited memory signal (arch/riscv/kernel/head.S) rather than on an interprocessor interrupt, making everything less dependent on OpenSBI;

2) if that doesn't work, build OpenSBI with double SBI_PLATFORM_DEFAULT_HART_STACK_SIZE and SBI_SCRATCH_SIZE values.

The second option is less ideal because the OpenSBI binary is stored in a system ROM, so updating it requires re-synthesizing the whole system. Also, it would be mostly a shot in the dark, since I am not very sure about what should fit into each of these memory regions.

I do not have completely ruled out that the bug might be in my own system integration, but since scaling up to 16 cores was pretty straightforward, I am currently assuming the FPGA port is mostly functional. If anyone could confirm having been able to boot more than 16 cores on a similar software stack without modifications, that would make me think otherwise, though.

Thanks,

Lucas

[Anup Patel]

Hi,

I have tested upto 64 CPUs on QEMU virt machine with the latest OpenSBI.

To allow more CPUs in the QEMU virt machine, we need changes in QEMU
virt machine code. There are already patches for it.

To allow more CPUs in Linux, we need to update the NR_CPUS kconfig
option. I already have a patch for it which I will send soon.

Regards,
Anup

> --
> You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CAPZR5J0X6aR1BDirNFf-YvZc1vUWryuHDWTP7nfErZgsL_JdxQ%40mail.gmail.com.

[Lucas]

Thanks a lot for the reply. I'll now check if my 5.10 / v0.9 combination can boot 32 cores on QEMU and upgrade to 5.15 / v1.0 if needed.

Regards,

Lucas

[Metthew Liang]

Hi Anup,

    Thanks a lot for your information. According to the current Kconfig file, it seems that RISC-V-based Linux system can support up 32 cores.

    I know whether there is a plan to support more cores or whether there is any approach to realize many-core system with RISC-V cores.

    Thanks again for your time!

Best regards,

Matthew

[Lucas]

Anup,

Thanks again for the support. Shortly after I received your message, I was able to verify that 32-core boot OpenSBI boot works on QEMU, and eventually I found out that my particular FPGA setup was failing to boot because while Linux was receiving a working 32-core device tree blob, OpenSBI was reading a 16-core one hardcoded to the zero-level bootloader.

In case anyone ever has a similar problem, please make sure that all bootloader levels are using the same device tree. Otherwise, the amount of cores that gets brought up might be limited by the minimum core count described by any of the device trees.

Regards,

Lucas

[Matthew Leung]

Hi Anup,

    Thanks for your effort! May I know whether the patches which enable more CPU for RISC-V Linux are available recently?

    Thanks again! ^_^

Best regards,

Matthew

On Wednesday, 29 December 2021 at 20:44:14 UTC+8 Anup Patel wrote:

[Anup Patel]

+Palmer

Hi Matthew,

There is only one patch required in Linux which is already on LKML.
https://lore.kernel.org/lkml/20220319121206...@ventanamicro.com/T/

Maybe it will be picked for Linux-5.19 or later.

Regards,
Anup

[Matthew Leung]

Hi Anup,

     WOW!! Awesome!!! THANKS a lot for your effort and information which means a lot for me.

     BTW, I am a bit curious why the previous range is 2-32 and whether the only required change is the Kconfig file since maybe there might some inter-processor interrupt signals which restricts the settings.

     Thanks again!

Best regards,

Matthew