New Linux Kernel memory layout introduced in 6.4-rc1 causing kernel to crash if not taking careful action

jee heng Sia

unread,

May 9, 2023, 11:20:09 PM5/9/23

to RISC-V SW Dev, anup....@wdc.com

In the past, the physical memory range from 0x80000000 to 0x80200000 was not usable for Linux. However, in Linux 6.4-rc1, it appears that the memory layout has changed. The memory range from 0x80000000 to 0x80200000 now seems to be usable by Linux, but it will cause a kernel crash if the kernel tries to access that memory range. This memory range is typically reserved by OpenSBI. So, to solve the problem, OpenSBI has to avoid using the physical memory region within 0x80000000 to 0x80200000. I am not sure if Opensbi community knows about it.

Below is the Linux's PTDump

---[ Linear mapping ]--- [Start of Physical Addr]

0xff60000000000000-0xff60000000200000 0x0000000080000000 <<<<-------- This memory range is now visible to Linux, but it is reserved in OpenSBI

0xff60000000200000-0xff60000000c00000 0x0000000080200000

0xff60000000c00000-0xff60000001c00000 0x0000000080c00000

0xff60000001c00000-0xff60000002200000 0x0000000081c00000

0xff60000002200000-0xff60000040000000 0x0000000082200000

---[ Kernel mapping ]--- [Start of Physical Addr]

0xffffffff80000000-0xffffffff80a00000 0x0000000080200000

0xffffffff80a00000-0xffffffff80c00000 0x0000000080c00000

0xffffffff80c00000-0xffffffff81a00000 0x0000000080e00000

0xffffffff81a00000-0xffffffff82000000 0x0000000081c00000

0xffffffff82000000-0xffffffff82200000 0x0000000082200000

Below is the memory info from OpenSBI console:

Domain0 Name : root

Domain0 Boot HART : 1

Domain0 HARTs : 0*,1*

Domain0 Region00 : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()

Domain0 Region01 : 0x0000000080000000-0x000000008001ffff M: (R,X) S/U: () <<<<<<----- This memory range is not accessible to S/U mode

Domain0 Region02 : 0x0000000080020000-0x000000008003ffff M: (R,W) S/U: () <<<<<<----- This memory range is not accessible to S/U mode

Domain0 Region03 : 0x0000000000000000-0xffffffffffffffff M: (R,W,X) S/U: (R,W,X)

Domain0 Next Address : 0x0000000080200000

Thanks

Regards

Jee Heng

Palmer Dabbelt

unread,

May 10, 2023, 1:03:08 AM5/10/23

to sw-...@groups.riscv.org, an...@brainfault.org

On Tue, 09 May 2023 20:20:08 PDT (-0700), sw-...@groups.riscv.org wrote:
>
>
> In the past, the physical memory range from 0x80000000 to 0x80200000 was
> not usable for Linux. However, in Linux 6.4-rc1, it appears that the memory
> layout has changed. The memory range from 0x80000000 to 0x80200000 now
> seems to be usable by Linux, but it will cause a kernel crash if the kernel
> tries to access that memory range. This memory range is typically reserved
> by OpenSBI. So, to solve the problem, OpenSBI has to avoid using the
> physical memory region within 0x80000000 to 0x80200000. I am not sure if
> Opensbi community knows about it.

It's probably best to post this on linux-riscv, if it's crashing systems
then it's a regression and we should at least work around it. We've got
a lot of unspecified behavior floating around the boot process, and
while this isn't tripping up on my test systems they're all just QEMU so
it's not super surprising that stuff like this gets missed.

Ideally we'd document this sort of stuff too, though we've been saying
that for a while...

atish patra

unread,

May 10, 2023, 2:37:45 AM5/10/23

to jee heng Sia, RISC-V SW Dev, Anup Patel

Fixing Anup's email address:

On Tue, May 9, 2023 at 8:20 PM 'jee heng Sia' via RISC-V SW Dev <sw-...@groups.riscv.org> wrote:

In the past, the physical memory range from 0x80000000 to 0x80200000 was not usable for Linux. However, in Linux 6.4-rc1, it appears that the memory layout has changed. The memory range from 0x80000000 to 0x80200000 now seems to be usable by Linux, but it will cause a kernel crash if the kernel tries to access that memory range. This memory range is typically reserved by OpenSBI. So, to solve the problem, OpenSBI has to avoid using the physical memory region within 0x80000000 to 0x80200000. I am not sure if Opensbi community knows about it.

Opensbi continues to mark that region as reserved and the kernel log reflects that as well.

[ 0.000000] Linux version 6.4.0-rc1-00012-g16a8829130ca (ati...@atishp.ba.rivosinc.com) (riscv64-unknown-linux-gnu-gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.37) #190 SMP Tue May 9 23:16:
28 PDT 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: riscv-virtio,qemu
[ 0.000000] earlycon: ns16550a0 at MMIO 0x0000000010000000 (options '')
[ 0.000000] printk: bootconsole [ns16550a0] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) map non-reusable mmode_resv0@80000000

As per my understanding, the kernel will not try to use that memory as it is marked reserved.

That region is mappable now in 6.4-rc1 to allow users to use the first 2MB of region if they want to load/boot OpenSBI from somewhere else in the memory. Earlier it was not possible.

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/39a3bd49-2162-4cc8-90aa-43310aaab595n%40groups.riscv.org.

--

Regards,

Atish

jee heng Sia

unread,

May 10, 2023, 10:18:48 AM5/10/23

to RISC-V SW Dev, atis...@gmail.com, RISC-V SW Dev, an...@brainfault.org, jee heng Sia

On Wednesday, May 10, 2023 at 2:37:45 PM UTC+8 atis...@gmail.com wrote:

Fixing Anup's email address:

On Tue, May 9, 2023 at 8:20 PM 'jee heng Sia' via RISC-V SW Dev <sw-...@groups.riscv.org> wrote:
In the past, the physical memory range from 0x80000000 to 0x80200000 was not usable for Linux. However, in Linux 6.4-rc1, it appears that the memory layout has changed. The memory range from 0x80000000 to 0x80200000 now seems to be usable by Linux, but it will cause a kernel crash if the kernel tries to access that memory range. This memory range is typically reserved by OpenSBI. So, to solve the problem, OpenSBI has to avoid using the physical memory region within 0x80000000 to 0x80200000. I am not sure if Opensbi community knows about it.

Opensbi continues to mark that region as reserved and the kernel log reflects that as well.

[JH] That is true.

[ 0.000000] Linux version 6.4.0-rc1-00012-g16a8829130ca (ati...@atishp.ba.rivosinc.com) (riscv64-unknown-linux-gnu-gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.37) #190 SMP Tue May 9 23:16:
28 PDT 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: riscv-virtio,qemu
[ 0.000000] earlycon: ns16550a0 at MMIO 0x0000000010000000 (options '')
[ 0.000000] printk: bootconsole [ns16550a0] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) map non-reusable mmode_resv0@80000000

As per my understanding, the kernel will not try to use that memory as it is marked reserved.
That region is mappable now in 6.4-rc1 to allow users to use the first 2MB of region if they want to load/boot OpenSBI from somewhere else in the memory. Earlier it was not possible.

[JH] This could be the reason that this memory region is now made available to the Linux kernel. Is there a documentation update or patch available for OpenSBI to describe the PMP restrictions being lifted for this memory region?

Anup Patel

unread,

May 10, 2023, 10:30:18 AM5/10/23

to jee heng Sia, RISC-V SW Dev, atis...@gmail.com

Hi,

On Wed, May 10, 2023 at 7:48 PM jee heng Sia <siaje...@googlemail.com> wrote:
>
>
>
> On Wednesday, May 10, 2023 at 2:37:45 PM UTC+8 atis...@gmail.com wrote:
>
> Fixing Anup's email address:
>
> On Tue, May 9, 2023 at 8:20 PM 'jee heng Sia' via RISC-V SW Dev <sw-...@groups.riscv.org> wrote:
>
> In the past, the physical memory range from 0x80000000 to 0x80200000 was not usable for Linux. However, in Linux 6.4-rc1, it appears that the memory layout has changed. The memory range from 0x80000000 to 0x80200000 now seems to be usable by Linux, but it will cause a kernel crash if the kernel tries to access that memory range. This memory range is typically reserved by OpenSBI. So, to solve the problem, OpenSBI has to avoid using the physical memory region within 0x80000000 to 0x80200000. I am not sure if Opensbi community knows about it.
>
>
> Opensbi continues to mark that region as reserved and the kernel log reflects that as well.
>
> [JH] That is true.
>
>
> [ 0.000000] Linux version 6.4.0-rc1-00012-g16a8829130ca (ati...@atishp.ba.rivosinc.com) (riscv64-unknown-linux-gnu-gcc (GCC) 11.1.0, GNU ld (GNU Binutils) 2.37) #190 SMP Tue May 9 23:16:
> 28 PDT 2023
> [ 0.000000] random: crng init done
> [ 0.000000] Machine model: riscv-virtio,qemu
> [ 0.000000] earlycon: ns16550a0 at MMIO 0x0000000010000000 (options '')
> [ 0.000000] printk: bootconsole [ns16550a0] enabled
> [ 0.000000] efi: UEFI not found.
> [ 0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) map non-reusable mmode_resv0@80000000
>
> As per my understanding, the kernel will not try to use that memory as it is marked reserved.
> That region is mappable now in 6.4-rc1 to allow users to use the first 2MB of region if they want to load/boot OpenSBI from somewhere else in the memory. Earlier it was not possible.
> [JH] This could be the reason that this memory region is now made available to the Linux kernel. Is there a documentation update or patch available for OpenSBI to describe the PMP restrictions being lifted for this memory region?

OpenSBI does not lift the PMU restrictions from FW regions (512KB @ 0x80000000).

The current approach is OpenSBI continues to restrict access to its
own FW memory (512KB @ 0x80000000) and lets kernel reclaim other parts
of memory. In other words, we don't expect kernel to reserve the
entire 2M @ 0x80000000.

Regards,
Anup

jee heng Sia

unread,

May 10, 2023, 8:42:03 PM5/10/23

to RISC-V SW Dev, an...@brainfault.org, RISC-V SW Dev, atis...@gmail.com, jee heng Sia

[JH] If that is the case, we should expect a fix in the kernel's page table because, in the recent kernel 6.4-rc1, the reserved memory region is exposed to the kernel's page table, and the page attributes in the region are marked as "readable" and "writable." For the above use case, we should use the "no-map" approach so that this region is not exposed to the page table.

Reply all

Reply to author

Forward