Kernel 5.10 woes with OpenSBI + u-boot

Zoltan Herpai

unread,

Dec 19, 2020, 7:05:02 PM12/19/20

to RISC-V SW Dev

Hi all,

I'm trying to boot a 5.10 kernel with OpenSBI trunk and u-boot 2020.10 on an Unleashed board, but the kernel booting crashes at a very early stage at initmem. I'm using the defconfig, and have also tried OpenSBI 0.8 and the 2021.01-rcX releases of u-boot - all of them fail with the same error at the kernel. Trying to bisect, 5.10-rc3 had this issue already (-rc1 didn't boot for me for some reason), while the latest 5.9 boots fine.

Earlier in December, Michael had a similar issue with qemu + u-boot on -rc5 - this looks to be similar if not the same issue. Can someone please advise if there is a known issue around this, or if I've left out maybe a config option?

Thanks,

Zoltan H

---- CUT HERE ----

U-Boot SPL 2020.10 (Dec 18 2020 - 16:15:29 +0000)

Trying to boot from MMC1

U-Boot 2020.10 (Dec 18 2020 - 16:15:29 +0000)

CPU: rv64imafdc

Model: SiFive HiFive Unleashed A00

DRAM: 8 GiB

MMC: spi@10050000:mmc@0: 0

Loading Environment from SPIFlash... SF: Detected is25wp256 with page size 256 Bytes, erase size 4 KiB, total 32 MiB

*** Warning - bad CRC, using default environment

In: serial@10010000

Out: serial@10010000

Err: serial@10010000

Net: eth0: ethernet@10090000

Hit any key to stop autoboot: 0

switch to partitions #0, OK

mmc0 is current device

Scanning mmc 0:3...

Found U-Boot script /boot.scr

279 bytes read in 2 ms (135.7 KiB/s)

## Executing script at 88100000

17710680 bytes read in 9637 ms (1.8 MiB/s)

## Loading kernel from FIT Image at 84000000 ...

Using 'config@1' configuration

Trying 'kernel@1' kernel subimage

Description: RISCV Linux-5.10.1

Type: Kernel Image

Compression: uncompressed

Data Start: 0x840000e8

Data Size: 17701812 Bytes = 16.9 MiB

Architecture: RISC-V

OS: Linux

Load Address: 0xa4000000

Entry Point: 0xa4000000

Hash algo: crc32

Hash value: 0ae219b8

Hash algo: sha1

Hash value: eb8b389b9f0b9d1874d05933b8a5072cf0d52a5b

Verifying Hash Integrity ... crc32+ sha1+ OK

## Loading fdt from FIT Image at 84000000 ...

Using 'config@1' configuration

Trying 'fdt@1' fdt subimage

Description: RISCV OpenWrt sifive_fu540 device tree blob

Type: Flat Device Tree

Compression: uncompressed

Data Start: 0x850e1dd8

Data Size: 6987 Bytes = 6.8 KiB

Architecture: RISC-V

Hash algo: crc32

Hash value: 67c8558f

Hash algo: sha1

Hash value: bd3ad2fc4693a20996f03dbb8883f71a98b507cc

Verifying Hash Integrity ... crc32+ sha1+ OK

Booting using the fdt blob at 0x850e1dd8

Loading Kernel Image

Using Device Tree in place at 00000000850e1dd8, end 00000000850e6922

Starting kernel ...

[ 0.000000] Linux version 5.10.1 (wigyori@trabant) (riscv64-openwrt-linux-gcc (OpenWrt GCC 8.4.0 r15216+10-aab8aa4cd0) 8.4.0, GNU ld (GNU Binutils) 2.34) #2 SMP Sat Dec 19 16:13:43 CET 2020

[ 0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0xa4000000

[ 0.000000] earlycon: sbi0 at I/O port 0x0 (options '')

[ 0.000000] printk: bootconsole [sbi0] enabled

[ 0.000000] Zone ranges:

[ 0.000000] DMA32 [mem 0x00000000a4000000-0x00000000ffffffff]

[ 0.000000] Normal [mem 0x0000000100000000-0x000000027fffffff]

[ 0.000000] Movable zone start for each node

[ 0.000000] Early memory node ranges

[ 0.000000] node 0: [mem 0x00000000a4000000-0x000000027fffffff]

[ 0.000000] Initmem setup node 0 [mem 0x00000000a4000000-0x000000027fffffff]

[ 0.000000] Unable to handle kernel paging request at virtual address ffffffdfe10e1dd9

[ 0.000000] Oops [#1]

[ 0.000000] Modules linked in:

[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.1 #2

[ 0.000000] epc: ffffffe0004ad41a ra : ffffffe00001ee46 sp : ffffffe001003f60

[ 0.000000] gp : ffffffe0010e1580 tp : ffffffe00100bd40 t0 : ffffffe1d57d7c40

[ 0.000000] t1 : 0000000000000030 t2 : 2d7472776e65706f s0 : ffffffe001003f80

[ 0.000000] s1 : ffffffdfe10e1dd8 a0 : ffffffdfe10e1dd8 a1 : ffffffe1d57d7c00

[ 0.000000] a2 : fffffffffffff000 a3 : 0000000000000000 a4 : 0000000000000000

[ 0.000000] a5 : ffffffdf5c000000 a6 : ffffffe0010104b8 a7 : 0000000000000080

[ 0.000000] s2 : ffffffe0010e20a8 s3 : 000000000000071f s4 : 0000000000000001

[ 0.000000] s5 : 00000000ff77b928 s6 : 00000000fffe8998 s7 : 0000000000000000

[ 0.000000] s8 : 00000000fff8409a s9 : 00000000a4000000 s10: 00000000840000e8

[ 0.000000] s11: 0000000000000000 t3 : ffffffe0008c26d0 t4 : 0000000000000001

[ 0.000000] t5 : 0000000000000018 t6 : 00000001dc000000

[ 0.000000] status: 0000000200000100 badaddr: ffffffdfe10e1dd9 cause: 000000000000000d

[ 0.000000] random: get_random_bytes called from print_oops_end_marker+0x22/0x4a with crng_init=0

[ 0.000000] ---[ end trace 0000000000000000 ]---

[ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task!

[ 0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

Michael Opdenacker

unread,

Dec 22, 2020, 10:42:09 AM12/22/20

to Zoltan Herpai, RISC-V SW Dev

Hi Zoltan,

Thanks for sharing your issue.

On 12/20/20 1:05 AM, Zoltan Herpai wrote:
> Hi all,
>
> I'm trying to boot a 5.10 kernel with OpenSBI trunk and u-boot 2020.10
> on an Unleashed board, but the kernel booting crashes at a very early
> stage at initmem. I'm using the defconfig, and have also tried OpenSBI
> 0.8 and the 2021.01-rcX releases of u-boot - all of them fail with the
> same error at the kernel. Trying to bisect, 5.10-rc3 had this issue
> already (-rc1 didn't boot for me for some reason), while the latest
> 5.9 boots fine.
>
> Earlier in December, Michael had a similar issue with qemu + u-boot on
> -rc5 - this looks to be similar if not the same issue. Can someone
> please advise if there is a known issue around this, or if I've left
> out maybe a config option?

Interesting. I tried with 5.9, but still get the same error booting from
U-Boot from QEMU:

[    0.000000] Sorting __ex_table...

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off

[    0.000000] Unable to handle kernel paging request at virtual address ffffffe001f4f008

[ 0.000000] Oops [#1]

[ 0.000000] Modules linked in:

[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.9.16 #1

[    0.000000] epc: ffffffe0000240a6 ra : ffffffe00000dbc8 sp : ffffffe001003ed0

[    0.000000] gp : ffffffe001070c80 tp : ffffffe00100bd40 t0 : ffffffe001082460

[    0.000000] t1 : ffffffe001072938 t2 : 0000000000000018 s0 : ffffffe001003ee0

[    0.000000] s1 : ffffffe0010720a8 a0 : ffffffe001f4f000 a1 : 0000000000088001

[    0.000000] a2 : ffffffffffffffff a3 : 0000000000001000 a4 : ffffffe001f4f008

[    0.000000] a5 : 0000000000088000 a6 : 0000000000000038 a7 : ffffffe001072930

[    0.000000] s2 : ffffffffffffffff s3 : ffffffe001072928 s4 : ffffffe001072038

[    0.000000] s5 : ffffffe001072030 s6 : ffffffe00089d008 s7 : 0000000000000000

[    0.000000] s8 : 00000000bff6a052 s9 : 0000000080200000 s10: 0000000000000000

[    0.000000] s11: 0000000000000000 t3 : ffffffe001072108 t4 : 0000000000000000

[    0.000000] t5 : ffffffe001072448 t6 : ffffffe00107f486

[    0.000000] status: 0000000000000100 badaddr: ffffffe001f4f008 cause: 000000000000000f

[    0.000000] random: get_random_bytes called from print_oops_end_marker+0x22/0x44 with crng_init=0

[    0.000000] ---[ end trace 0000000000000000 ]---

[    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!

[    0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

Indeed, it goes a little further on Linux 5.9 compared to 5.10 (latest
master during the merge window), like we have the kernel command line
info and a few extra details.

As usual, booting Linux directly from QEMU works.

Still investigating... slowly ;-)

Cheers,

Michael.

--
Michael Opdenacker, CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

atish patra

unread,

Dec 22, 2020, 1:43:03 PM12/22/20

to Michael Opdenacker, Zoltan Herpai, RISC-V SW Dev

Can you share your qemu command line arguments and U-boot head commit ?

I am able to boot Linux kernel 5.10 with U-Boot (v2021.01-rc1) without any issues.
I have attached my boot log with the qemu commandline.

I am not using U-Boot SPL though. I will try using U-Boot SPL next.

Indeed, it goes a little further on Linux 5.9 compared to 5.10 (latest
master during the merge window), like we have the kernel command line
info and a few extra details.

As usual, booting Linux directly from QEMU works.

Still investigating... slowly ;-)

Cheers,

Michael.

--
Michael Opdenacker, CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/a7932080-71ce-0027-8d08-fea0a9d698b9%40bootlin.com.

--

Regards,

Atish

qemu_uboot_linux.txt

atish patra

unread,

Dec 22, 2020, 2:40:23 PM12/22/20

to Michael Opdenacker, Zoltan Herpai, RISC-V SW Dev

Linux boot seems to work fine with U-Boot SPL as well.

qemu-system-riscv64 -nographic -machine virt -m 2G -smp 4 -bios spl/u-boot-spl -device loader,file=u-boot.itb,addr=0x80200000 -device loader,file=/home/atish/workspace/linux/arch/riscv/boot/Image,addr=0xa4000000 -object rng-random,filename=/dev/urandom,id=rng0

U-Boot SPL 2021.01-rc1-00007-g3e73da6c8dd6-dirty (Dec 22 2020 - 11:22:19 -0800)
Trying to boot from RAM

U-Boot 2021.01-rc1-00007-g3e73da6c8dd6-dirty (Dec 22 2020 - 11:22:19 -0800)

CPU: rv64imafdcsu
Model: riscv-virtio,qemu
DRAM: 2 GiB
In: uart@10000000
Out: uart@10000000
Err: uart@10000000
Net: No ethernet found.

Hit any key to stop autoboot: 0

=> booti 0xa4000000 - $fdtcontroladdr
Moving Image from 0xa4000000 to 0x80200000, end=81d39000
## Flattened Device Tree blob at ff765600
Booting using the fdt blob at 0xff765600
Using Device Tree in place at 00000000ff765600, end 00000000ff76be1f

Starting kernel ...

[ 0.000000] Linux version 5.10.0-00025-ga1685807fb5a (atish@jedi-01) (riscv64-unknown-linux-gnu-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35) #564 SMP Mon Dec 21 16:15:13 PST 2020
[ 0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[ 0.000000] efi: UEFI not found.
[ 0.000000] Zone ranges:
[ 0.000000] DMA32 [mem 0x0000000080200000-0x00000000ffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node

[ 0.000000] Early memory node ranges

[ 0.000000] node 0: [mem 0x0000000080200000-0x00000000ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000000ffffffff]
[ 0.000000] SBI specification v0.2 detected
[ 0.000000] SBI implementation ID=0x1 Version=0x8
[ 0.000000] SBI v0.2 TIME extension detected
[ 0.000000] SBI v0.2 IPI extension detected
[ 0.000000] SBI v0.2 RFENCE extension detected
[ 0.000000] software IO TLB: mapped [mem 0x00000000f9b65000-0x00000000fdb65000] (64MB)
[ 0.000000] SBI v0.2 HSM extension detected
[ 0.000000] riscv: ISA extensions acdfimsu
[ 0.000000] riscv: ELF capabilities acdfim
[ 0.000000] percpu: Embedded 17 pages/cpu s32232 r8192 d29208 u69632
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 516615
[ 0.000000] Kernel command line:
[ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[ 0.000000] Sorting __ex_table...

[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off

[ 0.000000] Memory: 1969244K/2095104K available (7033K kernel code, 3683K rwdata, 4096K rodata, 11559K init, 343K bss, 125860K reserved, 0K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] fixmap : 0xffffffcefee00000 - 0xffffffceff000000 (2048 kB)
[ 0.000000] pci io : 0xffffffceff000000 - 0xffffffcf00000000 ( 16 MB)
[ 0.000000] vmemmap : 0xffffffcf00000000 - 0xffffffcfffffffff (4095 MB)
[ 0.000000] vmalloc : 0xffffffd000000000 - 0xffffffdfffffffff (65535 MB)
[ 0.000000] lowmem : 0xffffffe000000000 - 0xffffffe07fe00000 (2046 MB)

Please check your fdt/kernel loading address and make sure that they are correct.

--

Regards,

Atish

Palmer Dabbelt

unread,

Dec 22, 2020, 3:46:12 PM12/22/20

to michael.o...@bootlin.com, zhe...@gmail.com, sw-...@groups.riscv.org

5.10 is out, the merge window (and therefor Linus' master) is 5.11. Anything
before rc1 is generally a minefield, but my PR boots for me -- that's all on
QEMU, though, as I don't have (and therefor don't test) an Unleashed. There
may be someone from SiFive still testing the board, and last time I checked the
WD guys were using them.

Regardless of what's going on, it's probably best to have this discussion on
linux-riscv. sw-dev isn't really a good place for this, as despite the name no
software development actually happens here. While this may not be a Linux
issue, the bootloader people are generally pretty active on linux-riscv
(there's not that many of us).

> As usual, booting Linux directly from QEMU works.
>
> Still investigating... slowly ;-)
>
> Cheers,
>
> Michael.
>
> --
> Michael Opdenacker, CEO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

Zoltan Herpai

unread,

Dec 23, 2020, 12:03:19 PM12/23/20

to atish patra, Michael Opdenacker, RISC-V SW Dev

Hi Atish,

Indeed it was the kernel load address - changing it to 0x80200000 from 0xa4000000 fixed the crash when using bootm on the Unleashed board - thanks. Not sure how the original address worked with 5.9 or earlier kernels on the board with the same OpenSBI/u-boot versions, though, but I'll check the qemu build and the earlier kernel versions in the next few days.

Palmer, thanks for the pointer, next time I'll take these questions to the linux-riscv list.

Best regards,

Zoltan H

Michael Opdenacker

unread,

Dec 23, 2020, 3:06:04 PM12/23/20

to atish patra, Zoltan Herpai, RISC-V SW Dev

Hi Atish,

On 12/22/20 7:42 PM, atish patra wrote:
>
>
>
> Can you share your qemu command line arguments and U-boot head commit ?
> I am able to boot Linux kernel 5.10 with U-Boot (v2021.01-rc1) without
> any issues.
> I have attached my boot log with the qemu commandline.
>
> I am not using U-Boot SPL though. I will try using U-Boot SPL next.

Many thanks for your help. It's good to know that it works.

I eventually managed to make this work. It turns out that my dtb was
wrong, and it seems that it was cause of my issue.

What I wonder is why there is no dts in the Linux kernel sources for the
QEMU virtio board. However, U-Boot will refuse to load a Linux image
without one. To get one, I had to boot Linux directly, and get the DTB
from the /sys/firmware/fdt file. Didn't you have the same constraint?
How did you adress it? I don't see how the dtb was loaded from your log.

I'll ask this question on the linux...@lists.infradead.org mailing
list too...

So, here are my raw notes for successfully booting Linux from U-Boot in
the QEMU virt machine...

Thanks again!

Michael.

---

QEMU Risc-V instructions
For booting Linux from U-Boot in QEMU's virt machine

Linux (5.10.0)
export ARCH=riscv
make defconfig
make -j8 Image

Create a disk image with 2 partitions, to mimic a real-life case:
- A fat32 one (for the kernel and DTB)
- An ext2 or ext4 one (for the root fs)

To boot this kernel from QEMU, I need to compile opensbi for Linux
cd opensbi
make PLATFORM=generic FW_PAYLOAD_PATH=../linux/arch/arm/boot/Image

Booting Linux directly:
qemu-system-riscv64 -M virt -m 2G -nographic -kernel
opensbi/build/platform/generic/firmware/fw_payload.elf -device
virtio-blk-device,drive=hd0 -drive file=disk.img,format=raw,id=hd0
-append "console=ttyS0 rw root=/dev/vda2 earlycon=sbi"

From the booted system, copy /sys/firmware/fdt to the first partition of
the virtio drive, naming the file "dtb". Why can't we build one from the
Linux kernel sources?

Now, we can prepare U-Boot for booting Linux. Without a dtb, U-Boot
refused to start the Linux kernel.

U-Boot (c15f44acf9d473f4682bfdc63b8aebd313492b15, 2021.04-rc4)

cd u-boot
make qemu-riscv64_smode_defconfig

Modify the configuration to support loading an environment from a FAT
partition on a virtio disk. That's closer to real life:
CONFIG_ENV_IS_IN_FAT=y
CONFIG_ENV_FAT_INTERFACE="virtio"
CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
make -j 8

Now, let's compile opensbi again:
cd opensbi
make clean
make PLATFORM=generic FW_PAYLOAD_PATH=../u-boot/u-boot.bin

We can now boot U-Boot:
qemu-system-riscv64 -M virt -m 2G -nographic -bios
opensbi/build/platform/generic/firmware/fw_payload.elf -device
virtio-blk-device,drive=hd0 -drive file=disk.img,format=raw,id=hd0

The first time, interrupt the U-Boot countdown and set environment
variables:
setenv bootcmd 'fatload virtio 0:1 80200000 Image; fatload virtio 0:1
0x82000000 dtb; booti 0x80200000 - 0x82000000'
setenv bootargs 'root=/dev/vda2 rootwait console=ttyS0
earlycon=sbiroot=/dev/vda2 rootwait console=ttyS0 earlycon=sbi'
saveenv

You can now restart QEMU and Linux boots!

See the attached log.

final-u-boot-linux-boot.log

Michael Opdenacker

unread,

Dec 23, 2020, 3:08:27 PM12/23/20

to atish patra, Zoltan Herpai, RISC-V SW Dev

Hi Atish,

Thanks for sharing this too. Once again, I don't understand how the dtb
gets loaded. Does it come from U-Boot sources?

Michael Opdenacker

unread,

Dec 23, 2020, 3:10:08 PM12/23/20

to Palmer Dabbelt, zhe...@gmail.com, sw-...@groups.riscv.org

Hi Palmer,

On 12/22/20 9:46 PM, Palmer Dabbelt wrote:
>
> Regardless of what's going on, it's probably best to have this
> discussion on
> linux-riscv. sw-dev isn't really a good place for this, as despite
> the name no
> software development actually happens here. While this may not be a
> Linux
> issue, the bootloader people are generally pretty active on linux-riscv
> (there's not that many of us).

Agreed. I should have paid attention to the Linux MAINTAINERS file and
use that list instead.

I applied for membership...

Thanks,

atish patra

unread,

Dec 23, 2020, 3:17:26 PM12/23/20

to Michael Opdenacker, Zoltan Herpai, RISC-V SW Dev

On Wed, Dec 23, 2020 at 12:06 PM Michael Opdenacker <michael.o...@bootlin.com> wrote:

Hi Atish,

On 12/22/20 7:42 PM, atish patra wrote:
>
>
>
> Can you share your qemu command line arguments and U-boot head commit ?
> I am able to boot Linux kernel 5.10 with U-Boot (v2021.01-rc1) without
> any issues.
> I have attached my boot log with the qemu commandline.
>
> I am not using U-Boot SPL though. I will try using U-Boot SPL next.

Many thanks for your help. It's good to know that it works.

I eventually managed to make this work. It turns out that my dtb was
wrong, and it seems that it was cause of my issue.

What I wonder is why there is no dts in the Linux kernel sources for the
QEMU virtio board. However, U-Boot will refuse to load a Linux image
without one. To get one, I had to boot Linux directly, and get the DTB
from the /sys/firmware/fdt file. Didn't you have the same constraint?
How did you adress it? I don't see how the dtb was loaded from your log.

You shouldn't have to do that. There are few copy happens while placing the DTB.
In RISC-V, every boot loader stage passes the memory address of DTB in a1.

Thus, Qemu loader will pass the DTB address in a1 to OpenSBI and OpenSBI passes that to U-Boot via a1.

Before that, OpenSBI may copy the DTB depending on the firmware type (fw_jump or fw_dynamic).

U-Boot relocates the DT the $fdtcontroladdr. You can verify the correct fdt by doing the following at U-Boot prompt.

#fdt addr $fdtcontroladdr

#fdt print

Depending on how you have configured U-Boot, it may again need to be copied to fdt_addr_r if that's the address

you are using while using booti/bootm command so that Linux kernel gets the correct DTB address in a1.

I'll ask this question on the linux...@lists.infradead.org mailing
list too...

Yeah. As Palmer suggested, linux-riscv is a much better place to discuss these things as many more folks

watch that.

--

Regards,

Atish

Jose Martins

unread,

Dec 24, 2020, 11:34:50 AM12/24/20

to atish patra, Michael Opdenacker, Zoltan Herpai, RISC-V SW Dev

I've run into the same issue using Linux over the bao hypervisor when
switching from 5.9 to 5.10 with everything else being exactly the
same.

For me, the exception happens in fdt_check_header. This seems to have
something to do with where the dtb is loaded. Previously I loaded it
below the kernel image. By loading it above it, the problem went away.
Also, I now have to load the dtb to a 2 MiB aligned address.
Previously, I loaded to a 4KiB and it worked fine.

Hope it helps,
José

> --
> You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CAELrHRANGb2vfcG6Pv7MTiqMOgW_pG%2BOWkV565tAuDzZ%3D3C7Gw%40mail.gmail.com.

Reply all

Reply to author

Forward