How to create riscv-linux kernel BBL image

[Pintu Kumar]

Hi All,

When I build riscv-linux using riscv-64 linux tool chain I observed
that only the vmlinux image is getting created.
But does not create any zImage/Image/etc under arch/riscv/boot/ (no
boot folder).

However, I learned that BBL kernel image is required, to boot the
linux-kernel using qemu.
So, how do we create a kernel BBL image ourselves, without using the
riscv-pk tool?

If anybody have already done it, please share the steps.

Thanks,
Pintu

[Jim Wilson]

On Wed, Sep 19, 2018 at 11:36 PM Pintu Kumar <pintu...@gmail.com> wrote:
> However, I learned that BBL kernel image is required, to boot the
> linux-kernel using qemu.
> So, how do we create a kernel BBL image ourselves, without using the
> riscv-pk tool?

The linux kernel runs at supervisor mode. The bbl runs at machine
mode, and provides services to the linux kernel. You can't run the
kernel without the bbl. There is a separate issue about booting.
Most current boot loaders can only load a single image, so the bbl and
kernel have to be linked together. Qemu has some support to load bbl
and kernel separately, I think there is a bios option used for bbl.
But I think most people still link them together.

You can see an example of how to build and boot a kernel in the
sifive/freedom-u-sdk project. "make qemu" will build a linux kernel
plus bbl plus buildroot and boot it on qemu.
https://github.com/sifive/freedom-u-sdk

Fedora has their own scripts for building kernels which you could also look at.
https://github.com/rwmjones/fedora-riscv-kernel

Jim

[Pintu Kumar]

On Fri, Sep 21, 2018 at 11:10 PM Jim Wilson <ji...@sifive.com> wrote:
>
> On Wed, Sep 19, 2018 at 11:36 PM Pintu Kumar <pintu...@gmail.com> wrote:
> > However, I learned that BBL kernel image is required, to boot the
> > linux-kernel using qemu.
> > So, how do we create a kernel BBL image ourselves, without using the
> > riscv-pk tool?
>
> The linux kernel runs at supervisor mode. The bbl runs at machine
> mode, and provides services to the linux kernel. You can't run the
> kernel without the bbl. There is a separate issue about booting.
> Most current boot loaders can only load a single image, so the bbl and
> kernel have to be linked together. Qemu has some support to load bbl
> and kernel separately, I think there is a bios option used for bbl.
> But I think most people still link them together.
>

Oh. I have some more doubts.
Is this bbl image required only for riscv-qemu, or even for the real
riscv hardware?
For riscv, why we cannot have someting like zImage or Image, like for
arm/arm64?
For creting bbl, first we create vmlinux-stripped and then give this
to riscv-pk to create a bbl image.
So, always there is a dependency of having riscv-pk for building the
final kernel image.
How can we remove this dependency?

> You can see an example of how to build and boot a kernel in the
> sifive/freedom-u-sdk project. "make qemu" will build a linux kernel
> plus bbl plus buildroot and boot it on qemu.
> https://github.com/sifive/freedom-u-sdk
>

Yes, I have used the freedom-sdk for booting with qemu.
Even this have dependecy with riscv-pk.

> Fedora has their own scripts for building kernels which you could also look at.
> https://github.com/rwmjones/fedora-riscv-kernel
>

Oh, even fedora script used riscv-pk to create bbl image.

# Build bbl with embedded kernel.
bbl: vmlinux
test $$(uname -m) = "riscv64"
rm -f $@
rm -rf riscv-pk/build
mkdir -p riscv-pk/build
cd riscv-pk/build && \
../configure \
--prefix=$(ROOT)/bbl-tmp \
--with-payload=$(ROOT)/$< \
--enable-logo



> Jim

[Jim Wilson]

On Fri, Sep 21, 2018 at 11:27 AM Pintu Kumar <pintu...@gmail.com> wrote:
> Is this bbl image required only for riscv-qemu, or even for the real
> riscv hardware?

Both qemu and hardware. The linux kernel depends on services provided
by bbl, and can't work without it. If it helps, you can think of bbl
as our equivalent to the PC bios, or as a hardware abstraction layer
(HAL). The linux kernel runs on top of the bios/hal and won't work
without it.

> For riscv, why we cannot have someting like zImage or Image, like for
> arm/arm64?

Because no one has written such support yet. RISC-V is still a new
target, and does not have all features that more mature targets have
yet.

There are people working on boot loaders. It might be possible to
include bbl in the boot loaders, so that the kernel can be separate.
Or maybe the bootloaders can be written to support two images, so that
the bbl and kernel can be loaded separately. qemu already has support
for loading bbl and kernel separately.

> For creting bbl, first we create vmlinux-stripped and then give this
> to riscv-pk to create a bbl image.
> So, always there is a dependency of having riscv-pk for building the
> final kernel image.
> How can we remove this dependency?

Again, bbl is required. But you don't necessarily have to link them
together, if you write a boot loader that can load them separately, or
use a new enough qemu that can load them separately.
https://patchwork.kernel.org/patch/10419975/
Linking them together is the way this worked in the past, and is the
way that most people are still doing this.

Jim

[Michael Clark]

Hi Jim, Pintu,

On 22/09/2018, at 5:39 AM, Jim Wilson <ji...@sifive.com> wrote:

The linux kernel runs at supervisor mode.  The bbl runs at machine
mode, and provides services to the linux kernel.  You can't run the
kernel without the bbl.  There is a separate issue about booting.
Most current boot loaders can only load a single image, so the bbl and
kernel have to be linked together.  Qemu has some support to load bbl
and kernel separately, I think there is a bios option used for bbl.
But I think most people still link them together.

We need to work on the -bios option in QEMU. Currently the separated firmware and kernel mechanism only works in the “virt” machine and requires a recent bbl compiled with no payload. The documentation is in the code currently. It’s similar in concept to an FSBL that loads multiple images and places the addresses in device-tree, and it is based on the same mechanism used to point the kernel at a loaded initrd. It uses the device-tree “chosen” node which contains configuration versus hardware description (chosen can also point to the chosen console device):

chosen {

  “riscv,kernel-start” = [ ... ],

  “riscv,kernel-end” = [ ... ],

  “linux,initrd-start” = [ ... ],

  “linux,initrd-end” = [ ... ]

}

See here: https://github.com/riscv/riscv-qemu/blob/riscv-all/hw/riscv/boot.c

There were some email bug reports about ROM overlaps and I discovered while testing, that QEMU tries to add the ROMs in the -bios <rom_file> option even if this is not supported by the machine, so if this experimental BBL/vmlinux split -bios option is used on another QEMU machine such as “sifive_e” or “sifive_u”, then this ROM overlap message can appear. This appears to be some issue with target independent code. I need to look into it. We didn’t add the option to the other machines as we want them to match hardware, and currently HiFive Unleashed FSBL only loads one binary.

Tangentially, there is an -initrd option that was added to “virt” but it does not work as the associated Linux patch was not added. This might have been a good decision, as the code was copied from target/arm and makes different assumptions on load address of the initrd/initramfs. The code loaded the initrd in the middle of memory so that it did not overlap with a kernel that decompressed itself. It may be more wise to specify that the initrd is loaded below the kernel image.

These issues will get fleshed out as the boot protocol evolves and there are multiple RISC-V Supervisor loaders. u-boot, coreboot and other loaders will allow us to work on how the early boot mechanism becomes standardized.

Here are some things we need to keep in mind:

- firmware, kernel split in hardware

- supporting FreeBSD and other S-mode OSes

- initrd loading (linux specific)

- Mask ROM overrides in QEMU

Some folk also want to control the first instruction after reset in spike or QEMU. This is more equivalent to a Mask ROM override and is slightly different from a firmware override. Currently the -bios option in virt still relies on a tiny Mask ROM at 0x1000. For simulation, it would be useful to override this. QEMU has generic code for handling multiple ROMs, and I think may even support .hex files on some architectures. We’ll have to think carefully how we evolve QEMU (and spike) so we can keep backwards compatibility that use the current options.

To get back to the original posters question, I think freedom-u-sdk has the best build recipe for bbl/vmlinux. I have some scripts I am working on that I have not published yet for running a statically linked /sbin/init from a ramdisk that just prints hello and then powers off. We need that in the CI...

Oh forgot about this. We have the bbl build steps in documentation vs source form here:

- https://github.com/riscv/riscv-qemu/wiki

We need to update the Linux versions mentioned in the docs.

Michael

[Karsten Merker]

Hello,

just for the record: the associated Linux patch has in the
meantime been added to the mainline Linux kernel and is part of
the 4.19-rc4 release:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e866d3e84eb7c9588afb77604d417e8cc49fe216

I'm right now running 4.19-rc4 with an initrd in a qemu "virt"
machine and it's working without problems.

Regards,
Karsten
--
Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung
sowie der Weitergabe meiner personenbezogenen Daten für Zwecke der
Werbung sowie der Markt- oder Meinungsforschung.

[Michael Clark]

Hi Karsten, oh good. I’ll update my kernel and try. BTW I still have to work on the riscv-qemu patch queue (in-progress). Thanks, Michael.

[Michael Clark]

Thanks Jim, this makes it clear. We still need bbl. Indeed when we evolve the boot protocol standard, it can serve as a compatible reference implementation. BBL is primarily required as it is the reference implementation for the SBI.

Yes. We made the new feature relatively easy to support and consistent with other mechanisms (linux,initrd) but general for use by other RISC-V OS (riscv,kernel), while maintaining backwards compatibility with how it is done presently.

Breaking changes are sometimes necessary, but ideally need some public discussion, so folk are aware. We have to step carefully.

While new extensions won’t necessarily break things, we do have to consider how they contribute to cruft and/or bloat. We should think carefully about what features we add. The mechanism we used for QEMU to load the kernel was similar to PowerPC, but not QEMU specific. We used “riscv,” instead of “qemu,” as the feature could potentially be used by an FSBL that can load n+1 blobs.

<side-notes>
The usual approach to this problem is to have a boot services API to load files (like UEFI, GPT and GUIDs) or for the boot loader to have filesystems and drivers (u-boot, grub). Folk can do whatever they like if they have silicon and can replace the firmware. Ideally we don’t end up with a repeat of the arm boot mess there every vendor has their own boot loader and firmware verification mechanism (some easier to crack than others). Conceptually UEFI Secure Boot is quite nice, and is extensible by vendors (Apple, Dell both use on laptops and Dell on servers), and there is also a cut-down version of UEFI called EBBR that uses device-tree instead of ACPI (arm had to adopt ACPI to get into the data-center). That said, even if we did adopt some standard like UEFI/EBBR (a subset of UEFI), it would be nice to have some small audited implementations, as the large UEFI implementations have a larger attack service. I do like the way that Intel has an open source reference implementation of Secure Boot where an open source developer can make themselves the Root of Trust. Canonical and Ubuntu are using Secure Boot “Capsule Update” to deliver timely firmware and microcode updates (same thing in RISC-V land) via the Capsule Update mechanism. UEFI verifies the capsule the has many shared relocatable “modules” so that a vendor can add modules for the hardware in their design. While they could indeed be open-source, the typical user doesn’t want to type “make world” when their is a vendor update. binutils has quite good PE support and OVMF can be built and run in QEMU. It’s all open source.

Tangentially: of course if there is an external dependency that supports many architectures, then target independent changes can affect RISC-V, so it’s understandable the difficultly maintaining perfect backwards compatibility with RISC-V GCC and Binutils. I understand that you can do things like ignore some options (-mexplicit-relocs) if it’s buggy, or break options if say for examples sake, LLVM is not emitting the correct relocs for gas, e.g. using segment+offset In LO relocs instead of labels, making linker relaxation more tricky, given the offsets are not valid after relaxation but the labels are always valid. This is just a made up example for examples sake; this is solvable with abs offsets but trickier as one needs linker passes and to maintain a map of where absolute section offsets have been translated to after relaxation. Quite a tricky problem.
</side-notes>