OSv boots and runs simple hello world on Firecracker ARM edition ... on Raspberry PI 4

115 views
Skip to first unread message

Waldek Kozaczuk

unread,
May 21, 2020, 4:58:25 PM5/21/20
to OSv Development

ubuntu@ubuntu-rasp4:~$ uname -a

Linux ubuntu-rasp4 5.3.0-1025-raspi2 #27-Ubuntu SMP Fri May 8 08:32:04 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@ubuntu-rasp4:~$ ./firecracker-v0.21.1-aarch64 --no-api --config-file ./osv-config.json

OSv v0.55.0-9-g840428ad

PSCI: version 65536.0 detected.

setup_arm_clock() ENTERED, lr=00000000800d54e0

arm_clock(): frequency read as 000000000337f980

interrupt_table::interrupt_table() ENTERED, lr=00000000800a8d1c

gic_driver::init_cpu() ENTERED, lr=0000000080201c80

CPU interface enabled.

gic_driver::init_dist() ENTERED, lr=0000000080201c8c

number of supported IRQs: 0000000000000080

interrupt table: gic driver created.

registered IRQ id=0000000000000004

registered IRQ id=000000000000001b

registered IRQ id=0000000000000000

Premain complete!

smp_launch ENTERED, lr=00000000800d709c

Booted up in 0.00 ms

Cmdline: console=ttyS0 --verbose --nomount --maxnic=0 /tools/hello.so earlycon=uart,mmio,0x40001000

faulting address 000010000001fea0

faulting address 0000100000000328

faulting address 0000100000020000

faulting address 0000100000050018

faulting address 00001000000301c8

faulting address 000010000004fe00

Hello from C code

2020-05-21T20:24:04.080765641 [anonymous-instance:ERROR:src/vmm/src/vstate.rs:951] Unexpected exit reason on vcpu run: SystemEvent


In general it took me a bit of research as I am not really familiar with ARM architecture and even reading the assembly was a bit a challenge to say the least. And then debugging without debugger and any console (:-( .. so not debug() for long time. But all in all it was not too bad and the changes that I had to make to OSv are in my opinion much smaller and easier comparing to x86_64.

Below you will see the "hack-patch" showing what changes I had to make. Logically, following things had to be changed:
  • The most important thing was to move kernel from 0x40000000 (1GB) to 0x80000000 (2GB) which required changing one line in Makefile (see below) and changing boot paging table to map the 2GB-3GB area of memory; only then I could actually start debugging :-) I wonder if it will also work on QEMU/KVM - possinly qemu boot loader will inspect ELF and place it accordingly in memory; firecracker does not read ELF and simply places it at 2GB
  • To get console working I guessed that that need to create equivalent class for isa_serial_console but communicating over mmio (see mmio_isa_serial_console.hh/cc) which in essence invokes mmio_set*/mmio_get* (it would be nice to extract common code somehow - suggestions welcome)
  • Some more trivial changes - for now mostly disabling things - for now quite disorganized

diff --git a/Makefile b/Makefile
index db3c68cf..ffd570a5 100644
--- a/Makefile
+++ b/Makefile
@@ -354,6 +354,7 @@ tools := tools/mkfs/mkfs.so tools/cpiod/cpiod.so
 $(out)/tools/%.o: COMMON += -fPIC
 
 tools += tools/uush/uush.so
+tools += tools/uush/hello.so
 tools += tools/uush/ls.so
 tools += tools/uush/mkdir.so
 
@@ -451,8 +452,8 @@ endif # x64
 
 ifeq ($(arch),aarch64)
 
-kernel_base := 0x40080000
-kernel_vm_base := 0x40080000
+kernel_base := 0x80080000
+kernel_vm_base := 0x80080000
 app_local_exec_tls_size := 0x0
 
 include $(libfdt_base)/Makefile.libfdt
@@ -816,6 +817,7 @@ drivers += drivers/xenplatform-pci.o
 endif # x64
 
 ifeq ($(arch),aarch64)
+drivers += drivers/mmio-isa-serial.o
 drivers += drivers/pl011.o
 drivers += drivers/xenconsole.o
 drivers += drivers/virtio.o
diff --git a/arch/aarch64/arch-dtb.cc b/arch/aarch64/arch-dtb.cc
index b59f1dcc..cd0719e8 100644
--- a/arch/aarch64/arch-dtb.cc
+++ b/arch/aarch64/arch-dtb.cc
@@ -225,7 +225,8 @@ bool dtb_get_gic_v2(u64 *dist, size_t *dist_len, u64 *cpu, size_t *cpu_len)
     if (!dtb)
         return false;
 
-    node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
+    //node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
+    node = fdt_node_offset_by_compatible(dtb, -1, "arm,gic-400");
     if (node < 0)
         return false;
 
@@ -613,11 +614,13 @@ void  __attribute__((constructor(init_prio::dtb))) dtb_setup()
         abort("dtb_setup: failed to parse cpu mpid.\n");
     }
 
+    //TODO: Parse PCI only if required and
+    // abort if missing only if required
     dtb_timer_irq = dtb_parse_timer_irq();
     dtb_pci_irqmask = dtb_parse_pci_irqmask();
     dtb_pci_irqmap_count = dtb_parse_pci_irqmap_count();
     if (!dtb_parse_pci_irqmap(dtb_pci_bdfs, dtb_pci_irq_ids, dtb_pci_irqmap_count)) {
-        abort("dtb_setup: failed to parse pci_irq_map.\n");
+        //abort("dtb_setup: failed to parse pci_irq_map.\n");
     }
 
     register u64 edata;
diff --git a/arch/aarch64/arch-setup.cc b/arch/aarch64/arch-setup.cc
index bdbb1266..b12546ae 100644
--- a/arch/aarch64/arch-setup.cc
+++ b/arch/aarch64/arch-setup.cc
@@ -17,6 +17,7 @@
 #include <osv/debug.hh>
 #include <osv/commands.hh>
 #include <osv/xen.hh>
+#include <osv/version.h>
 
 #include "arch-mmu.hh"
 #include "arch-dtb.hh"
@@ -25,6 +26,7 @@
 #include "drivers/pl011.hh"
 #include "early-console.hh"
 #include <osv/pci.hh>
+#include "drivers/mmio-isa-serial.hh"
 
 #include <alloca.h>
 
@@ -93,9 +95,10 @@ void arch_setup_free_memory()
 
     if (!is_xen()) {
         /* linear_map [TTBR0 - UART] */
-        addr = (mmu::phys)console::aarch64_console.pl011.get_base_addr();
-        mmu::linear_map((void *)addr, addr, 0x1000, mmu::page_size,
-                        mmu::mattr::dev);
+        //TODO: Only run it if pl011 is detected
+        //addr = (mmu::phys)console::aarch64_console.pl011.get_base_addr();
+        //mmu::linear_map((void *)addr, addr, 0x1000, mmu::page_size,
+        //                mmu::mattr::dev);
     }
 
     /* linear_map [TTBR0 - GIC DIST and GIC CPU] */
@@ -110,12 +113,17 @@ void arch_setup_free_memory()
     mmu::linear_map((void *)cpu, (mmu::phys)cpu, cpu_len, mmu::page_size,
                     mmu::mattr::dev);
 
-    arch_setup_pci();
+    //TODO: Call it only if PCI available
+    //arch_setup_pci();
 
     // get rid of the command line, before memory is unmapped
     osv::parse_cmdline(cmdline);
 
     mmu::switch_to_runtime_page_tables();
+
+    //TODO: This is probably wrong place to init the console
+    arch_init_early_console();
+    debug_early("OSv " OSV_VERSION "\n");
 }
 
 void arch_setup_tls(void *tls, const elf::tls_data& info)
@@ -185,27 +193,35 @@ void arch_init_early_console()
         return;
     }
 
+    /* Comment out for now
     new (&console::aarch64_console.pl011) console::PL011_Console();
     console::arch_early_console = console::aarch64_console.pl011;
     int irqid;
-    u64 addr = dtb_get_uart(&irqid);
-    if (!addr) {
+    u64 addr = dtb_get_uart(&irqid);*/
+    //if (!addr) {
         /* keep using default addresses */
-        return;
-    }
+    //    return;
+    //}
 
-    console::aarch64_console.pl011.set_base_addr(addr);
-    console::aarch64_console.pl011.set_irqid(irqid);
+    //console::aarch64_console.pl011.set_base_addr(addr);
+    //console::aarch64_console.pl011.set_irqid(irqid);
+
+    new (&console::aarch64_console.mmio_isa_serial) console::mmio_isa_serial_console();
+    console::arch_early_console = console::aarch64_console.mmio_isa_serial;
+
+    console::mmio_isa_serial_console::early_init();
 }
 
 bool arch_setup_console(std::string opt_console)
 {
+    /*
     if (opt_console.compare("pl011") == 0) {
         console::console_driver_add(&console::arch_early_console);
     } else if (opt_console.compare("all") == 0) {
         console::console_driver_add(&console::arch_early_console);
     } else {
         return false;
-    }
+    }*/
+    console::console_driver_add(&console::arch_early_console);
     return true;
 }
diff --git a/arch/aarch64/boot.S b/arch/aarch64/boot.S
index b29fd16a..1d2af75c 100644
--- a/arch/aarch64/boot.S
+++ b/arch/aarch64/boot.S
@@ -188,21 +188,15 @@ ident_pt_l4_ttbr0:
         .quad 0
         .endr
 ident_pt_l3_ttbr0:
-        .quad ident_pt_l2_0_ttbr0 + 0x3
-        .quad ident_pt_l2_1_ttbr0 + 0x3
-        .rept 510
+        .quad 0
+        .quad 0
+        .quad ident_pt_l2_2_ttbr0 + 0x3 // Map 4GB-6GB one-to-one
+        .rept 509
         .quad 0
         .endr
-ident_pt_l2_0_ttbr0:
-        index = 0
-        offset = 0x00000000
-        .rept 512
-        .quad (index << 21) + 0x401
-        index = index + 1
-        .endr
-ident_pt_l2_1_ttbr0:
+ident_pt_l2_2_ttbr0:
         index = 0
-        offset = 0x40000000
+        offset = 0x80000000
         .rept 512
         .quad offset + (index << 21) + 0x411
         index = index + 1
diff --git a/arch/aarch64/early-console.cc b/arch/aarch64/early-console.cc
index d5196c74..8c91c965 100644
--- a/arch/aarch64/early-console.cc
+++ b/arch/aarch64/early-console.cc
@@ -13,6 +13,7 @@
 namespace console {
 
 union AARCH64_Console aarch64_console;
-console_driver & arch_early_console = aarch64_console.pl011;
+//console_driver & arch_early_console = aarch64_console.pl011;
+console_driver & arch_early_console = aarch64_console.mmio_isa_serial;
 
 }
diff --git a/arch/aarch64/early-console.hh b/arch/aarch64/early-console.hh
index cbc2cac4..cd39738b 100644
--- a/arch/aarch64/early-console.hh
+++ b/arch/aarch64/early-console.hh
@@ -11,12 +11,15 @@
 #include <drivers/console-driver.hh>
 #include <drivers/pl011.hh>
 #include <drivers/xenconsole.hh>
+#include <drivers/mmio-isa-serial.hh>
+
 
 namespace console {
 
 union AARCH64_Console {
     PL011_Console pl011;
     XEN_Console xen;
+    mmio_isa_serial_console mmio_isa_serial;
 
     AARCH64_Console() {};  /* placement new is used to initialize object */
     ~AARCH64_Console() {}; /* won't ever be called */
diff --git a/arch/aarch64/mmu.cc b/arch/aarch64/mmu.cc
index fef48e5a..012d3c57 100644
--- a/arch/aarch64/mmu.cc
+++ b/arch/aarch64/mmu.cc
@@ -18,11 +18,11 @@ void page_fault(exception_frame *ef)
 {
     sched::fpu_lock fpu;
     SCOPE_LOCK(fpu);
-    debug_early_entry("page_fault");
+    //debug_early_entry("page_fault");
     u64 addr;
     asm volatile ("mrs %0, far_el1" : "=r"(addr));
     debug_early_u64("faulting address ", (u64)addr);
-    debug_early_u64("elr exception ra ", (u64)ef->elr);
+    //debug_early_u64("elr exception ra ", (u64)ef->elr);
 
     if (fixup_fault(ef)) {
         debug_early("fixed up with fixup_fault\n");
@@ -45,7 +45,7 @@ void page_fault(exception_frame *ef)
         mmu::vm_fault(addr, ef);
     }
 
-    debug_early("leaving page_fault()\n");
+    //debug_early("leaving page_fault()\n");
 }
 
 namespace mmu {
diff --git a/drivers/mmio-isa-serial.cc b/drivers/mmio-isa-serial.cc
new file mode 100644
index 00000000..c59cec8d
--- /dev/null
+++ b/drivers/mmio-isa-serial.cc
@@ -0,0 +1,134 @@
+/*
+ * Copyright (C) 2013 Cloudius Systems, Ltd.
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#include "mmio-isa-serial.hh"
+
+namespace console {
+
+// UART registers, offsets to ioport:
+enum regs {
+    IER = 1,    // Interrupt Enable Register
+    FCR = 2,    // FIFO Control Register
+    LCR = 3,    // Line Control Register
+    MCR = 4,    // Modem Control Register
+    LSR = 5,    // Line Control Register
+    MSR = 6,    // Modem Status Register
+    SCR = 7,    // Scratch Register
+    DLL = 0,    // Divisor Latch LSB Register
+    DLM = 1,    // Divisor Latch MSB Register
+};
+
+enum lcr {
+    // When bit 7 (DLAB) of LCR is set to 1, the two registers 0 and 1
+    // change their meaning and become two bytes controlling the baud rate
+    DLAB     = 0x80,    // Divisor Latch Access Bit in LCR register
+    LEN_8BIT = 3,
+};
+
+// Various bits of the Line Status Register
+enum lsr {
+    RECEIVE_DATA_READY  = 0x1,
+    OVERRUN             = 0x2,
+    PARITY_ERROR        = 0x4,
+    FRAME_ERROR         = 0x8,
+    BREAK_INTERRUPT     = 0x10,
+    TRANSMIT_HOLD_EMPTY = 0x20,
+    TRANSMIT_EMPTY      = 0x40,
+    FIFO_ERROR          = 0x80,
+};
+
+// Various bits of the Modem Control Register
+enum mcr {
+    DTR                 = 0x1,
+    RTS                 = 0x2,
+    AUX_OUTPUT_1        = 0x4,
+    AUX_OUTPUT_2        = 0x8,
+    LOOPBACK_MODE       = 0x16,
+};
+
+mmioaddr_t mmio_isa_serial_console::_addr_mmio;
+
+void mmio_isa_serial_console::early_init()
+{
+    u64 address = 0x40001000; //TODO: Should parse from boot command line ('earlycon=uart,mmio,0x40001000')
+    u64 size = 4096;
+
+    _addr_mmio = mmio_map(address, size);
+
+    // Set the UART speed to to 115,200 bps, This is done by writing 1,0 to
+    // Divisor Latch registers, but to access these we need to temporarily
+    // set the Divisor Latch Access Bit (DLAB) on the LSR register, because
+    // the UART has fewer ports than registers...
+    mmio_setb(_addr_mmio + (int)regs::LCR, lcr::LEN_8BIT | lcr::DLAB);
+    mmio_setb(_addr_mmio + (int)regs::DLL, 1);
+    mmio_setb(_addr_mmio + (int)regs::DLM, 0);
+    mmio_setb(_addr_mmio + (int)regs::LCR, lcr::LEN_8BIT);
+
+    //  interrupt threshold
+    mmio_setb(_addr_mmio + (int)regs::FCR, 0);
+
+    // disable interrupts
+    mmio_setb(_addr_mmio + (int)regs::IER, 0);
+
+    // Most physical UARTs need the MCR AUX_OUTPUT_2 bit set to 1 for
+    // interrupts to be generated. QEMU doesn't bother checking this
+    // bit, but interestingly VMWare does, so we must set it.
+    mmio_setb(_addr_mmio + (int)regs::MCR, mcr::AUX_OUTPUT_2);
+}
+
+void mmio_isa_serial_console::write(const char *str, size_t len)
+{
+    while (len-- > 0)
+        putchar(*str++);
+}
+
+bool mmio_isa_serial_console::input_ready()
+{
+    u8 val = mmio_getb(_addr_mmio + (int)regs::LSR);
+    // On VMWare hosts without a serial port, this register always
+    // returns 0xff.  Just ignore it instead of spinning incessantly.
+    return (val != 0xff && (val & lsr::RECEIVE_DATA_READY));
+}
+
+char mmio_isa_serial_console::readch()
+{
+    u8 val;
+    char letter;
+
+    do {
+        val = mmio_getb(_addr_mmio + (int)regs::LSR);
+    } while (!(val & (lsr::RECEIVE_DATA_READY | lsr::OVERRUN | lsr::PARITY_ERROR | lsr::FRAME_ERROR)));
+
+    letter = mmio_getb(_addr_mmio);
+
+    return letter;
+}
+
+void mmio_isa_serial_console::putchar(const char ch)
+{
+    u8 val;
+
+    do {
+        val = mmio_getb(_addr_mmio + (int)regs::LSR);
+    } while (!(val & lsr::TRANSMIT_HOLD_EMPTY));
+
+    mmio_setb(_addr_mmio, ch);
+}
+
+void mmio_isa_serial_console::enable_interrupt()
+{
+    // enable interrupts
+    mmio_setb(_addr_mmio + (int)regs::IER, 1);
+}
+
+void mmio_isa_serial_console::dev_start() {
+    //TODO: Figure out which interrupt and what kind to use
+    //_irq.reset(new sgi_edge_interrupt(4, [&] { _thread->wake(); }));
+    enable_interrupt();
+}
+
+}
diff --git a/drivers/mmio-isa-serial.hh b/drivers/mmio-isa-serial.hh
new file mode 100644
index 00000000..f422dfda
--- /dev/null
+++ b/drivers/mmio-isa-serial.hh
@@ -0,0 +1,39 @@
+/*
+ * Copyright (C) 2013 Cloudius Systems, Ltd.
+ *
+ * This work is open source software, licensed under the terms of the
+ * BSD license as described in the LICENSE file in the top-level directory.
+ */
+
+#ifndef DRIVERS_MMIO_ISA_SERIAL_HH
+#define DRIVERS_MMIO_ISA_SERIAL_HH
+
+#include "console-driver.hh"
+#include <osv/pci.hh>
+#include <osv/sched.hh>
+#include <osv/interrupt.hh>
+#include <osv/mmio.hh>
+
+namespace console {
+
+class mmio_isa_serial_console : public console_driver {
+public:
+    static void early_init();
+    virtual void write(const char *str, size_t len);
+    virtual void flush() {}
+    virtual bool input_ready() override;
+    virtual char readch();
+
+private:
+    //std::unique_ptr<sgi_edge_interrupt> _irq;
+    static mmioaddr_t _addr_mmio;
+
+    virtual void dev_start();
+    void enable_interrupt();
+    static void putchar(const char ch);
+    virtual const char *thread_name() { return "mmio-isa-serial-input"; }
+};
+
+}
+
+#endif
diff --git a/loader.cc b/loader.cc
index 66bfb52c..9867ca49 100644
--- a/loader.cc
+++ b/loader.cc
@@ -94,12 +94,12 @@ extern "C" {
 
 void premain()
 {
-    arch_init_early_console();
+    //arch_init_early_console();
 
     /* besides reporting the OSV version, this string has the function
        to check if the early console really works early enough,
        without depending on prior initialization. */
-    debug_early("OSv " OSV_VERSION "\n");
+    //debug_early("OSv " OSV_VERSION "\n");
 
     arch_init_premain();
 
@@ -116,6 +116,7 @@ void premain()
     for (auto init = inittab.start; init < inittab.start + inittab.count; ++init) {
         (*init)();
     }
+    debug_early("Premain complete!\n");
     boot_time.event(".init functions");
 }
 
@@ -230,6 +231,7 @@ static void parse_options(int loader_argc, char** loader_argv)
         opt_maxnic = true;
         maxnic = options::extract_option_int_value(options_values, "maxnic", handle_parse_error);
     }
+    maxnic = 0;
 
     if (extract_option_flag(options_values, "trace-backtrace")) {
         opt_log_backtrace = true;
@@ -261,6 +263,7 @@ static void parse_options(int loader_argc, char** loader_argv)
     }
 
     opt_mount = !extract_option_flag(options_values, "nomount");
+    opt_mount = false;
     opt_pivot = !extract_option_flag(options_values, "nopivot");
     opt_random = !extract_option_flag(options_values, "norandom");
     opt_init = !extract_option_flag(options_values, "noinit");
@@ -351,7 +354,8 @@ std::vector<std::vector<std::string> > prepare_commands(char* app_cmdline)
     bool ok;
 
     printf("Cmdline: %s\n", app_cmdline);
-    commands = osv::parse_command_line(app_cmdline, ok);
+    //commands = osv::parse_command_line(app_cmdline, ok);
+    commands = osv::parse_command_line("/tools/hello.so", ok);
 
     if (!ok) {
         puts("Failed to parse command line.");
diff --git a/usr_ramfs.manifest.skel b/usr_ramfs.manifest.skel
index 4bdfdd32..1a04c22f 100644
--- a/usr_ramfs.manifest.skel
+++ b/usr_ramfs.manifest.skel
@@ -3,6 +3,8 @@
 /libvdso.so: libvdso.so
 /tools/mount-fs.so: tools/mount/mount-fs.so
 /tools/umount.so: tools/mount/umount.so
+/tools/uush.so: tools/uush/uush.so
+/tools/hello.so: tools/uush/hello.so
 /usr/lib/libgcc_s.so.1: %(libgcc_s_dir)s/libgcc_s.so.1
 /&/etc/hosts: ../../static/&



Please not that I have not tested how it works with multiple CPUs or if virtual block and network devices operate.

Waldek

Dor Laor

unread,
May 21, 2020, 5:26:59 PM5/21/20
to Waldek Kozaczuk, OSv Development
Kudos Waldek! 

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/f4c6cafe-690a-4619-b7b6-e414e3f9343a%40googlegroups.com.

Pekka Enberg

unread,
May 22, 2020, 12:17:36 AM5/22/20
to Dor Laor, Waldek Kozaczuk, OSv Development
On Fri, May 22, 2020 at 12:26 AM Dor Laor <d...@scylladb.com> wrote:
Kudos Waldek! 

Indeed, nice work!

- Pekka 

Roman Shaposhnik

unread,
May 22, 2020, 12:21:23 AM5/22/20
to Waldek Kozaczuk, OSv Development
This is super awesome, Waldek! I'm now inspired to run OSv as a VM
backed by Firecracker on Project EVE/RPi! ;-)

Btw, can you share your osv-config.json and also details of what
version of Firecracker you used?

Better yet -- if there's a github repo with all the bits and pieces
required to replicate your setup -- that'd be worthy of a gift of a
beer (post COVID tho!) ;-)

Thanks,
Roman.

On Thu, May 21, 2020 at 1:58 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:
>
>

Waldek Kozaczuk

unread,
May 22, 2020, 3:04:39 PM5/22/20
to OSv Development
I thing I forgot to mention is that currently OSv arm version only supports GIC 2 interrupt controller which is enough for RP 4. But we need GIC 3/4 to be able to run on Firecracker on "real"/non-toy hardware. I do not know how difficult it would be to support GIC 3/4.


On Friday, May 22, 2020 at 12:21:23 AM UTC-4, Roman Shaposhnik wrote:
This is super awesome, Waldek! I'm now inspired to run OSv as a VM
backed by Firecracker on Project EVE/RPi! ;-)

Btw, can you share your osv-config.json and also details of what
version of Firecracker you used?
 
{
  "boot-source": {
    "kernel_image_path": "/home/ubuntu/loader_with_console.img",
    "boot_args": "console=ttyS0 --verbose --nomount --maxnic=0 /tools/hello.so"
  },
  "drives": []
}
 

firecracker-v0.21.1-aarch64 - which I believe is the latest.



Better yet -- if there's a github repo with all the bits and pieces
required to replicate your setup -- that'd be worthy of a gift of a
beer (post COVID tho!) ;-)
There is a build-able branch (which is in essence current master + patch I have enclosed) - https://github.com/wkozaczuk/osv/tree/arm_on_firecracker2

./scripts/build image=empty arch=aarch64 fs=ramfs (it add simple hello world binary).

and then run

./firecracker-v0.21.1-aarch64 --no-api --config-file ./osv-config.json

> To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

Nadav Har'El

unread,
May 25, 2020, 3:19:05 AM5/25/20
to Waldek Kozaczuk, OSv Development
Very nice!

I'll try to review a few points below, but I'm even a less of an ARM expert (and definitely firecracker expert) than you so I'm not sure how valuable my review will be.
We just need to take care not to break what was already working in the existing aarch64 code, because I don't know how it was tested, and how various changes can break other arm variants and hypervisors.
 

Below you will see the "hack-patch" showing what changes I had to make. Logically, following things had to be changed:
  • The most important thing was to move kernel from 0x40000000 (1GB) to 0x80000000 (2GB) which required changing one line in Makefile (see below) and changing boot paging table to map the 2GB-3GB area of memory; only then I could actually start debugging :-) I wonder if it will also work on QEMU/KVM - possinly qemu boot loader will inspect ELF and place it accordingly in memory; firecracker does not read ELF and simply places it at 2GB
  • To get console working I guessed that that need to create equivalent class for isa_serial_console but communicating over mmio (see mmio_isa_serial_console.hh/cc) which in essence invokes mmio_set*/mmio_get* (it would be nice to extract common code somehow - suggestions welcome)

Perhaps you can have class class mmio_isa_serial_console inherit from isa_serial_console, but change the implementation to use a method out() instead of pci::outb() - and the new mmio_isa_serial_console will just override the out() function (perhaps out() needs to be virtual for this to work?).

 
  • Some more trivial changes - for now mostly disabling things - for now quite disorganized

diff --git a/Makefile b/Makefile
index db3c68cf..ffd570a5 100644
--- a/Makefile
+++ b/Makefile
@@ -354,6 +354,7 @@ tools := tools/mkfs/mkfs.so tools/cpiod/cpiod.so
 $(out)/tools/%.o: COMMON += -fPIC
 
 tools += tools/uush/uush.so
+tools += tools/uush/hello.so
 tools += tools/uush/ls.so
 tools += tools/uush/mkdir.so
 
@@ -451,8 +452,8 @@ endif # x64
 
 ifeq ($(arch),aarch64)
 
-kernel_base := 0x40080000
-kernel_vm_base := 0x40080000
+kernel_base := 0x80080000
+kernel_vm_base := 0x80080000

I'm a bit worried about these magic numbers... I don't know if 0x40080000 was important previously and things would break with a different number or was it just picked randomly, and why 0x80080000 is necessary now.
 
 app_local_exec_tls_size := 0x0
 
 include $(libfdt_base)/Makefile.libfdt
@@ -816,6 +817,7 @@ drivers += drivers/xenplatform-pci.o
 endif # x64
 
 ifeq ($(arch),aarch64)
+drivers += drivers/mmio-isa-serial.o
 drivers += drivers/pl011.o
 drivers += drivers/xenconsole.o
 drivers += drivers/virtio.o
diff --git a/arch/aarch64/arch-dtb.cc b/arch/aarch64/arch-dtb.cc
index b59f1dcc..cd0719e8 100644
--- a/arch/aarch64/arch-dtb.cc
+++ b/arch/aarch64/arch-dtb.cc
@@ -225,7 +225,8 @@ bool dtb_get_gic_v2(u64 *dist, size_t *dist_len, u64 *cpu, size_t *cpu_len)
     if (!dtb)
         return false;
 
-    node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
+    //node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
+    node = fdt_node_offset_by_compatible(dtb, -1, "arm,gic-400");

I don't know what this means... But is there a way to support both?
 
--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages