OSv boots in 3ms on QEMU microvm machine

[Waldek Kozaczuk]

As you may have heard QEMU developers added new machine type: microvm - https://github.com/qemu/qemu/blob/master/docs/microvm.rst. It has been merged into master but not part of any official QEMU release. So if you want to try you need to build QEMU from the source.

Recently I have played with it and eventually managed to boot OSv successfully on it after some adjustments (see some of my recent virtio code tweaks). And the same OSv ROFS image that would boot in ~5 ms on firecracker, can boot in only ~3ms on QEMU microvm.

OSv v0.54.0-34-gccb9cd12

Booted up in 2.91 ms

Cmdline: /hello 

Hello from C code

real 0m0.057s

user 0m0.037s

sys 0m0.020s

and with these options:

qemu-system-x86_64 \

-m 64M \

-smp 1 \

--nographic \

-kernel ./scripts/../build/last/loader-stripped.elf \

-append "--nopci /hello" \

-M microvm,x-option-roms=off,pit=off,pic=off,rtc=off \

-nodefaults \

-no-user-config \

-no-reboot \

-global virtio-mmio.force-legacy=off \

-device virtio-blk-device,id=blk0,drive=hd0,scsi=off \

-drive file=./build/last/usr.img,if=none,id=hd0,cache=writeback,aio=threads \

-enable-kvm \

-cpu host,+x2apic \

-serial stdio

Adding 1 NIC makes it boot slower by ~1 ms:

OSv v0.54.0-34-gccb9cd12

eth0: 192.168.122.15

Booted up in 4.30 ms

Cmdline: /hello  

Hello from C code

Regards,

Waldek

[Asias He]

On Fri, Nov 22, 2019 at 9:34 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:
>
> As you may have heard QEMU developers added new machine type: microvm - https://github.com/qemu/qemu/blob/master/docs/microvm.rst. It has been merged into master but not part of any official QEMU release. So if you want to try you need to build QEMU from the source.
>
> Recently I have played with it and eventually managed to boot OSv successfully on it after some adjustments (see some of my recent virtio code tweaks). And the same OSv ROFS image that would boot in ~5 ms on firecracker, can boot in only ~3ms on QEMU microvm.

Wow! This is pretty impressive!

> --
> You received this message because you are subscribed to the Google Groups "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/8d146284-3b04-4afb-b31d-158412e8f6fa%40googlegroups.com.



--
Asias

[Waldek Kozaczuk]

Forgot to mention that if you want to try it for yourself, there are parameters you need to pass to the newly modified run.py (assuming you built the latest QEMU from latest source):

./scripts/run.py -p qemu_microvm --nogdb --qemu-path <wherever_your_qemu_is>/qemu-system-x86_64 --nics 0 -c 1 -m 64M --block-device-cache writeback

or with a NIC:

./scripts/run.py -p qemu_microvm --nogdb --qemu-path <wherever_your_qemu_is>/qemu-system-x86_64 -c 1 -m 64M --block-device-cache writeback

Enjoy!

> To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

[Nadav Har'El]

On Fri, Nov 22, 2019 at 3:34 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

As you may have heard QEMU developers added new machine type: microvm - https://github.com/qemu/qemu/blob/master/docs/microvm.rst. It has been merged into master but not part of any official QEMU release. So if you want to try you need to build QEMU from the source.

Recently I have played with it and eventually managed to boot OSv successfully on it after some adjustments (see some of my recent virtio code tweaks). And the same OSv ROFS image that would boot in ~5 ms on firecracker, can boot in only ~3ms on QEMU microvm.

Very nice!

OSv v0.54.0-34-gccb9cd12

Booted up in 2.91 ms

Cmdline: /hello 

Hello from C code

real 0m0.057s

user 0m0.037s

sys 0m0.020s

Can you please remind me what this 2.91ms includes? When do we start measuring and when do we stop measuring?

The 57ms at the end, that is 20 slower - what does *that* include that the 2.91ms figure didn't?

[Waldek Kozaczuk]

First please note that this and other old (5ms) firecracker boot times were recorded when running on my old 2013 MacBook Pro with 4-way 2.3GHz i7 cpu with Ubuntu 19.04 on it. I think it still a decent laptop and the cpus have not gotten that much faster (Moore's law is dead now, right?), but it would be interesting to measure boot times on something more modern and powerful.

The 'Booted up' message is what I added couple of months ago and simply prints the total from bootchart. The boot time with full bootchart will explain what is included:

OSv v0.54.0-40-g08466354

disk read (real mode): 0.00ms, (+0.00ms)

uncompress lzloader.elf: 0.00ms, (+0.00ms)

TLS initialization: 0.51ms, (+0.51ms)

.init functions: 1.22ms, (+0.71ms)

SMP launched: 1.55ms, (+0.32ms)

VFS initialized: 1.76ms, (+0.21ms)

Network initialized: 2.01ms, (+0.25ms)

pvpanic done: 2.08ms, (+0.07ms)

drivers probe: 2.12ms, (+0.04ms)

drivers loaded: 2.70ms, (+0.59ms)

ROFS mounted: 2.93ms, (+0.22ms)

Total time: 2.93ms, (+0.00ms)

Cmdline: /hello 

Hello from C code

Please note that OSv boots in PVH mode (direct kernel mode) so the real-mode boot is bypassed and that is why disk read and uncompress are 0 (just like on firecracker).

The 57ms at the end, that is 20 slower - what does *that* include that the 2.91ms figure didn't?

Some of that will the execution of the application and poweroff. But most of it will be the overhead of the QEMU itself. Even though OSv boots very fast, QEMU process with microvm is still relatively slow to start.

Here is the gist with the same thing with timed strace around it - https://gist.github.com/wkozaczuk/0806809f4003f1c3761ce4ba8c2f634f. Please note that running with strace slows everything down twice.

My sense is that most of the overall 50-60ms are spent in QEMU startup. 

Now if you compare it with firecracker, firecracker starts much faster (based on the printouts I added to firecracker) and only adds 5 ms when it gives control to the guest. But then wastes 30-40 ms shutting down - see this from my firecracker issue comments - https://github.com/firecracker-microvm/firecracker/issues/1164 and https://github.com/firecracker-microvm/firecracker/issues/923#issuecomment-497336403. Somehow from the moment exit_group() is called until the process officially dies it takes 30-45 ms. Weird.

[Waldek Kozaczuk]

The review and discussion around proper poweroff/reboot sequence on microvm/firecracker with Nadav (see other email) inspired me to check another thing - remove '-no-reboot` from QEMU parameter list. This should then trigger reboot rather than poweroff, right? And then constantly re-execute the same OSv image over and over sequentially. And this is what I did and the results are astonishing.

cat test_micro.sh

time /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64 \

-m 64M \

-smp 1 \

--nographic \

-kernel /home/wkozaczuk/projects/osv/scripts/../build/last/loader-stripped.elf \

-append "--nopci /hello" \

-M microvm,x-option-roms=off,pit=off,pic=off,rtc=off \

-nodefaults \

-no-user-config \

-global virtio-mmio.force-legacy=off \

-device virtio-blk-device,id=blk0,drive=hd0,scsi=off \

-drive file=/home/wkozaczuk/projects/osv/build/last/usr.img,if=none,id=hd0,cache=writeback,aio=threads \

-enable-kvm \

-cpu host,+x2apic \

-serial stdio

/test_micro.sh > /tmp/log

qemu-system-x86_64: terminating on signal 2

real 0m10.467s

user 0m6.366s

sys 0m3.328s

grep Hello /tmp/log | wc -l

1772

As you can see OSv re-executed 1772 times in 10.467 seconds. Which gives you:

• 6ms - average execution time including boo time
• 170 - number of executions per second 

So in this scenario, we pay for QEMU overhead once.

Waldek

[Nadav Har'El]

Too bad the Guiness Book of World Records doesn't keep records of which OS can reboot the quickest :-)

• 6ms - average execution time including boo time
• 170 - number of executions per second 

So in this scenario, we pay for QEMU overhead once.

By the way, with such quick executions, the serial port code writing all the output may prove to be a noticable slowdown. I wonder is we can shave another millisecond by spewing less output :-)

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/287b75a3-b556-4ac4-a598-9f04323875fd%40googlegroups.com.

[Waldek Kozaczuk]

Imagine the potential for serverless given OSv supports runscript boot option. If we can replace the runscript file for every reboot, OSv can execute different app with different parameters. Even better reboot and wait until the runscript file changes or some other touch mechanism. But for that we would need type of fs like 9p or new virtio-fs. 

Sent from my iPhone

On Nov 26, 2019, at 09:47, Nadav Har'El <n...@scylladb.com> wrote:



[slop...@redhat.com]

On Friday, November 22, 2019 at 2:34:32 PM UTC+1, Waldek Kozaczuk wrote:

As you may have heard QEMU developers added new machine type: microvm - https://github.com/qemu/qemu/blob/master/docs/microvm.rst. It has been merged into master but not part of any official QEMU release. So if you want to try you need to build QEMU from the source.

Recently I have played with it and eventually managed to boot OSv successfully on it after some adjustments (see some of my recent virtio code tweaks). And the same OSv ROFS image that would boot in ~5 ms on firecracker, can boot in only ~3ms on QEMU microvm.

OSv v0.54.0-34-gccb9cd12

Booted up in 2.91 ms

Cmdline: /hello 

Hello from C code

real 0m0.057s

user 0m0.037s

sys 0m0.020s

I suspect those 57ms can be bought down significantly by using a minimal QEMU build, which saves a noticeable amount of time spent in resolving dynamic symbols.

Something like this should do the trick:

../configure --target-list=x86_64-softmmu --disable-user --disable-linux-user \
             --disable-bsd-user --disable-docs --disable-guest-agent \
             --disable-guest-agent-msi --disable-modules --disable-debug-tcg \
             --disable-debug-info --disable-sparse --disable-gnutls \
             --disable-nettle --disable-gcrypt --disable-auth-pam \
             --disable-sdl --disable-sdl-image --disable-gtk --disable-vte \
             --disable-curses --disable-iconv --disable-vnc \
             --disable-vnc-sasl --disable-vnc-jpeg --disable-vnc-png \
             --disable-cocoa --disable-virtfs --disable-mpath --disable-xen \
             --disable-xen-pci-passthrough --disable-brlapi --disable-curl \
             --disable-fdt --disable-bluez --disable-hax --disable-hvf \
             --disable-whpx --disable-rdma --disable-pvrdma --disable-vde \
             --disable-netmap --disable-linux-aio --disable-cap-ng \
             --disable-attr --disable-vhost-net --disable-vhost-vsock \
             --disable-vhost-scsi --disable-vhost-crypto \
             --disable-vhost-kernel --disable-vhost-user --disable-spice \
             --disable-rbd --disable-libiscsi --disable-libnfs \
             --disable-smartcard --disable-libusb \
             --disable-live-block-migration --disable-usb-redir \
             --disable-lzo --disable-snappy --disable-bzip2 --disable-lzfse \
             --disable-seccomp --disable-coroutine-pool --disable-glusterfs \
             --disable-tpm --disable-libssh --disable-numa --disable-libxml2 \
             --disable-tcmalloc --disable-jemalloc --disable-avx2 \
             --disable-replication --disable-opengl --disable-virglrenderer \
             --disable-xfsctl --disable-qom-cast-debug --disable-tools \
             --disable-vxhs --disable-bochs --disable-cloop --disable-dmg \
             --disable-qcow1 --disable-vdi --disable-vvfat --disable-qed \
             --disable-parallels --disable-sheepdog --disable-crypto-afalg \
             --disable-capstone --disable-debug-mutex --disable-libpmem \
             --disable-xkbcommon

In the near future it'd be also possible to disable other x86 machine types and leave only microvm, but that just reduces the size of the binary and shouldn't have any impact on boot time.

Regards,

Sergio.