Experimental KVM microvm implementation

[Rudolph Bott]

Hey Everyone,

I started an experiment and created a (very hacky) implementation of the "microvm" machine type of Qemu. You can see/try its current state here: https://github.com/rbott/ganeti/tree/microvm

- What is "microvm"?

- Why another Ganeti hypervisor type?

- Current state

- Whats next? [hint: I need your feedback!]

What is "microvm"?

Qemu/KVM set out to implement with a lightweight alternative to the "fat" qemu/KVM that we all know. Possibly inspired by simpler alternatives like Firecracker it will drastically reduce complexity and init times, also reducing possible attack surface. Qemu/KVM decided to implement this in two steps:

- a new machine type (-M microvm)

- a stripped down binary (e.g. qemu-system-x86_64-microvm, which is only linked against a fraction of the usual libraries)

What is missing compared to regular KVM guests (most prominent changes):

- PCI(e)

- no VGA, no VNC, Spice etc. (only serial)

- no hotplugging/hotremoving

- no live migration

- no device emulation (e.g. Intel NICs, specific SCSI controllers)

What is available?

- VirtIO Devices (e.g. serial, network, disk) on an ISA bus

- ACPI (although most docs will tell you its not there)

Why another Ganeti hypervisor type?

As you can guess from the above, many Qemu/KVM features the existing Ganeti KVM implementation relies on are not available with "microvm". I decided to start off with a new Ganeti hypervisor implementation to focus on the actual proof-of-concept implementation and not get caught up in the existing code base, sprinkling conditionals everywhere to get it to do what I want.

Current State

The Github repository/branch mentioned above contains a working implementation that has been tested on Debian Bookworm with a single Ganeti node (using "plain" and "file" disk templates). I also configured ganeti-instance-debootstrap to install Ubuntu Noble guests. The following is the output from the instance creation (with ganeti-instance-debootstrap cache in use) along with the ready-message from inside the instance:

gnt-instance add -t plain --disk=0:size=2.5g -B minmem=1G,maxmem=1G,vcpus=2 -n debian-kvm -o debootstrap+noble  kvm-test-instance2

Sun Mar  9 22:26:28 2025 * disk 0, size 2.5G
Sun Mar  9 22:26:28 2025 * creating instance disks...
Sun Mar  9 22:26:28 2025 adding instance kvm-test-instance2 to cluster config
Sun Mar  9 22:26:28 2025 adding disks to cluster config
Sun Mar  9 22:26:28 2025  - INFO: Waiting for instance kvm-test-instance2 to sync disks
Sun Mar  9 22:26:28 2025  - INFO: Instance kvm-test-instance2's disks are in sync
Sun Mar  9 22:26:28 2025  - INFO: Waiting for instance kvm-test-instance2 to sync disks
Sun Mar  9 22:26:28 2025  - INFO: Instance kvm-test-instance2's disks are in sync
Sun Mar  9 22:26:28 2025 * running the instance OS create scripts...
Sun Mar  9 22:26:36 2025 * starting instance...

Mar 09 21:26:43 kvm-test-instance2 systemd[1]: Startup finished in 5.213s (kernel) + 674ms (userspace) = 5.888s.

I have also attached the dmesg output, if anyone is interested. Following is the associated command line:

/usr/bin/kvm -name kvm-test-instance2

 -pidfile /var/run/ganeti/kvmmicro-hypervisor/pid/kvm-test-instance2

 -cpu host -m 1024 -smp 2

 -device virtio-serial-device -device virtconsole,chardev=char0,name=console0

 -chardev socket,path=/var/run/ganeti/kvmmicro-hypervisor/ctrl/kvm-test-instance2.serial,server=on,wait=off,id=char0

 -M microvm,x-option-roms=off,pit=off,pic=off,rtc=off,acpi=on,isa-serial=off

 -kernel /root/vmlinuz-6.8.0-55-generic

 -initrd /root/initrd.img-6.8.0-55-generic -append earlyprintk=hvc0 console=hvc0 root=/dev/vda1

 -netdev type=tap,id=net-ffdc5eeb-a2f6-414a,fd=10

 -device virtio-net-device,netdev=net-ffdc5eeb-a2f6-414a,mac=aa:00:00:c7:87:b0

 -blockdev driver=raw,node-name=disk-1dd3a452-b158-4bcf,file.driver=host_device,file.filename=/var/run/ganeti/instance-disks/kvm-test-instance2:0

 -device virtio-blk-device,id=disk-1dd3a452-b158-4bcf,drive=disk-1dd3a452-b158-4bcf

 -daemonize -nodefaults -no-user-config -nographic

 -D /var/log/ganeti/kvm/kvm-test-instance2.log

 -qmp unix:/var/run/ganeti/kvmmicro-hypervisor/ctrl/kvm-test-instance2.qmp,server,nowait

Of course there are still many knobs to tweak/test (disk i/o related settings, machine parameters) but you get the general picture. I have only tested direct kernel boot so far, but microvm also supports SeaBIOS, so a "full" boot should also be possible.

I have so far only tested this with nested virtualisation. No tests directly on hardware have been made so I cannot yet make any comparisons of disk/network performance compared to regular KVM guests.

Also the code currently is very hackish and many parts have been shamelessly copied from the hv_kvm implementation. This would need some serious rethinking / design document(s) if it should get implemented.

Whats next?

Does anyone here have any usecases for Ganeti to support microvm? If you can do without live migration and hotplugging, microvm should provide you with a very lightweight environment with faster boot times (although I do not think that boot times are a specific issue to most Ganeti users). If nobody has any use for it, I guess there is no use to pursue this path any further.

--

 Rudolph Bott - bo...@sipgate.de

 Telefon: +49 (0)211-63 55 55-55

 Telefax: +49 (0)211-63 55 55-22

 sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf

 HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois

 Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391

 www.sipgate.de - www.sipgate.co.uk