Running apps from Linux host in disk-less mode with virtio-fs

[Waldek Kozaczuk]

Traditionally, in order to run an app on OSv one would have to build a ZFS or ROFS image with a kernel in it and have it exposed as virtio-blk or similar block device. With virtio-fs, new host filesystem sharing virtualization mechanism (https://fosdem.org/2020/schedule/event/vai_virtio_fs/attachments/slides/3666/export/events/attachments/vai_virtio_fs/slides/3666/virtio_fs_A_Shared_File_System_for_Virtual_Machines_FOSDEM.pdf), it is possible to run an arbitrary app from Linux host directly using pure OSv kernel (please note kernel.elf artifact below which is in essence loader.elf with empty bootfs):

#In another terminal

./build/virtiofsd --socket-path=/tmp/vhostqemu -o source=/home/wkozaczuk/projects/osv/apps/native-example -o cache=always

/home/wkozaczuk/projects/qemu/build/x86_64-softmmu/qemu-system-x86_64 \

-m 4G \

-smp 4 \

-vnc :1 \

-gdb tcp::1234,server,nowait \

-kernel /home/wkozaczuk/projects/osv/build/last/kernel.elf \

-append "/virtiofs/hello" \

-netdev user,id=un0,net=192.168.122.0/24,host=192.168.122.1 \

-device virtio-net-pci,netdev=un0 \

-device virtio-rng-pci \

-enable-kvm \

-cpu host,+x2apic \

-chardev stdio,mux=on,id=stdio,signal=off \

-mon chardev=stdio,mode=readline \

-device isa-serial,chardev=stdio \

-chardev socket,id=char0,path=/tmp/vhostqemu \

-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \

-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on -numa node,memdev=mem

OSv v0.54.0-179-g2f92fc91

Solaris: NOTICE: Cannot find the pool label for '/dev/vblk0.1'

eth0: 192.168.122.15

Booted up in 103.70 ms

Cmdline: /virtiofs/hello

WARNING: application::prepare_argv(): missing libvdso.so -> may prevent shared libraries specifically Golang ones from functioning

Hello from C code

Because bootfs is empty, I had to apply the following hack-patch to explicitly mount virtio-fs filesystem:

diff --git a/fs/vfs/main.cc b/fs/vfs/main.cc

index 46dcb62f..6df78d34 100644

--- a/fs/vfs/main.cc

+++ b/fs/vfs/main.cc

@@ -2322,6 +2322,13 @@ void pivot_rootfs(const char* path)

         closedir(fs_lib_dir);

     }

 

+    // /dev/virtiofs1, /virtiofs, virtiofs

+    mkdir("/virtiofs", 0666);

+    ret = sys_mount("/dev/virtiofs0", "/virtiofs", "virtiofs", MNT_RDONLY, nullptr);

+    if (ret) {

+        printf("failed to virtiofs mount, error = %s\n", strerror(ret));

+    }

+

     auto ent = setmntent("/etc/fstab", "r");

     if (!ent) {

         return;

In the long run, we could add a new kernel parameter letting one pass this information as a boot parameter. 

Why am I writing about it and why I find virtio-fs so exciting? 

0) No need to create images: expose an arbitrary app directory on Linux host filesystem and run it on OSv. 

1) No duplication of application files if running multiple OSv instances on a host.  

2) More memory efficient as eventually with DAX (physical pages from the host could be directly mapped to virtual memory on OSv) almost no memory copying. 

In this mode running a Linux app on OSv might be better described as a "highly isolated" process than traditional unikernel. What do you think? 

Waldek