MicroVM profile, making ZFS optional or how to shrink kernel down to almost 3MB

[Waldek Kozaczuk]

Recently I have been experimenting more with trying to make the kernel even smaller than 4.3M which I reported in my previous email - https://groups.google.com/g/osv-dev/c/JsJfnhTX_EY/m/yiVXFlWEAgAJ. My previous experiments to shrink OSv kernel to 4.3M involved mainly these three things:

• linking the standard C++ library in '--no-whole-archive' mode
• hiding most symbols except the ones from standard C libraries (like libc.so.6) that OSv actually implements and exposes per its dynamic linker (I hope to arrive with the exact recipe how to do that properly soon) 
• compiling all sources with flags '-ffunction-sections -fdata-sections' and linking with '--gc-sections' flags (for details please see the email I am referring above) to garbage collect all unneeded code and data

The last 2 elements are actually critical, I think, to help me further reduce kernel size as I am describing below.

Firstly, as I was suggesting in the "Librarization/Modularization" conversation (see https://groups.google.com/g/osv-dev/c/BHwN6Stm3n4/m/arvdOwl3AAAJ), I modified makefile and few source files to remove all driver code but the one needed to run OSv on firecracker and QEMU microvm machine, or in other words, strip down to a "microVM" profile, in order to see if is even worth the effort. More specifically I kept only virtio-blk, virtio-net, kvmclock, serial console, and dropped most of the other drivers - all PCI related code, other drivers than those mentioned before, XEN code, libstm and acpica (this by itself reduced size by 200K). This exercise reduced the size of kernel.elf from 4.27M (4482128 bytes) to 3.78M (3961872 bytes) which means we shaved of around 0.5M.

Secondly, I also removed ZFS code (mostly commented out relevant entries in the makefile). Ideally, we would want to move all ZFS code from kernel to a shared library that could be dynamically loaded if added to the image just like we did it to the NFS code. We already have a related issue - https://github.com/cloudius-systems/osv/issues/1009. Anyway eliminating ZFS code shrinks kernel.elf down to 3.02M (3167856 bytes) which means ZFS code is worth around 0.8M.   

Please note that moving ZFS out as an optional shared library would make both generic and microVM kernel smaller by 0.8M so generic kernel.elf might around 3.5M and microVM one around 3MB.

For what it's worth I was able to run quite many apps including java example. For now, I obviously built the images with ROFS and ran them on firecracker and QEMU microvm machine.

Regards,

Waldek