Compile and optimize application and OSv together as one unit?

[Joe Duarte]

Hi all – Is there a way to compile OSv with whatever application you want to run in it, all together in one optimized build? It seems like there would be Link Time Optimization (LTO) opportunities there.

There was a project out there that basically did this with LLVM, but I can't find it. It would allow you compile an OS (probably Linux) and application together, and would LTO the LLVM bytecode or something, crossing the OS and app boundaries.

There would also be Profile Guided Optimization (PGO) opportunities, but I'm not sure if that would be any different from just profiling OSv and an application separately.

With a uni-build, there could also be interesting security wins, like tree-shake out any system calls you weren't going to use. It would be like Linux's seccomp filters, but better because the filtered system calls would no longer exist at all. Same with unneeded drivers.

Anyway, just some Thursday thoughts.

Cheers,

Joe

[Waldek Kozaczuk]

Hi,

On Thursday, May 7, 2020 at 8:00:40 AM UTC-4, Joe Duarte wrote:

Hi all – Is there a way to compile OSv with whatever application you want to run in it, all together in one optimized build? It seems like there would be Link Time Optimization (LTO) opportunities there.

There was a project out there that basically did this with LLVM, but I can't find it. It would allow you compile an OS (probably Linux) and application together, and would LTO the LLVM bytecode or something, crossing the OS and app boundaries.

Could it be UKL - (Linux Unikernel) - as described in this paper by Ali Raza and Ulrich Drepper - https://www.cs.bu.edu/~jappavoo/Resources/Papers/unikernel-hotos19.pdf? 

There would also be Profile Guided Optimization (PGO) opportunities, but I'm not sure if that would be any different from just profiling OSv and an application separately.

With a uni-build, there could also be interesting security wins, like tree-shake out any system calls you weren't going to use. It would be like Linux's seccomp filters, but better because the filtered system calls would no longer exist at all. Same with unneeded drivers.

An application binary can be put on RAMFS that is part of OSv kernel ELF even now. But I am guessing this would not achieve what you have in mind, would it? Can you elaborate what exactly you envision as "uni-build"?

Anyway, just some Thursday thoughts.

Cheers,

Joe

So is your main motivation improved security or memory optimizations coming from smaller kernel binary? Or are you advocating for making OSv "real" unikernel - "only what's needed and nothing more" ;-) ? 

In general, I would like to work on optimizing OSv kernel size in the next release. I think the 1st related issue that I would tackle is hiding all unnecessary symbols (in essence exporting only the symbols from these libraries -https://github.com/cloudius-systems/osv#kernel-size) and is described here - https://github.com/cloudius-systems/osv/issues/97.

Here is a list of other things I would like to consider:

1. Add a conditional compiler/linker mechanism to the build process allowing creating smaller/more custom tailored kernel. As you point out, one one of the dimension could be devices/drivers or type of hypervisor OSv would run on. We would still produce a default loader.elf but allows creating smaller one targeting specific hypervisor - for example firecracker only needs virtio-net and virtio-blk, no need for all PCI code, other drivers, VGA, acpica, XEN, etc. Another dimensions - is whether one needs ZFS, exported full C++ library (not needed for any non C++ apps), etc. So concluding we can devise a custom-kernel-build process that could target these types of subsets of functionality and even automatically build them and publish for more common cases as part of the Travis CI/CD pipeline.
2. Make more "things" a library. Just like we did it with NFS (see this commit - https://github.com/cloudius-systems/osv/commit/4ffb0fa9329849cd587d62f91b6979bc0e0ce6d1), we could do the same with other filesystems whose code size is bigger - like ZFS or even virtiofs. Also what about drivers - these might be libraries as well loaded by OSv dynamic linker. This might be harder to achieve but not impossible. This in the end would allows to have even smaller "core" kernel and everything else (like "lego blocks") that is part of it now, would be loaded by dynamic linker. So further dynamic modularization if that makes sense.
3. More sophisticated build process (like probably what you want) where individual application would be analyzed and ONLY what is needed would be pulled from loader.elf. If we ever implement static ELF support (see https://github.com/cloudius-systems/osv/issues/212) we might allow creating kernel without libc. But there are limitations - what about apps that use dlopen/dlsym to dynamically invoke any code from kernel? I think that Hermitux is the one that does something like this - https://ssrg-vt.github.io/hermitux/

Regards,

Waldek

[Nadav Har'El]

On Thu, May 7, 2020 at 3:00 PM Joe Duarte <songof...@gmail.com> wrote:

Hi all – Is there a way to compile OSv with whatever application you want to run in it, all together in one optimized build? It seems like there would be Link Time Optimization (LTO) opportunities there.

There is no reason why this should not be possible. After all, the OSv itself *is* one big statically-linked executable, and there is no reason why the application code can't be included into it as well. We *prefer* the application to be a shared object so it can be compiled separately, but there is no reason why we can't compile it in.

But someone will need to try what kind of wins you may get from this (in space, run speed, etc.). I am afraid that in most non-trivial applications the savings would be minimal, but can definitely be a fun project to play with and see what the results are, maybe we'll be pleasantly surprised.

 

There was a project out there that basically did this with LLVM, but I can't find it. It would allow you compile an OS (probably Linux) and application together, and would LTO the LLVM bytecode or something, crossing the OS and app boundaries.

There would also be Profile Guided Optimization (PGO) opportunities, but I'm not sure if that would be any different from just profiling OSv and an application separately.

With a uni-build, there could also be interesting security wins, like tree-shake out any system calls you weren't going to use. It would be like Linux's seccomp filters, but better because the filtered system calls would no longer exist at all. Same with unneeded drivers.

True, but will be especially useful for small hand-written C applications. Applications which use large libraries, runtime environments, etc., will probably have a hard time optimizing anything out, because almost anything "can be used" depending on various options, configurations, etc. For example, Java - the runtime environment - uses almost every system call imaginable. The linker wouldn't know what small class file you'll end up running, and that it only needs 10% of these system calls.

Anyway, just some Thursday thoughts.

Cheers,

Joe

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/0a44ba9d-1b2a-4b6e-ad3e-00dbc8aa3b1a%40googlegroups.com.