Roadmap going forward

[Waldek Kozaczuk]

Hi,

It has been a while since the last release but I have only recently found some time and energy to update the roadmap - https://github.com/cloudius-systems/osv/wiki/Roadmap. It is meant to be a "living" document that more-less captures what we (me?) would like to work on in near future. So it is more-less my wish list but I would like to use this occasion to seek your input as well. I would like to know what you would want to be implemented/improved/changed etc. Feel free to respond to this email or directly update the roadmap wiki or add comments. Also, all OSv issues are here - https://github.com/cloudius-systems/osv/issues and if you find any interesting ones please feel free to add them to the wiki.

In order to facilitate the discussion I am enclosing the roadmap here with some of my extra comments:

Modularization/"Librarization"

• Hide most symbols and C++ standard library (Partially addressed by this and other commits)
  • Issue 97 - Be more selective on symbols exported from the kernel
  • The research results with numbers described in this discussion and the older one seem to indicate that both C++ standard library and other non-public symbols should be hidden in the kernel to make this goal achievable
• Make ZFS optional as a shared library - the patches to address this issue were sent a couple of days ago
• Create toolchain to optionally build custom kernel tailored to specific hypervisor or app
  • See this original discussion
• Reduce kernel size
  • Shrinking kernel described in this early discussion would really be a result of hiding C++ library and most symbols (see above) but also conditionally compiling in/out various modules (see the first bullet in the Modularization/"Librarization")
• Expose C++ APIs as C to hide C++ completely

Support statically linked executables

• See https://github.com/cloudius-systems/osv/issues?q=is%3Aissue+is%3Aopen+label%3Astatic-elf

Improve tooling

• Improve capstan (see Capstan "2.0" Wiki)
  • THIS is a project on its own, somewhat independent but I think it is very important to improve building and running OSv images; it would be nice to fine a volunteer that would focus on it entirely
• Create IntelliJ, Maven and or Gradle plugins to deploy Java, Node, etc apps
• Run Docker images on OSv (possibly musl-based like Alpine are better)
  • There is already an existing example - https://github.com/cloudius-systems/osv-apps/tree/master/openjdk12-jre-from-docker

Merge parts of ipv6 branch into master

• Port partial netlink support in order to fix libc: Fix if_indextoname(), if_nametoindex()
• Also, reach out to Spirent to see if any more networking related bugs fixed can be brought over from the https://github.com/SpirentOrion/osv fork

Others

• Help review and merge the "lazy-allocated stack" patch
• Fix cli/lua on newer Fedora (see https://github.com/cloudius-systems/osv/issues/1166)
• Consider implementing a read-write filesystem alternative to ZFS - ext2/3/4 ?
• Improve gdb support
• #1069 - Add vsock support
  • Would there really be real-world application of that? Maybe to provide better OSv-host configuration capabilities?
• Support other booting methods like UEFI
• Support other hypervisors
  • Support new KVM-bases AWS instances (see https://github.com/cloudius-systems/osv/issues/924)
  • Support Hyper-V (see https://github.com/cloudius-systems/osv/issues/735)
  • What about XEN? Is it still "a thing" in terms of supporting it? I have very little experience with XEN and I sense that XEN has evolved a lot in the last 6-7 years.
• Support .NET Core executables
  • See what needs to be finished in this thread
• Refresh look and feel of http://osv.io
• RISC-V port

Various finer-level improvements

• Optimize memory allocator to better deal with "small" amount of memory - https://github.com/cloudius-systems/osv/issues/1013
• #28 - Optimize atomic operations on UP (single-VCPU)
• #1016 - Add support for smb2/3 (samba) file system T
  • There is already a patch sent to the list
• #1041 - Idea for easily creating OSv images from a list of Ubuntu packages
• Fix networking bugs
  • https://github.com/cloudius-systems/osv/issues/420
  • https://github.com/cloudius-systems/osv/issues/454
• Speedup VFS - https://github.com/cloudius-systems/osv/issues/450
• Improve XEN support
  • https://github.com/cloudius-systems/osv/issues/498
  • https://github.com/cloudius-systems/osv/issues/345
• Speed-up realloc() by mremap() - https://github.com/cloudius-systems/osv/issues/184
• Investigate virtio-net “slow path” - https://github.com/cloudius-systems/osv/issues/177

Improve AArch64 support

• Work on aarch64 issues
• Fix various bugs/limitations
• Implement gaps in respect to x86_64

Finally, I also hope to find some new volunteers to help me work on some of those improvements ;-)

Regards,

Waldek

[Nadav Har'El]

On Mon, Dec 20, 2021 at 4:23 AM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Hi,

It has been a while since the last release but I have only recently found some time and energy to update the roadmap - https://github.com/cloudius-systems/osv/wiki/Roadmap. It is meant to be a "living" document that more-less captures what we (me?) would like to work on in near future. So it is more-less my wish list but I would like to use this occasion to seek your input as well. I would like to know what you would want to be implemented/improved/changed etc. Feel free to respond to this email or directly update the roadmap wiki or add comments. Also, all OSv issues are here - https://github.com/cloudius-systems/osv/issues and if you find any interesting ones please feel free to add them to the wiki.

In order to facilitate the discussion I am enclosing the roadmap here with some of my extra comments:

Modularization/"Librarization"

• Hide most symbols and C++ standard library (Partially addressed by this and other commits)
  • Issue 97 - Be more selective on symbols exported from the kernel
  • The research results with numbers described in this discussion and the older one seem to indicate that both C++ standard library and other non-public symbols should be hidden in the kernel to make this goal achievable
• Make ZFS optional as a shared library - the patches to address this issue were sent a couple of days ago
• Create toolchain to optionally build custom kernel tailored to specific hypervisor or app
  • See this original discussion
• Reduce kernel size
  • Shrinking kernel described in this early discussion would really be a result of hiding C++ library and most symbols (see above) but also conditionally compiling in/out various modules (see the first bullet in the Modularization/"Librarization")
• Expose C++ APIs as C to hide C++ completely

You've made some really nice progress on this!

Support statically linked executables

• See https://github.com/cloudius-systems/osv/issues?q=is%3Aissue+is%3Aopen+label%3Astatic-elf

What's the motivation for this? What uses statically linked executables?

Improve tooling

• Improve capstan (see Capstan "2.0" Wiki)
  • THIS is a project on its own, somewhat independent but I think it is very important to improve building and running OSv images; it would be nice to fine a volunteer that would focus on it entirely
• Create IntelliJ, Maven and or Gradle plugins to deploy Java, Node, etc apps
• Run Docker images on OSv (possibly musl-based like Alpine are better)
  • There is already an existing example - https://github.com/cloudius-systems/osv-apps/tree/master/openjdk12-jre-from-docker

Merge parts of ipv6 branch into master

• Port partial netlink support in order to fix libc: Fix if_indextoname(), if_nametoindex()
• Also, reach out to Spirent to see if any more networking related bugs fixed can be brought over from the https://github.com/SpirentOrion/osv fork

Others

• Help review and merge the "lazy-allocated stack" patch
• Fix cli/lua on newer Fedora (see https://github.com/cloudius-systems/osv/issues/1166)
• Consider implementing a read-write filesystem alternative to ZFS - ext2/3/4 ?
• Improve gdb support

What's the motivation of this? I thought the existing gdb support was pretty good, what's missing?

The issue 1167 doesn't describe the motivation.

Are you trying to gain an ability to debug individual threads (not CPU cores) running on OSv (not the OSv kernel)?

Are you trying to connect gdb to a remote OSv virtual machine (not running on the same machine)?

Or something else?

 

• #1069 - Add vsock support
  • Would there really be real-world application of that? Maybe to provide better OSv-host configuration capabilities?
• Support other booting methods like UEFI
• Support other hypervisors
  • Support new KVM-bases AWS instances (see https://github.com/cloudius-systems/osv/issues/924)
  • Support Hyper-V (see https://github.com/cloudius-systems/osv/issues/735)
  • What about XEN? Is it still "a thing" in terms of supporting it? I have very little experience with XEN and I sense that XEN has evolved a lot in the last 6-7 years.

This will be very hard without a personal "itch" to see OSv work on those hypervisors, which you actually use.

Personally, I don't even know how to install Xen to test anything on it :-(

 

• Support .NET Core executables
  • See what needs to be finished in this thread
• Refresh look and feel of http://osv.io
• RISC-V port

Various finer-level improvements

• Optimize memory allocator to better deal with "small" amount of memory - https://github.com/cloudius-systems/osv/issues/1013
• #28 - Optimize atomic operations on UP (single-VCPU)
• #1016 - Add support for smb2/3 (samba) file system T
  • There is already a patch sent to the list
• #1041 - Idea for easily creating OSv images from a list of Ubuntu packages
• Fix networking bugs
  • https://github.com/cloudius-systems/osv/issues/420
  • https://github.com/cloudius-systems/osv/issues/454
• Speedup VFS - https://github.com/cloudius-systems/osv/issues/450
• Improve XEN support
  • https://github.com/cloudius-systems/osv/issues/498
  • https://github.com/cloudius-systems/osv/issues/345
• Speed-up realloc() by mremap() - https://github.com/cloudius-systems/osv/issues/184
• Investigate virtio-net “slow path” - https://github.com/cloudius-systems/osv/issues/177

Improve AArch64 support

• Work on aarch64 issues
• Fix various bugs/limitations
• Implement gaps in respect to x86_64

Finally, I also hope to find some new volunteers to help me work on some of those improvements ;-)

That would be great. Good luck!

Regards,

Waldek

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/b302c81c-d81a-48f3-b025-20b40d6cd356n%40googlegroups.com.

[Waldek Kozaczuk]

Hi,

On Mon, Dec 20, 2021 at 03:15 Nadav Har'El <n...@scylladb.com> wrote:

On Mon, Dec 20, 2021 at 4:23 AM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Hi,

It has been a while since the last release but I have only recently found some time and energy to update the roadmap - https://github.com/cloudius-systems/osv/wiki/Roadmap. It is meant to be a "living" document that more-less captures what we (me?) would like to work on in near future. So it is more-less my wish list but I would like to use this occasion to seek your input as well. I would like to know what you would want to be implemented/improved/changed etc. Feel free to respond to this email or directly update the roadmap wiki or add comments. Also, all OSv issues are here - https://github.com/cloudius-systems/osv/issues and if you find any interesting ones please feel free to add them to the wiki.

In order to facilitate the discussion I am enclosing the roadmap here with some of my extra comments:

Modularization/"Librarization"

• Hide most symbols and C++ standard library (Partially addressed by this and other commits)
  • Issue 97 - Be more selective on symbols exported from the kernel
  • The research results with numbers described in this discussion and the older one seem to indicate that both C++ standard library and other non-public symbols should be hidden in the kernel to make this goal achievable
• Make ZFS optional as a shared library - the patches to address this issue were sent a couple of days ago
• Create toolchain to optionally build custom kernel tailored to specific hypervisor or app
  • See this original discussion
• Reduce kernel size
  • Shrinking kernel described in this early discussion would really be a result of hiding C++ library and most symbols (see above) but also conditionally compiling in/out various modules (see the first bullet in the Modularization/"Librarization")
• Expose C++ APIs as C to hide C++ completely

You've made some really nice progress on this!

Support statically linked executables

• See https://github.com/cloudius-systems/osv/issues?q=is%3Aissue+is%3Aopen+label%3Astatic-elf

What's the motivation for this? What uses statically linked executables?

For example ffmpeg is distributed as static executable. Also the motivation is to limit number of symbols we keep adding for new kinds of apps. The performance hit might not be that bad. 

Improve tooling

• Improve capstan (see Capstan "2.0" Wiki)
  • THIS is a project on its own, somewhat independent but I think it is very important to improve building and running OSv images; it would be nice to fine a volunteer that would focus on it entirely
• Create IntelliJ, Maven and or Gradle plugins to deploy Java, Node, etc apps
• Run Docker images on OSv (possibly musl-based like Alpine are better)
  • There is already an existing example - https://github.com/cloudius-systems/osv-apps/tree/master/openjdk12-jre-from-docker

Merge parts of ipv6 branch into master

• Port partial netlink support in order to fix libc: Fix if_indextoname(), if_nametoindex()
• Also, reach out to Spirent to see if any more networking related bugs fixed can be brought over from the https://github.com/SpirentOrion/osv fork

Others

• Help review and merge the "lazy-allocated stack" patch
• Fix cli/lua on newer Fedora (see https://github.com/cloudius-systems/osv/issues/1166)
• Consider implementing a read-write filesystem alternative to ZFS - ext2/3/4 ?
• Improve gdb support

What's the motivation of this? I thought the existing gdb support was pretty good, what's missing?

The issue 1167 doesn't describe the motivation.

Are you trying to gain an ability to debug individual threads (not CPU cores) running on OSv (not the OSv kernel)?

Are you trying to connect gdb to a remote OSv virtual machine (not running on the same machine)?

Or something else?

The main motivation is to be able to debug on other hypervisors than QEMU which we support well.