Modernizing and cleaning build system

[Waldek Kozaczuk]

Here is a list of things we should try to do:

• Propose and define model app layout (Makefile, GET, ...) under osv-apps
• Typically each app has GET shell script, Makefile and module.py:
• GET typically downloads ad puts source files into upstream directory (sometimes using latest script)
• Makefile typically does what needs to be done (sometime compiles) and puts source files into install subdirectory (sometimes usr.manifest is generated, sometime usr.manifest is pre-defined).
• module.py is called to execute extra logic - sometimes to generate usr.manifest, define cmdline and possibly multiple flavors 
• This is somewhat documented in the README.md of osv-apps project but is not clear enough and the apps are all over the place following this loose convention and I am myself guilty of not following it (see python2x/3x apps that do too much in GET). I doubt it is worth fixing existing apps but I think we could improve the documentation in this area and and possibly point to a good model app.
  
• I know our build system (scripts/build and related python scripts) is complicated enough. But it would be super nice to allow passing parameters from scripts/build to app Makefile. The best use case for that is building java9 (10,11,12 as well) JREs using jlink where it would be nice to pass a list of JRE modules to incorporate - see https://github.com/cloudius-systems/osv-apps/blob/master/openjdk9-zulu-java-base/Makefile#L26; I think there are many other apps that would benefit
  
• Eliminate external - https://github.com/cloudius-systems/osv/issues/743
  
• It would be nice to detail what to do to each subdirectory (I am not positive we want to eliminate all - see x64/acpica)
• Convert to CMakefile 
• I am not sure how difficult it would be, but one benefit of it would be the ability to load a project in IDEs like CLion that are able to work with CMakefile,
• Some apps (like python2x/python3x) pull most of the binaries from host instead of re-compiling from scratch. It would be nice to have a shared app/tool to build apps from host like that.

Feel free to add any other things you would like to be improved in the build system.

Waldek

[Nadav Har'El]

n Sun, Mar 31, 2019 at 5:58 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Here is a list of things we should try to do:

I think there are two very different issues involved here, that don't really need the same solution:

The OSv kernel's makefile, and the "apps" build system.

• Propose and define model app layout (Makefile, GET, ...) under osv-apps
• Typically each app has GET shell script, Makefile and module.py:
• GET typically downloads ad puts source files into upstream directory (sometimes using latest script)
• Makefile typically does what needs to be done (sometime compiles) and puts source files into install subdirectory (sometimes usr.manifest is generated, sometime usr.manifest is pre-defined).
• module.py is called to execute extra logic - sometimes to generate usr.manifest, define cmdline and possibly multiple flavors 
• This is somewhat documented in the README.md of osv-apps project but is not clear enough and the apps are all over the place following this loose convention and I am myself guilty of not following it (see python2x/3x apps that do too much in GET). I doubt it is worth fixing existing apps but I think we could improve the documentation in this area and and possibly point to a good model app.
  

I agree. Also note that the both the "GET" and "makefile" there are rather arbitrary conventions -

all we need from the Makefile is to have the "make" and "make clean" target. That's it. Because people

were too lazy to write a proper makefile around this requirement, "make" just calls "GET" which is a

shell script that just does everything.

We could easily have any other convention to build and clean a package.

I think the more difficult part is understand what the heck is module.py and what is usr.manifest, etc.

I think you're right that just documenting the conventions we used will go a long way in helping people to use this system, even if we don't change anything.

 

• I know our build system (scripts/build and related python scripts) is complicated enough. But it would be super nice to allow passing parameters from scripts/build to app Makefile. The best use case for that is building java9 (10,11,12 as well) JREs using jlink where it would be nice to pass a list of JRE modules to incorporate - see https://github.com/cloudius-systems/osv-apps/blob/master/openjdk9-zulu-java-base/Makefile#L26; I think there are many other apps that would benefit
  

Possibly.  Indeed if properly documented, it shouldn't matter that it's even more complicated.

• Eliminate external - https://github.com/cloudius-systems/osv/issues/743
  
• It would be nice to detail what to do to each subdirectory (I am not positive we want to eliminate all - see x64/acpica)

I agree. We should do that. "external" is an idea that really outlived its usefulness, I think.

Indeed we don't need to remove everything at once, we can do it piece by piece (and stop using pieces before actually removing them).

 

• Convert to CMakefile 
• I am not sure how difficult it would be, but one benefit of it would be the ability to load a project in IDEs like CLion that are able to work with CMakefile,

I would hate for this to happen. I really think OSv's makefile is well tuned and works well.

Switching to cmake just to make an editor happy would be very sad.

I don't know anything about clion, but is it possible to tell it OSv's include paths, etc., *without* switching the Makefile language?

E.g., could we have "make clion" which automatically generates a file CMakeLists.txt which somehow includes the right incantations to tell CLIon what are OSv's include path and predefined macros needed to have the source files properly formatted?

The file doesn't need to contain real instructions how to compile anything, I hope...

 

• Some apps (like python2x/python3x) pull most of the binaries from host instead of re-compiling from scratch. It would be nice to have a shared app/tool to build apps from host like that.

I agree. It would really be nice to have scripts or python code for module.py or something, of how to copy a bunch of files from the host and possibly the shared libraries they need, etc., and then just use this same script from various apps instead of repeating the same code (with variations nobody understands) in a dozen apps.

 

Feel free to add any other things you would like to be improved in the build system.

Waldek

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Geraldo Netto]

Hi!

I might be saying some nonsense, but...

On Sun, 31 Mar 2019 at 17:25, Nadav Har'El <n...@scylladb.com> wrote:

n Sun, Mar 31, 2019 at 5:58 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Here is a list of things we should try to do:

I think there are two very different issues involved here, that don't really need the same solution:

The OSv kernel's makefile, and the "apps" build system.

• Propose and define model app layout (Makefile, GET, ...) under osv-apps
• Typically each app has GET shell script, Makefile and module.py:
• GET typically downloads ad puts source files into upstream directory (sometimes using latest script)
• Makefile typically does what needs to be done (sometime compiles) and puts source files into install subdirectory (sometimes usr.manifest is generated, sometime usr.manifest is pre-defined).
• module.py is called to execute extra logic - sometimes to generate usr.manifest, define cmdline and possibly multiple flavors 
• This is somewhat documented in the README.md of osv-apps project but is not clear enough and the apps are all over the place following this loose convention and I am myself guilty of not following it (see python2x/3x apps that do too much in GET). I doubt it is worth fixing existing apps but I think we could improve the documentation in this area and and possibly point to a good model app.
  

I agree. Also, note that both the "GET" and "makefile" there are rather arbitrary conventions -

all we need from the Makefile is to have the "make" and "make clean" target. That's it. Because people

were too lazy to write a proper makefile around this requirement, "make" just calls "GET" which is a

shell script that just does everything.

We could easily have any other convention to build and clean a package.

In the past, I did start writing a python script to handle OSv apps in a uniform way the code is lost somewhere here

But the idea was to have a JSON file with 2 sections, one for configuration, another for stages

and based on some reserved names, we do operations like downloading files, checksum, etc

pretty much like and dependency/build manager like maven/gradle/pip/conan/...

Of course, any suggestion is welcomed :)

Also, because we are limited in terms of developers, I would suggest leaving documentation until the new app builder is ready, otherwise, we will take a lot of time to write documentation for current build that will be lost and people will certainly ask for help if they are really interested anyway

 

I think the more difficult part is understand what the heck is module.py and what is usr.manifest, etc.

if I understood, the script that handles OSv modules uses module.py to expose OSv apps as  OSv modules, otherwise, we would need to move OSv apps inside <osv-dir>/modules (or change the python script that handles modules in other directories accordingly)

Well, in fact, it might be a future refactoring, moving non-essential modules to OSv apps, for example, I think java, go, openssl, httpserver-*, monitoring-agent could be moved to OSv apps

As for usr.manifest, personally, I would prefer to not have user.manifest and instead have a default directory inside app directory and everything that you puts there will be added to the final osv file system image

I think most of the apps already generates a ROOTFS directory that creates such a structure

not sure

also, we could create tests to build each app automatically and see if they're still working

I guess we have a number of apps that are not working due to various reasons

I think you're right that just documenting the conventions we used will go a long way in helping people to use this system, even if we don't change anything.

 

• I know our build system (scripts/build and related python scripts) is complicated enough. But it would be super nice to allow passing parameters from scripts/build to app Makefile. The best use case for that is building java9 (10,11,12 as well) JREs using jlink where it would be nice to pass a list of JRE modules to incorporate - see https://github.com/cloudius-systems/osv-apps/blob/master/openjdk9-zulu-java-base/Makefile#L26; I think there are many other apps that would benefit
  

Possibly.  Indeed if properly documented, it shouldn't matter that it's even more complicated.

• Eliminate external - https://github.com/cloudius-systems/osv/issues/743
  
• It would be nice to detail what to do to each subdirectory (I am not positive we want to eliminate all - see x64/acpica)

I agree. We should do that. "external" is an idea that really outlived its usefulness, I think.

Indeed we don't need to remove everything at once, we can do it piece by piece (and stop using pieces before actually removing them).

It might also be interesting updating acpica to the latest version

Of course, I understand that talk is cheap (speaking for myself...)

 

 

• Convert to CMakefile 
• I am not sure how difficult it would be, but one benefit of it would be the ability to load a project in IDEs like CLion that are able to work with CMakefile,

I would hate for this to happen. I really think OSv's makefile is well tuned and works well.

Switching to cmake just to make an editor happy would be very sad.

I don't know anything about clion, but is it possible to tell it OSv's include paths, etc., *without* switching the Makefile language?

E.g., could we have "make clion" which automatically generates a file CMakeLists.txt which somehow includes the right incantations to tell CLIon what are OSv's include path and predefined macros needed to have the source files properly formatted?

The file doesn't need to contain real instructions on how to compile anything, I hope...

 

• Some apps (like python2x/python3x) pull most of the binaries from host instead of re-compiling from scratch. It would be nice to have a shared app/tool to build apps from host like that.

I agree. It would really be nice to have scripts or python code for module.py or something, of how to copy a bunch of files from the host and possibly the shared libraries they need, etc., and then just use this same script from various apps instead of repeating the same code (with variations nobody understands) in a dozen apps.

While I'm always in favor of "newer"/"experimental" work, I would also stick to the things that we can help and support by now because once again we have limited resources to develop OSv

 

 

Feel free to add any other things you would like to be improved in the build system.

Waldek

Let me know if I can help with anything :)

 

[Waldek Kozaczuk]

I also think there are two audiences in this context we need to take into account:

• OSv developers - people that build apps just to test stuff; typically experienced with OSv build system and its limitations -> can find their way around
• OSv users - people that build apps to deploy apps to run on OSv - should not need to understand OSv build system to deploy their apps

The suggestions I have listed mostly pertain to first group. And I agree we should just tune what already works and document it.

The second group's need should be addressed by Capstan. So we should avoid duplication between what Capstan does well (and hopefully will do even better in future) and OSv build system. Now capstan packages are often generated using osv-apps and OSv build system.

I must say I love Nadav's idea of having a way to generate CMakefile.txt. I will try to experiment with it.

Waldek

[Rick Payne]

On Sun, 2019-03-31 at 11:54 -0700, Waldek Kozaczuk wrote:
> The second group's need should be addressed by Capstan. So we should
> avoid duplication between what Capstan does well (and hopefully will
> do even better in future) and OSv build system. Now capstan packages
> are often generated using osv-apps and OSv build system.

As I've mentioned before - we're using a plugin I wrote to the erlang
build tool 'rebar3' which takes an erlang/OTP application and an OSv
base image and builts a unikernel image.

The reason for this approach is the slightly complex nature of making
the image. We're emulating what the OSv build system does, using cpio
to push in the components to the file system over a tcp port.

Is that what capstan does too? Or is there a simpler way that I've
missed?

Rick

[Waldek Kozaczuk]

In the next couple of weeks I will be trying to work on some of those issues. I think that eliminating external and fixing "openssl 1.0" issue are probably most pressing ones. I feel like we have gotten to the point where are some of these issues start hindering the development of the project.

Does anybody have suggestion in what sequence we should be proceeding? For example even though archaic openssl causes headache, is it actually necessary to build kernel or run unit tests? Should it be actually required in setup.py?

Also recently I commited manifest_from_host.sh (https://github.com/cloudius-systems/osv/blob/master/scripts/manifest_from_host.sh) script that I think can be helpful in this exercise to eliminate build from scratch in some of the modules by pulling artifacts from the host.

Waldek

[Nadav Har'El]

On Mon, May 20, 2019 at 6:48 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

In the next couple of weeks I will be trying to work on some of those issues. I think that eliminating external and fixing "openssl 1.0" issue are probably most pressing ones. I feel like we have gotten to the point where are some of these issues start hindering the development of the project.

The openssl 1.0 issue is definitely annoying for me, but to be honest, it only hurts the default module, using Lua. If you never use that, you don't even notice it.

The "external" thing is also a nusence, but we already got rid of most of it in the past, we just need to get rid of more, and actually remove some of the "external" options from the Makefile and whole directories from the "external" repository.

Doing this properly for aarch64 cross-compilation will be more of a challenge. I suggested some options in the issue (namely, fetch some libraries as needed from Fedora or similar place - not from OSv's "external" repository).

Does anybody have suggestion in what sequence we should be proceeding? For example even though archaic openssl causes headache, is it actually necessary to build kernel or run unit tests?

No, it is not. It is only needed for the default "scripts/build" without parameters, which builds Lua which needs it.

But it looks bad that our default "scripts/build" doesn't work. We could also fix it by changing the default ;-) But the lua shell is a sensible default.

Should it be actually required in setup.py?

Also recently I commited manifest_from_host.sh (https://github.com/cloudius-systems/osv/blob/master/scripts/manifest_from_host.sh) script that I think can be helpful in this exercise to eliminate build from scratch in some of the modules by pulling artifacts from the host.

Indeed.

We can even start by having in apps/ two versions for some of the packages - one (the existing one) which compiles from source, and another one which just picks up the files already installed on the build machine.

There's also a third option, similar to what we did in openjdk8-fedora:  fetch pre-compiled packages from some common Linux distributions (e.g., Fedora or Ubuntu) and use the files from it for the image. Regardless of what is installed on the build machine. It would be nice to work on common scripts (like you started to manifest_from_host.sh) so new apps can be added to apps/ easily e.g., by just calling a script and a list of Fedora or Ubuntu packages, and the script does everything (fetching those packages, etc.). I opened https://github.com/cloudius-systems/osv/issues/1041

[Waldek Kozaczuk]

On Monday, May 20, 2019 at 12:30:04 PM UTC-4, Nadav Har'El wrote:

On Mon, May 20, 2019 at 6:48 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

In the next couple of weeks I will be trying to work on some of those issues. I think that eliminating external and fixing "openssl 1.0" issue are probably most pressing ones. I feel like we have gotten to the point where are some of these issues start hindering the development of the project.

The openssl 1.0 issue is definitely annoying for me, but to be honest, it only hurts the default module, using Lua. If you never use that, you don't even notice it.

It also hurts building/using anything html or node related modules/apps that use npm, which rejects working with openssl 1. Am I also right that openssl 1 has these awful vulnerabilities discovered ages ago and should not be used? 

The "external" thing is also a nusence, but we already got rid of most of it in the past, we just need to get rid of more, and actually remove some of the "external" options from the Makefile and whole directories from the "external" repository.

It also makes cloning osv repo take much longer that could have been. 

Doing this properly for aarch64 cross-compilation will be more of a challenge. I suggested some options in the issue (namely, fetch some libraries as needed from Fedora or similar place - not from OSv's "external" repository).

There is also a dependency on libnfs in the main makefile which I think should be refactored as a module in a similar fashion I proposed in this smb2/3 patch -  https://groups.google.com/forum/#!searchin/osv-dev/smb2%7Csort:date/osv-dev/QToC22-QHiA/8EwW7uDwBgAJ.

Does anybody have suggestion in what sequence we should be proceeding? For example even though archaic openssl causes headache, is it actually necessary to build kernel or run unit tests?

No, it is not. It is only needed for the default "scripts/build" without parameters, which builds Lua which needs it.

But it looks bad that our default "scripts/build" doesn't work. We could also fix it by changing the default ;-) But the lua shell is a sensible default.

I even have a working version of lua module that uses lua from host and only compiles lua-rocks libraries and depends on modern openssl 1.1. Should be sending patch some time soon. 

Should it be actually required in setup.py?

Also recently I commited manifest_from_host.sh (https://github.com/cloudius-systems/osv/blob/master/scripts/manifest_from_host.sh) script that I think can be helpful in this exercise to eliminate build from scratch in some of the modules by pulling artifacts from the host.

Indeed.

We can even start by having in apps/ two versions for some of the packages - one (the existing one) which compiles from source, and another one which just picks up the files already installed on the build machine.

There's also a third option, similar to what we did in openjdk8-fedora:  fetch pre-compiled packages from some common Linux distributions (e.g., Fedora or Ubuntu) and use the files from it for the image. Regardless of what is installed on the build machine. It would be nice to work on common scripts (like you started to manifest_from_host.sh) so new apps can be added to apps/ easily e.g., by just calling a script and a list of Fedora or Ubuntu packages, and the script does everything (fetching those packages, etc.). I opened https://github.com/cloudius-systems/osv/issues/1041

Thanks for that.