adding scripting capabilities to ninja (perhaps with Lua or maybe Guile)

[Basile Starynkevitch]

Hello All,

Is there some ninja derivative (or fork) with internal scripting capabilities? I did not found any.

I feel that the obligation to generate the build.ninja before running ninja (and only permit scripting at that time) is a too strong constraint.

I was thinking of adding internal Lua scripting (or perhaps Guile scripting) inside ninja. I guess that Lua is preferable to Guile because it is smaller, probably slightly faster, and has a more permissive license (technically & as a free software enthusiast I would prefer Guile, but I guess that Lua is more acceptable).

I confess that my use case of ninja is currently not for building monster software like Chromium (but my small project bismon, FWIW). So I have no practical experience with using ninja on monster software bases.

My current thinking might be that ninja could be enhanced by internal scripting (much like GNU make is Guile extensible) and probably still be as fast as it is with the following approach and principles.

ninja builds not using internal scripts (i.e. all the current ones) should not slow down.

Internally we create a single lua_State at start of ninja (let's suppose it is a lua_State* ninja_luastate; global variable in C++). We might require a ninja program argument like --lua to enable this.

In addition to ninja six declaration kinds (rule, build, variable declaration, default, subninja or include, pool) we add the following top-level syntax (where variable expansion is understood in the ninja sense of expanding $var in ninja files....):

• luaload scriptname where scriptname is some lua file. The scriptname could be variable-expanded, so we could write luaload script.lua or luaload $var. When ninja parses such a declaration, it expands the scriptname then luaL_dofile it
• scripted variable declaration, with a syntax like variable := definition then the definition is variable-expanded, that expansion (as some string) is passed to luaL_dostring, we pop a Lua string result using lua_tostring then bind it to variable (in the ninja sense).
• ninja rules could have not only a command but instead a luacommand (but of course not both, however either one is required). The luacommand would variable-expand the rest of the line, and pass it to luaL_string (instead of running some external process).

I tend to think it might be enough. Of course we need to add a small set of Lua primitives to query ninja state.

Comments are welcome. If you feel it is a bad idea please tell.

(I could work on that, but IMHO it makes no sense to work on improving any build automation system if that is not acceptable to other users).

Cheers.

--

Basile Starynkevitch (Bourg La Reine, France)

http://starynkevitch.net/Basile

<bas...@starynkevitch.net>

[Evan Martin]

Thanks for taking the time to spell out your idea.

I think one of the strengths of ninja is that we clearly defined some things we *don't* want to do.  Our manual has a list of non-goals for example.  And in particular we have a discussion on whether using ninja is appropriate for your project, and also recommending other systems that might meet your needs:

  https://ninja-build.org/manual.html#_using_ninja_for_your_project

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evan Martin]

However, looking back over your idea, making an external program in lua that generates .ninja files would be exactly in line with ninja.

It appears someone contributed some lua code along these lines:

https://github.com/ninja-build/ninja/wiki/List-of-generators-producing-ninja-build-files#utilities

[Basile Starynkevitch]

On 08/11/2018 02:21 AM, Evan Martin wrote:

However, looking back over your idea, making an external program in lua that generates .ninja files would be exactly in line with ninja.

However, that illustrates a fundamental weakness of Ninja that is never documented. There is no way (and most other build automation tools have some way for that) to use ninja (without regenerating the build.ninja file) on files which are not statically expected (and explicitly mentioned) in that build.ninja
and I have such a use case in my bismon program; it generates plugins (I call them modules) at runtime, and runs a build command for them.
a generated plugin has a source file name (such as  modules/modbm_9oXtCgAbkqv_4y1xhhF5Nhz.c in bismon) which cannot be statically predicted. So the build system don't know statically that module name when it is built (it only knows that the collection of modules is the set of modules/modbm_*.c files). I solved that in bismon by using ninja to build the main program which would use make to build modules or plugins (and this solution is not elegant).

Notice that Ninja is presented as generating the build.ninja at configure time (that would be the equivalent of running configure scripts in GNU projects using autoconf + make technology) which is expected to be slow and is run rarely and later running ninja which should run quickly (and would be very often used). With that dichotomy in mind I would still expect Ninja to offer optionally more dynamic aspects. My lua scripting proposal (in the previous emails) deals with that (and probably would still keep ninja fast, with the philosophy "don't pay for dynamic features you don't use").

I also noticed that lua scripting was mentioned previously on that list, e.g. in https://groups.google.com/d/msg/ninja-build/gs2eMFdEnKc/wHIlzv-pAQAJ

Regards.

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France