Bootstrapping and regenerating ninja files

[Allan Odgaard]

I generate my ninja build files and can’t figure out how to make it so that a) no special action is required for bootstrapping the build on a clean system and b) the involved ninja files are regenerated when required.

Say I have ‘./gen_build’ (executable) which takes a manifest file describing the build (with lines like “sources = src/*.cc”) and outputs a) the build.ninja file that builds what is described in the manifest file and b) a dependency file (.d) which lists all the paths that should cause ‘build.ninja’ to be regenerated (e.g. the manifest file itself, the directories from which sources have been found via file match patterns, etc.).

With ‘make’ I would create a Makefile in the source directory containing these lines:

builddir = /tmp/build

-include $(builddir)/build.ninja.d
-include $(builddir)/build.ninja

$(builddir)/build.ninja: manifest | ./gen_build
./gen_build -d '$@.d' > '$@' '$<'

Two things here that I think ninja fails at:

1. The include is prefixed with a dash, this means it won’t fail if the file does not exist, meaning on a “clean system” the user need not take any special actions.

2. Make will check if the file to be included is out-of-date, even if this is not the specified goal.

My attempt at porting this to ninja has resulted in the following:

builddir = /tmp/build

rule gen_build
command = ./gen_build -d $out.d > $out $in
depfile = $out.d
generator = 1

build $builddir/build.ninja: gen_build manifest | ./gen_build

subninja $builddir/build.ninja

Unfortunately on a clean system I need to bootstrap the process with:

mkdir -p /tmp/build && touch -t 200101010000 /tmp/build/build.ninja && ninja /tmp/build/build.ninja

Furthermore, I regularly need to invoke:

ninja /tmp/build/build.ninja

To be sure the build file is up-to-date.

I should add that the conditional include feature of make is also useful to include ‘Makefile.$USER’ — this will allow users to augment the goals (targets). For example I may add something like this to Makefile.«me»:

app/scp: app_binary
scp $< «my_other_machine»

app/deploy: app/scp

No-one but me want the ‘app/scp’ target since it is specific to my development environment.

Furthermore the above show that make’s ability to append to dependencies is sometimes rather useful. E.g. I assume that the standard Makefile already define ‘app/deploy’ as a target which builds and links the executable. I add my custom ‘app/scp’ as a dependency to this goal so on my system ‘make app/deploy’ additionally copies the application to my other machine.

[Evan Martin]

On Wed, Jul 25, 2012 at 7:25 AM, Allan Odgaard <text...@gmail.com> wrote:
> I generate my ninja build files and can’t figure out how to make it so that a) no special action is required for bootstrapping the build on a clean system and b) the involved ninja files are regenerated when required.

To start with, I haven't put much thought into this area so I'm open
to improvements.

It sounds like for (a) you want bootstrapping to happen implicitly
when you run "ninja" first?

I think the reason this hasn't come up before is that many (most?)
projects allow for some sort of arguments to the generator which ninja
won't know to pass through, so such projects require you to always
manually run the generator first. E.g. even with a simple app like
ninja there are flags like --debug or --with-gtest=/path/to/gtest that
you can pass to configure.py. Larger tools like cmake, gyp, or
autoconf have more extensive configure-time flags.

> Two things here that I think ninja fails at:
>
> 1. The include is prefixed with a dash, this means it won’t fail if the file does not exist, meaning on a “clean system” the user need not take any special actions.

It wouldn't be hard to extend ninja to allow some sort of "optional
include" syntax, but I want to to be sure it ends up being useful.

> My attempt at porting this to ninja has resulted in the following:
>
> builddir = /tmp/build
>
> rule gen_build
> command = ./gen_build -d $out.d > $out $in
> depfile = $out.d
> generator = 1
>
> build $builddir/build.ninja: gen_build manifest | ./gen_build
>
> subninja $builddir/build.ninja
>
> Unfortunately on a clean system I need to bootstrap the process with:
>
> mkdir -p /tmp/build && touch -t 200101010000 /tmp/build/build.ninja && ninja /tmp/build/build.ninja
>
> Furthermore, I regularly need to invoke:
>
> ninja /tmp/build/build.ninja
>
> To be sure the build file is up-to-date.

To be sure I follow you: I think what you're saying is that due to the
"subninja $builddir/build.ninja", you want ninja to check for changes
in that file and rerun the generator if necessary?

Right now the dependency rule is simplistic: any build implicitly does
the equivalent of running "ninja build.ninja" before running the build
itself. What I think you're requesting is something like:
build the-build-itself: build.ninja $builddir/build.ninja
and for ninja to instead compute the dependencies "the-build-itself"
as the first rule.

Part of the reason this is awkward to express is that ninja is
designed around the idea that the topmost build.ninja itself will be
generated. If that were the case, expressing the above is as simple
as adding $builddir/build.ninja to your generator build line, like:

build build.ninja: gen_build manifest | ./gen_build $builddir/build.ninja

(It is unfortunate that build.ninja ends up being a build-time file
that is written in your source tree. An alternative approach -- the
one we use in Chrome and the one encouraged by autoconf etc. -- is to
generate the build files like build.ninja in the build directory and
run all builds starting from that directory, either by cd'ing into
that directory first or by the -C flag.)

> I should add that the conditional include feature of make is also useful to include ‘Makefile.$USER’ — this will allow users to augment the goals (targets). For example I may add something like this to Makefile.«me»:
>
> app/scp: app_binary
> scp $< «my_other_machine»
>
> app/deploy: app/scp
>
> No-one but me want the ‘app/scp’ target since it is specific to my development environment.
>
> Furthermore the above show that make’s ability to append to dependencies is sometimes rather useful. E.g. I assume that the standard Makefile already define ‘app/deploy’ as a target which builds and links the executable. I add my custom ‘app/scp’ as a dependency to this goal so on my system ‘make app/deploy’ additionally copies the application to my other machine.

I think I follow you here. I'll note that there's no way to express
Makefile.$USER (where $USER is expanded in the environment) as an
input in Ninja syntax itself, though something like Makefile.user
could surely be done.


In all, I think the reason you're having to fight with ninja's
behavior here is that ninja assumes you'll be (re)generating
build.ninja. In that case, adding conditional includes that obey
environment variables etc. are trivial as you can just do that logic
in your build file generator. I worry that conditional includes may
end up not being enough to get all the build muscle you want, but I'm
willing to hear arguments for it.

[Allan Odgaard]

On 25 Jul 2012, at 20:19, Evan Martin wrote:

> […] It sounds like for (a) you want bootstrapping to happen implicitly

> when you run "ninja" first?

Correct.

> […] I think what you're saying is that due to the

> "subninja $builddir/build.ninja", you want ninja to check for changes
> in that file and rerun the generator if necessary?

Sort of, yes.

Basically what I would like is that when ninja sees:

subninja «file»

It implicitly does:

ninja «file»

To ensure that this file is built/updated before ninja reads it.

In the example I gave the rule to create $(builddir)/build.ninja has already been seen by ninja plus its depfile. I wouldn’t mind if this was a requirement, i.e. anything below the ‘subninja «file»’ command is effectively ignored when trying to build «file».

> (It is unfortunate that build.ninja ends up being a build-time file
> that is written in your source tree. An alternative approach -- the
> one we use in Chrome and the one encouraged by autoconf etc. -- is to
> generate the build files like build.ninja in the build directory and
> run all builds starting from that directory, either by cd'ing into
> that directory first or by the -C flag.)

A minor issue with the -C approach is that all paths need to be absolute, which makes for worse descriptions during a build, right?

Also, do you then add the -C «path» flag each time you call ninja? Or is there some clever trick to avoid that? Of course I am thinking of putting a build.ninja in the source tree that re-invokes ninja with -C but then we are back to the setup I presently have (except I include the file rather than re-invoke ninja, but that is, AFAIK, the only way to have all the targets of the real ninja file available)…

> In all, I think the reason you're having to fight with ninja's
> behavior here is that ninja assumes you'll be (re)generating

> build.ninja […]

Indeed it seems like it is made for this.

I see big value in not ever putting generated content into my source tree and additionally not making a distinction about how to perform a build based on whether or not the system is “clean”.

Especially when moving all generated stuff outside the source tree, it is not unusual to simply “rm -rf” the build dir, which makes the need to then call ./configure or similar a little awkward.

I follow your logic about options that might affect how to generate the build files, but I think claiming that many/most projects require this is narrow minded ;) I presently use ‘make’ for a lot of task automation, many of which has nothing to do with C/C++ and many are shared with friends/colleagues, none of these need a configure process like that of e.g. Chrome, and of course I want to slowly migrate all my automated “build” tasks to ninja :)

[Nico Weber]

>> (It is unfortunate that build.ninja ends up being a build-time file
>> that is written in your source tree. An alternative approach -- the
>> one we use in Chrome and the one encouraged by autoconf etc. -- is to
>> generate the build files like build.ninja in the build directory and
>> run all builds starting from that directory, either by cd'ing into
>> that directory first or by the -C flag.)
>
> A minor issue with the -C approach is that all paths need to be absolute, which makes for worse descriptions during a build, right?

No, you can make them relative to the -C directory.

> Also, do you then add the -C «path» flag each time you call ninja? Or is there some clever trick to avoid that? Of course I am thinking of putting a build.ninja in the source tree that re-invokes ninja with -C but then we are back to the setup I presently have (except I include the file rather than re-invoke ninja, but that is, AFAIK, the only way to have all the targets of the real ninja file available)…

There's no clever trick. We recommend setting up an alias for `ninja
-C blah` in the chromium build documentation.

Nico

[Evan Martin]

On Wed, Jul 25, 2012 at 1:07 PM, Allan Odgaard <text...@gmail.com> wrote:
> Basically what I would like is that when ninja sees:
>
> subninja «file»
>
> It implicitly does:
>
> ninja «file»
>
> To ensure that this file is built/updated before ninja reads it.
>
> In the example I gave the rule to create $(builddir)/build.ninja has already been seen by ninja plus its depfile. I wouldn’t mind if this was a requirement, i.e. anything below the ‘subninja «file»’ command is effectively ignored when trying to build «file».

Ah, that is a much better way of looking at it. I think we can
implement that without much trouble.

[Evan Martin]

On Wed, Jul 25, 2012 at 1:14 PM, Nico Weber <tha...@chromium.org> wrote:
>> Also, do you then add the -C «path» flag each time you call ninja? Or is there some clever trick to avoid that? Of course I am thinking of putting a build.ninja in the source tree that re-invokes ninja with -C but then we are back to the setup I presently have (except I include the file rather than re-invoke ninja, but that is, AFAIK, the only way to have all the targets of the real ninja file available)…
>
> There's no clever trick. We recommend setting up an alias for `ninja
> -C blah` in the chromium build documentation.

Another way to look at it: fundamentally, if you don't have any
generated files in your source tree, you need a way to somehow tell
ninja where your build files are. Options include:
- the current directory (this is how autoconf etc. work: build
instructions are frequently of the form "mkdir build; cd build;
../configure ...; make)
- a command-line parameter (like -C)
- an absolute path (what it appears your build files use(?))

We borrowed -C from make, but I like to read it as "configuration".
In Chrome's case you can pick between debug/release modes by picking
different build directories (we generate all the build files twice,
once for each mode, in part because the other build systems that gyp
targets like Xcode allow for different configurations from the same
build files).

[Allan Odgaard]

On 25 Jul 2012, at 22:19, Evan Martin wrote:

> […] if you don't have any generated files in your source tree, you need a way to somehow tell ninja where your build files are. Options include […] an absolute path (what it appears your build files use(?))

Only a default build directory is hardcoded. With make I would normally use a Makefile in my source tree containing something like this:

PROJECT_NAME = «name of this project»

-include Makefile.$(USER)
BUILD_DIR ?= $(HOME)/build/$(PROJECT_NAME)

$(BUILD_DIR)/Makefile: Manifest
bin/gen_makefile -o '$@' '$<'

-include $(BUILD_DIR)/Makefile

This allows the user to set build directory either as an environment variable (this is my preference since I want a central build directory for all my projects), set it in a custom Makefile.«user», or just accept the defaults.

Not trying to make a case for the above, just letting you know how I used to work with make.

[Evan Martin]

Just so I don't forget, I stuffed this into the bug tracker:
https://github.com/martine/ninja/issues/370

[Richard Geary]

Allan's suggestion would be very helpful for my build, so I can cleanly invoke a generator. I would also like this rebuild check to happen for include as well as subninja.
Once we have this, the Makefile.<USER> feature could be added by adding a rule to generate a ninja file with user=<USER>, then include it, then include Makefile.$user.
 
One feature not yet discussed is conditional includes. It would be helpful if include and subninja only ever included any file once. I can't see why you would ever want it included twice, it would generate duplicate build & rule targets.

The reason this would be useful for me is I have a large solution with many projects. I generate a build.ninja in each project's source folder. The root folder's build.ninja includes "header.ninja" first to define compile rules, then subninja's the projects. In the project's build.ninja, I would like to be able to write "include header.ninja" as the first line, as then I could build the project on its own with a simple "ninja" command from the directory. At the moment, adding multiple include header.ninja commands will give duplicate rule errors.
There's a second include bug which would need to be fixed for this to work. When I call "subninja src/projA/build.ninja", the "include ../../header.ninja" line at the top of that file doesn't work as include is looking for a file relative to the root ninja file. If I change it to "include header.ninja", I get a different problem as then the build doesn't work if I invoke ninja from src/projA. Instead, when you subninja or include, the relative paths should be relative to that new file.

[Nico Weber]

On Thu, Aug 2, 2012 at 8:00 AM, Richard Geary <richar...@gmail.com> wrote:
> Allan's suggestion would be very helpful for my build, so I can cleanly
> invoke a generator. I would also like this rebuild check to happen for
> include as well as subninja.
> Once we have this, the Makefile.<USER> feature could be added by adding a
> rule to generate a ninja file with user=<USER>, then include it, then
> include Makefile.$user.
>
> One feature not yet discussed is conditional includes. It would be helpful
> if include and subninja only ever included any file once. I can't see why
> you would ever want it included twice, it would generate duplicate build &
> rule targets.
>
> The reason this would be useful for me is I have a large solution with many
> projects. I generate a build.ninja in each project's source folder. The root
> folder's build.ninja includes "header.ninja" first to define compile rules,
> then subninja's the projects. In the project's build.ninja, I would like to
> be able to write "include header.ninja" as the first line, as then I could
> build the project on its own with a simple "ninja" command from the
> directory. At the moment, adding multiple include header.ninja commands will
> give duplicate rule errors.

Can't you just create build.ninja files that look like this for your subfolders:

include header.ninja
include target.ninja

…and then subninja all the target.ninjas in your root project?

[Richard Geary]

Yes good point, that would work too. But it would double the number of ninja files.

[Nico Weber]

If you care about file system cleanliness, you could write all the
target.ninja files into a dedicated build directory that's not in your
src/ tree (you could put all generated .o and other files there too).

[Richard Geary]

That would be a good alternative, I'll probably do that. But if the user deletes the build dirs, ninja will fail. Allan's suggestion would allow me to write a generator rule to rerun the configure tool.

Regardless, I can't think of a reason why you'd want to include a ninja file twice, or why you'd want all paths relative to where you invoke ninja. These seem like bugs not features.

[Maxim Kalaev]

On Thursday, August 2, 2012 5:24:43 PM UTC+2, Richard Geary wrote:

That would be a good alternative, I'll probably do that. But if the user deletes the build dirs, ninja will fail. Allan's suggestion would allow me to write a generator rule to rerun the configure tool.

Regardless, I can't think of a reason why you'd want to include a ninja file twice, or why you'd want all paths relative to where you invoke ninja. These seem like bugs not features.

Consider writing a shell wrapper to step side bootstrapping, conditionals, user specific settings, and syntax issues. And using it instead of running 'ninja' directly.

We have a brute force python script which forks shell which then runs find ( [ "$(find -name build.xmake.py -cnewer build.ninja -print -quit 2>&1)" = "" ] ) on entire build tree (~40K files when dirty) to decide whatever to rebuild ninja files before invoking ninja (build.xmake.py contain ~metarules).
Conditional settings are addressed by setting environment variables in script and then using these in the comands invoked by ninja rules.

The script end-to-end takes ~0.6 sec for noop build (with warm disk caches).

[lee.j.i...@gmail.com]

It this feature still in limbo?

It would solve a lot of the issues I am wrestling with in trying to construct an incremental generator.