How ninja compares to other build systems for a large C project

[David Corvoysier]

Hi,

I did a quick benchmark of typical build system solutions for a large C project.

My primary goal was to compare recursive and non-recursive Make solutions, but I also included Ninja.

This won't probably surprise you that Ninja is really fast for incremental builds (so so for full builds, but CMake may be the culprit).

I also wrote a blog post detailing the several solutions I tested.

Cheers

David

[Nico Weber]

Cool, thanks for the note!

I'm surprised that ninja's slower for full builds than some of the alternatives. Do you know why that is? Do the ninja files generated by cmake contain unnecessary dependencies maybe? Have you considered writing the ninja files yourself, to see how fast an optimal-for-your-testcase ninja file would do? You can use misc/ninja_syntax.py to write ninja files from a Python script, and misc/write_fake_manifests.py is a slightly larger example. You'll want to make sure that there are no dependencies between all the compiles.

You could use https://github.com/nico/ninjatracing or similar to load a visualization of your ninja build into chrome's about:tracing, to see where the time goes.

Nico

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad King]

On 09/06/2016 09:37 AM, Nico Weber wrote:
> Do the ninja files generated by cmake contain unnecessary dependencies maybe?

Yes:

https://gitlab.kitware.com/cmake/cmake/issues/15555

CMake generates conservative extra dependencies to ensure that the
build semantics match those of other generators, in particular w.r.t.
generated files. With a deeper analysis many of them could be left
out, but that has not been implemented.

-Brad

[Alexander Usov]

To be compatible with Makefile generators and with projects which generate sources cmake inserts strong compilation barriers between different libraries.

So if you have in your cmake file something like:

add_library(libA   a1.c a2.c)

add_library(libB  b1.c b2.c)

target_link_libraries(libB libA)

Cmake will insert dependency on libA being done & linked before it actually starts compiling any file from libB.

In recent cmake versions there is relatively less painful method to work it around by using link-only dependencies and possibly manually propagating include paths.

[David Corvoysier]

I had actually tried generating the build.ninja from my Makefile fragments, but for the biggest trees I was stuck with the infamous 'Argument list too long' error.

I fixed it today using a response file and updated the benchmark: this improves cold start and rebuild quite a lot.

The build.ninja generation seems to be now the bottleneck: Make is definitely not the best tool for parsing a tree and generating a file.

I could give python a try, or maybe replace my Makefile fragments by Ninja fragments using include or subninja (that's what I did for the CMake solution).

The whole point however is to keep these fragments simple: I don't know if I can easily simplify the object/subdirs syntax using simple variable substitutions based on a set of predefined rules.  

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

[David Corvoysier]

I tried link-only dependencies but couldn't figure out how to use them recursively ...

[Evan Martin]

On Wed, Sep 7, 2016 at 7:23 AM, David Corvoysier
<david.co...@gmail.com> wrote:
> I had actually tried generating the build.ninja from my Makefile fragments,
> but for the biggest trees I was stuck with the infamous 'Argument list too
> long' error.
>
> I fixed it today using a response file and updated the benchmark: this
> improves cold start and rebuild quite a lot.

I'm surprised you're encountering this problem on non-Windows. Can
you pastebin your generated build.ninja file somewhere?

> The build.ninja generation seems to be now the bottleneck: Make is
> definitely not the best tool for parsing a tree and generating a file.

One assumption of Ninja (which comes from my experience) is that you
edit source code frequently, but you add or move files infrequently.
In the former case you don't need to regenerate the build.ninja file,
only in the latter, so a slow step to generate build.ninja isn't such
a bad problem.

> I could give python a try, or maybe replace my Makefile fragments by Ninja
> fragments using include or subninja (that's what I did for the CMake
> solution).
>
> The whole point however is to keep these fragments simple: I don't know if I
> can easily simplify the object/subdirs syntax using simple variable
> substitutions based on a set of predefined rules.

I agree that it's convenient to express your build in some
higher-level simple language.

One way to look at Ninja is that you pay for the evaluation of that
higher-level langauge (your fragments) into a concrete build graph
once, at the time you generate the build.ninja, and then subsequent
builds get to reuse that output quickly. So I think your current
design (complex program generates simple build.ninja) matches how I
intended Ninja to be used. Ninja isn't really designed to support
tricky layouts using includes or variable substitutions, though
sometimes they are possible.

[Nico Weber]

On Wed, Sep 7, 2016 at 10:23 AM, David Corvoysier <david.co...@gmail.com> wrote:

I had actually tried generating the build.ninja from my Makefile fragments,

Somewhat related to this: https://github.com/google/kati can convert Makefiles to ninja files. It's used to build Android nowadays; it might work with your makefiles as well.

 

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[David Corvoysier]

OK, I wrote a small python generator for the build.ninja file, and the cold start is now as fast as a full rebuild.

Although I understand you didn't design Ninja so that Ninja files can easily be tweaked by users, I may nevertheless try in the future a solution based on Ninja fragments.

The use case I have in mind is when you provide a framework to external developers: they may need to add custom rules in their code, which may be difficult if you only support a limited syntax for fragments.

That is one of the strength of Make based solutions: each fragment is a Makefile, and can be easily customized.

[Evan Martin]

I took a look at your generated build files.

It looks like you're using absolute paths. These are an antipattern
for Ninja and also explain why you're encountering command line limit
problems.

They look like this:
build /usr/local/google/home/evanm/projects/build-benchmark/output/src/main.o:
cc /us
r/local/google/home/evanm/projects/build-benchmark/output/src/main.c
cflags = -D'CURDIR=output/src'
build /usr/local/google/home/evanm/projects/build-benchmark/output/src/foo.o:
cc /usr
/local/google/home/evanm/projects/build-benchmark/output/src/foo.c
cflags = -D'CURDIR=output/src'

I don't really understand your layout but a patch like the following
seems to help.

The output then looks like this:
build ../src/main.o: cc ../src/main.c
cflags = -D'CURDIR=output/src'
build ../src/foo.o: cc ../src/foo.c
cflags = -D'CURDIR=output/src'

and you no longer need rspfiles. It should also be faster.

[Evan Martin]

On Thu, Sep 8, 2016 at 7:59 AM, David Corvoysier
<david.co...@gmail.com> wrote:
> Although I understand you didn't design Ninja so that Ninja files can easily
> be tweaked by users, I may nevertheless try in the future a solution based
> on Ninja fragments.
>
> The use case I have in mind is when you provide a framework to external
> developers: they may need to add custom rules in their code, which may be
> difficult if you only support a limited syntax for fragments.
>
> That is one of the strength of Make based solutions: each fragment is a
> Makefile, and can be easily customized.

The observation that led to Ninja is that in practice, within a given
project's build, the requirements to add a custom rule to the project
vary wildly: one project might want to put all of its build files next
to the source, another in some "build" subdir, and similarly with
where the outputs go. And also you need to figure out how to stitch
those outputs back in to the dependency graph (e.g. if you're
generating source, how do the other build steps learn where the
generated source lives?).

So you end up wanting to provide some higher-level API for your custom
rules that understands project-specific details like your directory
layout. And then once you're at that point you're back to a
general-purpose programming language and it's again slow. You're
certainly welcome to try to make such a thing work in Ninja but I
expect it won't be especially convenient. I think the way you have it
now, where your "fragments" specify the build in a nice user-friendly
way, is pretty much what I recommend doing.

[Evan Martin]

One last comment: your machinery around ninja itself contributes a lot of work.

$ (time -p make ninja) |& grep real
is about 2x as slow as
$ (time -p ninja -C output/ninja/) |& grep real

Both times are so short as to not really matter, but ideally you
wouldn't run a wrapper around ninja as it's designed to run with as
minimal work as possible.

[David Corvoysier]

Le jeudi 8 septembre 2016 21:03:28 UTC+2, Evan Martin a écrit :

Both times are so short as to not really matter, but ideally you
wouldn't run a wrapper around ninja as it's designed to run with as
minimal work as possible.

The build.ninja is not static: it must be regenerated if one of the fragments is modified, hence the 'machinery'.

For a tree with more than 10000 directories, doing a full build:

$time make -C output/ninja -j16

...

real    0m48.841s

user    2m46.818s

sys     1m28.392s

And rebuilding:

$time make ninja -j16

...

real    0m0.325s

user    0m0.197s

sys     0m0.128s

whereas:

$ time ninja -C output/ninja/ -j16

...

real    0m0.148s

user    0m0.102s

sys     0m0.045s

I am quite happy with my solution relying on find, but I would definitely accept any other methods that takes less than 150 ms to verify if one of the 10000 fragments has changed ...

[David Corvoysier]

There was indeed a mistake in my generator: the path to the objects should have been relative to the build directory.

I use absolute path for the sources because it is an out-of-tree build, and I cannot make any assumptions on the efficiency of a relative path between the source and build directories.

I tried your patch anyway: this doesn't solve the 'Argument list too long' error for the biggest tree (> 10000 files). 

[Nico Weber]

If you give your build.ninja a build rule for build.ninja itself and make it depend on all your fragment files, and set `generator = 1` on that rule, then ninja will figure out if it needs to rerun the generator script and do so if necessary. (That's how cmake knows it needs to rerun itself when you touch a CMakeLists.txt file for example.)

 

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[David Corvoysier]

Awesome ! I will do that ...

Thank you all for the support.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

[Evan Martin]

On Fri, Sep 9, 2016 at 6:03 AM, David Corvoysier
<david.co...@gmail.com> wrote:
> There was indeed a mistake in my generator: the path to the objects should
> have been relative to the build directory.
> I use absolute path for the sources because it is an out-of-tree build, and
> I cannot make any assumptions on the efficiency of a relative path between
> the source and build directories.

That makes sense. One thing we do in the Chrome build is set up a
variable for the base directories, so that paths look like $src/foo.c
and $out/foo.o. I added variables to Ninja originally just to help
make the files readable as I was debugging things but I suspect (but
haven't verified) that they may help with parsing input files (as it's
less bytes to read).

However, the variables are expanded before the command is executed so
it wouldn't help with with the command line length limit. In Chrome's
build we use intermediate static libraries (.a) to bundle together
intermediate outputs. (Note that ar supports "thin" archives, which
are effectively lists of files; so I guess it ends up effectively
working like multiple response files anyway.)

[Konstantin Tokarev]

09.09.2016, 19:06, "Evan Martin" <evan....@gmail.com>:

Static library and object list have different behavior, and I suppose thin archive
behaves like usual static library.

>
> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Regards,
Konstantin

[Behrooz Amoozad]

So, I'm trying to compile a very large vala project(~9000 files, fairly long file names, converted from C#) and it has to have all the files supplied to its compiler(valac), Any suggestions?