CMake Ninja rules

[Neil Mitchell]

Hi,

I have been looking at compiling LLVM with Ninja, using Ninja rules generated from CMake, on 32bit Windows. In doing so, I found three issues:

1) Linking clang.exe and clang++.exe simultaneously takes out my machine, causing it to swap furiously for a long period of time. It would be great if the generated Ninja rules included pools ideally limiting the build to at most one link step at a time. At the moment I am using -j1 on my 4 CPU machine just so I don't freeze at link time.

2) The rule to generate tools/clang/lib/Basic/CMakeFiles/clang_revision_tag does not actually generate that file. As a result, in every execution Ninja considers that file "dirty" and rebuilds it. To work around this problem I touch'd that file, and everything worked. I am not sure if this is an LLVM problem, or a Ninja CMake generator problem.

3) The rules do not use the new deps=gcc flag, which I calculate costs Ninja at least 0.1s of rebuild time when there is nothing to do. On IRC Evan mentioned that he was considering making deps the only way of doing things (and presumably on by default), in which case there would be no need to change the generator apart from when configuring to use Visual Studio.

The motivation behind this exercise was to compare the build speeds of Ninja against those of Shake (http://hackage.haskell.org/package/shake), a Haskell build library, to which I recently added a Ninja build script interpreter. As things currently stand, to build when nothing has changed Shake takes 0.805s while Ninja takes 0.880s. These timings are both working off the same build.ninja file. I calculate that the lack of deps=gcc costs Shake about 0.1s (since it has to check the modification times on all the makefiles) and presumably costs Ninja more than that (as it has to actually read the makefiles).

Thanks, Neil

[Evan Martin]

I think all of these questions are unfortunately limitations of the CMake ninja file generator, so I can't help you directly, but the authors of that code are on this list so they probably have better answers.

On Thu, Jun 6, 2013 at 1:38 PM, Neil Mitchell <ndmit...@gmail.com> wrote:

I have been looking at compiling LLVM with Ninja, using Ninja rules generated from CMake, on 32bit Windows. In doing so, I found three issues:

1) Linking clang.exe and clang++.exe simultaneously takes out my machine, causing it to swap furiously for a long period of time. It would be great if the generated Ninja rules included pools ideally limiting the build to at most one link step at a time. At the moment I am using -j1 on my 4 CPU machine just so I don't freeze at link time.

One workaround I've used before is to hack a linker binary into the front of your path that knows to run only one at a time.  This is trivial on Linux with the "flock" command; maybe there's something similar on Windows.

2) The rule to generate tools/clang/lib/Basic/CMakeFiles/clang_revision_tag does not actually generate that file. As a result, in every execution Ninja considers that file "dirty" and rebuilds it. To work around this problem I touch'd that file, and everything worked. I am not sure if this is an LLVM problem, or a Ninja CMake generator problem.

3) The rules do not use the new deps=gcc flag, which I calculate costs Ninja at least 0.1s of rebuild time when there is nothing to do. On IRC Evan mentioned that he was considering making deps the only way of doing things (and presumably on by default), in which case there would be no need to change the generator apart from when configuring to use Visual Studio.

As I mentioned on IRC, the deps thing is new, so we're being a bit cautious.  I should probably put a warning on it in the manual.

I think the CMake developers are even more cautious when introducing the use of new features so I'm not sure where they stand on adding this to CMake.  Perhaps in the next release of Ninja we'll just make it the default and so older versions of CMake will get the optimization transparently.

Note that on Windows, unless you're intentionally using gcc, you ought to build with Visual Studio's compiler and deps=msvc.  But CMake doesn't support that yet either.

The motivation behind this exercise was to compare the build speeds of Ninja against those of Shake (http://hackage.haskell.org/package/shake), a Haskell build library, to which I recently added a Ninja build script interpreter. As things currently stand, to build when nothing has changed Shake takes 0.805s while Ninja takes 0.880s. These timings are both working off the same build.ninja file. I calculate that the lack of deps=gcc costs Shake about 0.1s (since it has to check the modification times on all the makefiles) and presumably costs Ninja more than that (as it has to actually read the makefiles).

Cool!  For others on the mailing list, I'll add that I believe that while normal use of Shake involves a Haskell compiler to compile your build rules, I believe the Ninja support works by reading the Ninja files dynamically, which (I think?) means you can just drop a built shake.exe in to your build to use it.  (Please correct me if I got that wrong.)

[Neil Mitchell]

>> 1) Linking clang.exe and clang++.exe simultaneously takes out my machine,
>> causing it to swap furiously for a long period of time. It would be great if
>> the generated Ninja rules included pools ideally limiting the build to at
>> most one link step at a time. At the moment I am using -j1 on my 4 CPU
>> machine just so I don't freeze at link time.
>
> One workaround I've used before is to hack a linker binary into the front of
> your path that knows to run only one at a time. This is trivial on Linux
> with the "flock" command; maybe there's something similar on Windows.

It's probably easier for me to manually tweak the generated
build.ninja for the moment. The advantage of using pools over flock is
that you don't waste a build CPU waiting on the lock, as you can build
other things while waiting for the linker lock to become available (or
certainly Shake can).

> As I mentioned on IRC, the deps thing is new, so we're being a bit cautious.
> I should probably put a warning on it in the manual.
>
> I think the CMake developers are even more cautious when introducing the use
> of new features so I'm not sure where they stand on adding this to CMake.
> Perhaps in the next release of Ninja we'll just make it the default and so
> older versions of CMake will get the optimization transparently.

Given a command to produce foo with depsfile=bar, if you are happy to
declare that Ninja only needs to read the contents of bar after
running foo, and does not need to check bar every time, I'll implement
that logic in Shake. Shake always puts the conents of bar as
dependencies of foo, and at the moment it also introduces a dependency
of foo on bar. It sounds like given your thoughts dropping the
dependency is reasonable, which would save me 0.1s.

> Note that on Windows, unless you're intentionally using gcc, you ought to
> build with Visual Studio's compiler and deps=msvc. But CMake doesn't
> support that yet either.

I am intentionally building with gcc, since there were step-by-step
instructions on the web that worked first time with MingW/gcc.

> Cool! For others on the mailing list, I'll add that I believe that while
> normal use of Shake involves a Haskell compiler to compile your build rules,
> I believe the Ninja support works by reading the Ninja files dynamically,
> which (I think?) means you can just drop a built shake.exe in to your build
> to use it. (Please correct me if I got that wrong.)

You are correct. Normally people use Shake as a library, the usage
looks much more like the typical configure.py Ninja scripts, which are
then compiled to produce a build system for that project. You
typically need to recompile the script whenever in Ninja you would
change the configure.py script, not just when you rerun the
configure.py script. When running over a Ninja script, you can drop a
prebuilt binary of shake next to the ninja one, and do not need any
Haskell compiler on your build machine etc.

[Peter Kümmel]

On 06.06.2013 22:38, Neil Mitchell wrote:
> Hi,
>
> I have been looking at compiling LLVM with Ninja, using Ninja rules generated from CMake, on 32bit Windows. In doing so, I found three issues:
>
> 1) Linking clang.exe and clang++.exe simultaneously takes out my machine, causing it to swap furiously for a long period of time. It would be great if the generated Ninja rules included pools ideally limiting the build to at most one link step at a time. At the moment I am using -j1 on my 4 CPU machine just so I don't freeze at
> link time.

Add added this feature recently, git://github.com/syntheticpp/CMake.git
the ninja-pool branch. Example usage is listed in the commit message.

So if have the time to compile cmake and to patch Clang's cmake files,
you have a good chance that you can build with all your cores.

Peter

>
> 2) The rule to generate tools/clang/lib/Basic/CMakeFiles/clang_revision_tag does not actually generate that file. As a result, in every execution Ninja considers that file "dirty" and rebuilds it. To work around this problem I touch'd that file, and everything worked. I am not sure if this is an LLVM problem, or a Ninja CMake
> generator problem.
>
> 3) The rules do not use the new deps=gcc flag, which I calculate costs Ninja at least 0.1s of rebuild time when there is nothing to do. On IRC Evan mentioned that he was considering making deps the only way of doing things (and presumably on by default), in which case there would be no need to change the generator apart from
> when configuring to use Visual Studio.
>
> The motivation behind this exercise was to compare the build speeds of Ninja against those of Shake (http://hackage.haskell.org/package/shake), a Haskell build library, to which I recently added a Ninja build script interpreter. As things currently stand, to build when nothing has changed Shake takes 0.805s while Ninja takes
> 0.880s. These timings are both working off the same build.ninja file. I calculate that the lack of deps=gcc costs Shake about 0.1s (since it has to check the modification times on all the makefiles) and presumably costs Ninja more than that (as it has to actually read the makefiles).
>
> Thanks, Neil
>

> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Neil Mitchell]

>> 1) Linking clang.exe and clang++.exe simultaneously takes out my machine,
>> causing it to swap furiously for a long period of time. It would be great if
>> the generated Ninja rules included pools ideally limiting the build to at
>> most one link step at a time. At the moment I am using -j1 on my 4 CPU
>> machine just so I don't freeze at
>> link time.
>
> Add added this feature recently, git://github.com/syntheticpp/CMake.git
> the ninja-pool branch. Example usage is listed in the commit message.
>
> So if have the time to compile cmake and to patch Clang's cmake files,
> you have a good chance that you can build with all your cores.

Great. Will that be making it into a future CMake release? As soon as
they are upstream I can probably grab a nightly build.

I was talking to someone in real life recently and they mentioned they
had patches to CMake to use both deps and pools, I suspect the
addition of deps is trivial, but if you'd like I can try and get hold
of their code.

Thanks, Neil

[Peter Kümmel]

On 22.06.2013 23:29, Neil Mitchell wrote:
>>> 1) Linking clang.exe and clang++.exe simultaneously takes out my machine,
>>> causing it to swap furiously for a long period of time. It would be great if
>>> the generated Ninja rules included pools ideally limiting the build to at
>>> most one link step at a time. At the moment I am using -j1 on my 4 CPU
>>> machine just so I don't freeze at
>>> link time.
>>
>> Add added this feature recently, git://github.com/syntheticpp/CMake.git
>> the ninja-pool branch. Example usage is listed in the commit message.
>>
>> So if have the time to compile cmake and to patch Clang's cmake files,
>> you have a good chance that you can build with all your cores.
>
> Great. Will that be making it into a future CMake release? As soon as
> they are upstream I can probably grab a nightly build.

I've uploaded win32 binaries here,

https://sourceforge.net/projects/cmakescript/files/CMake%20Ninja%20pools/

so if you think it is usable I could try to bring it upstream.
Example cmake code is listed in the Readme.txt.

>
> I was talking to someone in real life recently and they mentioned they
> had patches to CMake to use both deps and pools, I suspect the
> addition of deps is trivial, but if you'd like I can try and get hold
> of their code.

ATM, I don't have much time, but you could mail the patches to the cmake list,
maybe someone picks them up.

Peter