cmake patch available to leverage ninja "deps" feature, to improve incremental build performance

[Edwin Jacques]

I have submitted a patch for feature request 14240 for cmake ninja generator to "Support new, faster header dependency handling".

Basically, it supports the "deps" feature which will make the cmake-generated ninja build system use ninja's fast, compact dependency database instead of reading all dependency files for every incremental build (one per C/C++ file).  This can be a big help for a huge code base.  Supports GNU, Clang and MSVC compilers.  I tested all but MSVC, and standard regression tests are passing.

I thought the ninja community may be interested in early access.

--Edwin

[Evan Martin]

Awesome! I think, for full Windows support, you (or someone) will
need to figure out a more resilient way of parsing the /showIncludes
output of msvc. As I recall, the cmake parser was extended to handle
the output of e.g. Chinese locales:
https://github.com/martine/ninja/issues/613

> --
> You received this message because you are subscribed to the Google Groups
> "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Nico Weber]

Cool!

I'm not very familiar with cmake, but I think it used to pipe cl output through a program called cmcldeps.exe (which writes /showIncludes output to a .d file, and ninja used to read it from there. With depsmode, you can remove the .d file completely by just adding /showIncludes to the compiler flags and piping the output into ninja -t msvc – then you save one process and some disk accesses at build time (if you do this, don't set the 'depfile' attribute in the generated ninja file, only set 'deps = msvc'). If you do this, you run into the problem Evan mentions though, so maybe it could be an experimental feature. (Or maybe the performance benefit isn't worth the additional complexity in cmake.)

Nico

On Thu, Oct 10, 2013 at 9:51 AM, Edwin Jacques <edwin....@gmail.com> wrote:

--

[Peter Kümmel]

I've uploaded a merge request for handling localized prefixes. When it is merged I will patch cmake, so cmcldeps.exe is not used for compiling with cl.exe any more.

[Edwin Jacques]

When I build/test cmake with my patch, I ran into problems related to cmcldeps.exe, where it needs to look at the dependency file that is expected to be written to the file system.  The problem of course is ninja deletes the file right after it's created, so it's not there.

I did an experiment where I removed the logic in ninja to remove the dependency files after processing, made regression tests pass, and then re-ran the same cmake regression tests that had failed.  Everything passed.

This brought a question to my mind.  Do we have to remove the dependency files after they are processed in "deps" mode?  It would be good if the files could be left around, even if only as an option (like -t keepdeps).  Then cmake in Windows can support fast incremental build for huge code trees without breaking any logic currently dependent on cmcldeps.exe.  Personally, I don't mind leaving the files around at all.

What do you think?

--Edwin

[Peter Kümmel]

cmcldeps will not be called when deps = msvc is used (only
for .rc files),
and no files will be created by it, so additionally creating
them means a slowdown.

I've extended you patch to not use cmcldeps here:
https://github.com/syntheticpp/CMake/commits/ninja-remove-cmcldeps
https://github.com/syntheticpp/CMake/commit/87d562ef0c079a811b29b0d82d235b521e393839

It's not ready. After pushing I realized we need cmcldeps
for .rc files and
the support for localized /showIncludes strings is missing.

Does it work for you?

Peter

On 17.10.2013 07:06, Edwin Jacques wrote:
> When I build/test cmake with my patch, I ran into problems related to
> cmcldeps.exe, where it needs to look at the dependency file that is
> expected to be written to the file system. The problem of course is ninja
> deletes the file right after it's created, so it's not there.
>
> I did an experiment where I removed the logic in ninja to remove the
> dependency files after processing, made regression tests pass, and then
> re-ran the same cmake regression tests that had failed. Everything passed.
>
> This brought a question to my mind. Do we have to remove the dependency
> files after they are processed in "deps" mode? It would be good if the
> files could be left around, even if only as an option (like -t keepdeps).
> Then cmake in Windows can support fast incremental build for huge code
> trees without breaking any logic currently dependent on cmcldeps.exe.
> Personally, I don't mind leaving the files around at all.
>
> What do you think?
>
> --Edwin
>
> On Thursday, October 10, 2013 10:39:11 PM UTC-4, Nico Weber wrote:
>>
>> Cool!
>>
>> I'm not very familiar with cmake, but I think it used to pipe cl output
>> through a program called cmcldeps.exe (which writes /showIncludes output to
>> a .d file, and ninja used to read it from there. With depsmode, you can
>> remove the .d file completely by just adding /showIncludes to the compiler

>> flags and piping the output into ninja -t msvc � then you save one process

>> and some disk accesses at build time (if you do this, don't set the
>> 'depfile' attribute in the generated ninja file, only set 'deps = msvc').
>> If you do this, you run into the problem Evan mentions though, so maybe it
>> could be an experimental feature. (Or maybe the performance benefit isn't
>> worth the additional complexity in cmake.)
>>
>> Nico
>>
>>

>> On Thu, Oct 10, 2013 at 9:51 AM, Edwin Jacques <edwin....@gmail.com<javascript:>
>>> wrote:
>>
>>> I have submitted a patch for feature request 14240<http://www.cmake.org/Bug/view.php?id=14240> for

>>> cmake ninja generator to "Support new, faster header dependency
>>> handling".
>>>
>>> Basically, it supports the "deps" feature which will make the
>>> cmake-generated ninja build system use ninja's fast, compact dependency
>>> database instead of reading all dependency files for every incremental
>>> build (one per C/C++ file). This can be a big help for a huge code base.
>>> Supports GNU, Clang and MSVC compilers. I tested all but MSVC, and
>>> standard regression tests are passing.
>>>
>>> I thought the ninja community may be interested in early access.
>>>
>>> --Edwin
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "ninja-build" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to ninja-build...@googlegroups.com <javascript:>.