How to make ninja aware of dynamic implicit outputs

65 views
Skip to first unread message

Marc Delorme

unread,
Mar 29, 2021, 4:06:57 AMMar 29
to ninja-build
Hello,

In my project, I have a tool generating reflection data by parsing C++ headers. This tool read a set of C++ headers, find classes to be reflected, generates one header file per reflected classes (implicit outputs) and generate one .cpp containing reflection data implementation (explicit outputs).

I would like ninja to re-generate the header file is they are missing. It is possible to do it using the dyndep feature, but it requires to split my code generation in two steps. Unfortunately discovering those implicit outputs require to parse all the C++, from that point it is a pity not to generate code at the same time but reparse everything in another step to generate code.

I wish I could register implicit outputs in depfile, but I did not find in depfile syntax to do that and by reading ninja source code it does not seem possible.

Ben Boeckel

unread,
Mar 29, 2021, 8:49:34 AMMar 29
to Marc Delorme, ninja-build
On Mon, Mar 29, 2021 at 00:36:51 -0700, Marc Delorme wrote:
> In my project, I have a tool generating reflection data by parsing C++
> headers. This tool read a set of C++ headers, find classes to be reflected,
> generates one header file per reflected classes (implicit outputs) and
> generate one .cpp containing reflection data implementation (explicit
> outputs).
>
> I would like ninja to re-generate the header file is they are missing. It
> is possible to do it using the *dyndep* feature, but it requires to split
> my code generation in two steps. Unfortunately discovering those implicit
> outputs require to parse all the C++, from that point it is a pity not to
> generate code at the same time but reparse everything in another step to
> generate code.

Dyndep needs to be separate because it is about finding what is *needed*
for another step to run first.

> I wish I could register implicit outputs in depfile, but I did not find in
> depfile syntax to do that and by reading ninja source code it does not seem
> possible.

There would need to be a new syntax in the depfile for this. Probably
guarded by an `dynout = 1` flag or the like. Alternatively, it could
just be a `dynout = list.txt` possibly with a `dynprefix =
path/to/prefix/to/each/entry` (to allow for `tar xf` / `tar tf`
simplicity).

FWIW, this could also be used to rerun rules when files are deleted from
"batched" operations (e.g., extracting a tarball) by tracking the
extracted files via `dynout`.

--Ben

Marc Delorme

unread,
Apr 2, 2021, 12:18:29 PMApr 2
to Ben Boeckel, ninja-build
Thank you for your comment Ben.

I will try to make an implementation on my side. I am considering reusing the same syntax as dyndep since the parser exists and the format already contains all the information required. It could be as simple as adding a new "deps" type. "deps" could be either "msvc", "gcc", or "dyn". The NinjaDeps database could be extended to include dynamic output information.

Actually there are a lot of design choice which would lead to the same result:
- Extending NinjaDeps or adding an output database. (Pro reuse existing code, Cons: "deps" naming become confusing because it will also include the concept of outputs)
- Re-using the dyndep format or creating a new one. (Pro reuse existing code, Cons: the format requires the tool to also print static input)

Marc Delorme

unread,
Apr 14, 2021, 7:51:08 AMApr 14
to ninja-build
I have implemented the `dynout = list.txt` variable for rule, it works good enough for my projects. (you can find it in that branch: https://github.com/Dragnalith/ninja/tree/dynamic_outputs)

I would love to have this feature on the official release of ninja. I can work more to make my implementation reach the quality to be integrated in master. I think the feature is more about discussing its interface and its design. The implementation itself is not so complicated. But I am lost about the contribution process. How to suggest on new feature? Who does validate the design? Who take decision? etc.

Ben Boeckel

unread,
Apr 14, 2021, 9:31:34 AMApr 14
to Marc Delorme, ninja-build
On Tue, Apr 13, 2021 at 23:46:36 -0700, Marc Delorme wrote:
> I have implemented the `dynout = list.txt` variable for rule, it works good
> enough for my projects. (you can find it in that branch:
> https://github.com/Dragnalith/ninja/tree/dynamic_outputs)
>
> I would love to have this feature on the official release of ninja. I can
> work more to make my implementation reach the quality to be integrated in
> master. I think the feature is more about discussing its interface and its
> design. The implementation itself is not so complicated. But I am lost
> about the contribution process. How to suggest on new feature? Who does
> validate the design? Who take decision? etc.

Jan Hasse (@jhasse) is the lead maintainer these days. I'd open an PR
with the change. I don't know what kind of schedule you can expect (I'm
just another fellow contributor to ninja).

--Ben

jha...@gmail.com

unread,
Apr 14, 2021, 10:07:11 AMApr 14
to ninja-build
Hi,

On Wednesday, 14 April 2021 at 13:51:08 UTC+2 delorm...@gmail.com wrote:
I would love to have this feature on the official release of ninja. I can work more to make my implementation reach the quality to be integrated in master. I think the feature is more about discussing its interface and its design. The implementation itself is not so complicated. But I am lost about the contribution process.

 
How to suggest on new feature?

Create an issue on GitHub, I will add the "feature" tag then.
 
Who does validate the design?

Anyone who likes to comment :)
 
Who take decision? etc.

You can spot users with commit rights by their tag "Collaborator" on GitHub.

Marc Delorme

unread,
Apr 15, 2021, 10:59:12 AMApr 15
to ninja-build
Hi Jan,

Thank you for sharing the CONTRIBUTING.md.

I have created two PRs related to those incremental build issues:
they address two scenarios where running ninja does not give the same result as if you where running it after a clean build.

-Marc
Reply all
Reply to author
Forward
0 new messages