How to make ninja aware of dynamic implicit outputs

[Marc Delorme]

Hello,

In my project, I have a tool generating reflection data by parsing C++ headers. This tool read a set of C++ headers, find classes to be reflected, generates one header file per reflected classes (implicit outputs) and generate one .cpp containing reflection data implementation (explicit outputs).

I would like ninja to re-generate the header file is they are missing. It is possible to do it using the dyndep feature, but it requires to split my code generation in two steps. Unfortunately discovering those implicit outputs require to parse all the C++, from that point it is a pity not to generate code at the same time but reparse everything in another step to generate code.

I wish I could register implicit outputs in depfile, but I did not find in depfile syntax to do that and by reading ninja source code it does not seem possible.

[Ben Boeckel]

On Mon, Mar 29, 2021 at 00:36:51 -0700, Marc Delorme wrote:
> In my project, I have a tool generating reflection data by parsing C++
> headers. This tool read a set of C++ headers, find classes to be reflected,
> generates one header file per reflected classes (implicit outputs) and
> generate one .cpp containing reflection data implementation (explicit
> outputs).
>
> I would like ninja to re-generate the header file is they are missing. It

> is possible to do it using the *dyndep* feature, but it requires to split

> my code generation in two steps. Unfortunately discovering those implicit
> outputs require to parse all the C++, from that point it is a pity not to
> generate code at the same time but reparse everything in another step to
> generate code.

Dyndep needs to be separate because it is about finding what is *needed*
for another step to run first.

> I wish I could register implicit outputs in depfile, but I did not find in
> depfile syntax to do that and by reading ninja source code it does not seem
> possible.

There would need to be a new syntax in the depfile for this. Probably
guarded by an `dynout = 1` flag or the like. Alternatively, it could
just be a `dynout = list.txt` possibly with a `dynprefix =
path/to/prefix/to/each/entry` (to allow for `tar xf` / `tar tf`
simplicity).

FWIW, this could also be used to rerun rules when files are deleted from
"batched" operations (e.g., extracting a tarball) by tracking the
extracted files via `dynout`.

--Ben

[Marc Delorme]

Thank you for your comment Ben.

I will try to make an implementation on my side. I am considering reusing the same syntax as dyndep since the parser exists and the format already contains all the information required. It could be as simple as adding a new "deps" type. "deps" could be either "msvc", "gcc", or "dyn". The NinjaDeps database could be extended to include dynamic output information.

Actually there are a lot of design choice which would lead to the same result:

- Extending NinjaDeps or adding an output database. (Pro reuse existing code, Cons: "deps" naming become confusing because it will also include the concept of outputs)

- Re-using the dyndep format or creating a new one. (Pro reuse existing code, Cons: the format requires the tool to also print static input)

[Marc Delorme]

I have implemented the `dynout = list.txt` variable for rule, it works good enough for my projects. (you can find it in that branch: https://github.com/Dragnalith/ninja/tree/dynamic_outputs)

I would love to have this feature on the official release of ninja. I can work more to make my implementation reach the quality to be integrated in master. I think the feature is more about discussing its interface and its design. The implementation itself is not so complicated. But I am lost about the contribution process. How to suggest on new feature? Who does validate the design? Who take decision? etc.

[Ben Boeckel]

On Tue, Apr 13, 2021 at 23:46:36 -0700, Marc Delorme wrote:
> I have implemented the `dynout = list.txt` variable for rule, it works good
> enough for my projects. (you can find it in that branch:
> https://github.com/Dragnalith/ninja/tree/dynamic_outputs)
>
> I would love to have this feature on the official release of ninja. I can
> work more to make my implementation reach the quality to be integrated in
> master. I think the feature is more about discussing its interface and its
> design. The implementation itself is not so complicated. But I am lost
> about the contribution process. How to suggest on new feature? Who does
> validate the design? Who take decision? etc.

Jan Hasse (@jhasse) is the lead maintainer these days. I'd open an PR
with the change. I don't know what kind of schedule you can expect (I'm
just another fellow contributor to ninja).

--Ben

[jha...@gmail.com]

Hi,

On Wednesday, 14 April 2021 at 13:51:08 UTC+2 delorm...@gmail.com wrote:

I would love to have this feature on the official release of ninja. I can work more to make my implementation reach the quality to be integrated in master. I think the feature is more about discussing its interface and its design. The implementation itself is not so complicated. But I am lost about the contribution process.

Have you seen https://github.com/ninja-build/ninja/blob/master/CONTRIBUTING.md yet?

 

How to suggest on new feature?

Create an issue on GitHub, I will add the "feature" tag then.

 

Who does validate the design?

Anyone who likes to comment :)

 

Who take decision? etc.

You can spot users with commit rights by their tag "Collaborator" on GitHub.

[Marc Delorme]

Hi Jan,

Thank you for sharing the CONTRIBUTING.md.

I have created two PRs related to those incremental build issues:

• Improve incremental build: make ninja handle dynamic outputs (problem addressed is the current discussion)
• Improve incremental build: Make outputs modified outside of the build system considered dirty

they address two scenarios where running ninja does not give the same result as if you where running it after a clean build.

-Marc