Interest in new deps format

[Brett Wilson]

At Google we have quite a number of scripts that generate Ninja depfiles. These have proven difficult for a few reasons.

Background: Sometimes you don't know what inputs a build step depends on until you run it. A typical example is C/C++ includes. But we have a lot of scripts that do similar things. Ninja supports two formats: gcc and msvc. But both formats are crufty:

• For GCC format, the output is actually a makefile. Makefile escaping is slightly insane and it also requires that you declare the output file (technically unnecessary from Ninja's perspective) which must match exactly the output of the build step. This is surprisingly difficult and confusing, especially when there are multiple output files (which Make can't express).
• The MSVC format has a line prefix that's actually localized (!) which makes dealing with it difficult.

Long ago I wrote a patch (currently out-of-date I'm sure) to add a new "deps = list" format which is one-literal-file-name-per-line with no escaping, no prefixes, and no output file requirements. This is very easy to generate from a script. Unfortunately, the pull for this patch went nowhere.

This keeps popping up for our projects and I would like to see if there is enough interest in this to get this patch working again.

Thanks!

Brett

[Dirk Pranke]

There's a proposal for a portable dependency format from a couple of Kitware folks that might be of interest here:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1689r3.html

-- Dirk

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CABiGVV9-ez4NopZdhS9iOkyR2cHO51eyJztb41syYZo7m3mG0w%40mail.gmail.com.

[Brad King]

On Thu, Jun 10, 2021 at 4:01 PM Dirk Pranke wrote:
> There's a proposal for a portable dependency format from a couple of Kitware
> folks that might be of interest here:
>
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1689r3.html

I'm one of the authors of that paper. That format is for module dependencies
that must be discovered before compilation. It's not the same use case.
Also, the next revision of the paper, p1689r4, will drop the preprocessor
dependencies to keep the files small and since those dependencies can
already be handled via depfiles.

-Brad

[Dirk Pranke]

Thanks for clarifying. I thought you were on this list. I guess I misunderstood the proposal as I had thought also addressing header dependencies was an intended use case for the format.

-- Dirk 

[Brad King]

On Thu, Jun 10, 2021 at 4:12 PM Dirk Pranke wrote:
> I had thought also addressing header dependencies was an intended use case for the format.

For r4 we're dropping that for simplicity and to keep the module dependency
files small. When the collator reads them, it should not have to parse through
all the header files just to get the provided/required module names.

-Brad

[Brad King]

On Thu, Jun 10, 2021 at 1:47 PM Brett Wilson wrote:
> Ninja supports two formats: gcc and msvc.

For reference, there is a PR for a new MSVC format:

* https://github.com/ninja-build/ninja/pull/1812

It works with the `cl /sourceDependencies` flag.

MSBuild also has a "tracking log" format:

* https://docs.microsoft.com/en-us/visualstudio/extensibility/visual-cpp-project-extensibility?view=vs-2019#tlog-files

It is close to your proposed "deps = list" format, but has a small
amount of syntax to support multiple rules in one file.

-Brad

[Mathias Stearn]

On Thu, Jun 10, 2021 at 7:47 PM Brett Wilson <bre...@chromium.org> wrote:

At Google we have quite a number of scripts that generate Ninja depfiles. These have proven difficult for a few reasons.

Background: Sometimes you don't know what inputs a build step depends on until you run it. A typical example is C/C++ includes. But we have a lot of scripts that do similar things. Ninja supports two formats: gcc and msvc. But both formats are crufty:

• For GCC format, the output is actually a makefile. Makefile escaping is slightly insane and it also requires that you declare the output file (technically unnecessary from Ninja's perspective) which must match exactly the output of the build step. This is surprisingly difficult and confusing, especially when there are multiple output files (which Make can't express).
• The MSVC format has a line prefix that's actually localized (!) which makes dealing with it difficult.

I've actually found deps=msvc the simplest format to use for our custom build tools. While the fact that msvc outputs a localized prefix for this is clearly insane, it ends up being somewhat of a benefit because it means the ninja needs to support msvc_deps_prefix anyway. While the example in the documentation shows it as a global variable, it also works as a rule variable, and that is how I usually use it. I'll set "msvc_deps_prefix=opening file:" and then teach the tool to just print "opening file: /path/to/file" to stdout every time it opens a file. This ends up being significantly simpler than teaching tools to generate correctly escaped makefile format output.

That said, one thing I've found slightly annoying is that it only covers *inputs*. Some tools also have dynamically discovered outputs, but neither deps=msvc nor deps=gcc has a way to support runtime-discovered outputs. There is now the dyndeps system, but it only supports preflight processing, so it is cumbersome for usecases where it could just be done as part of running the tool, and sometimes requires duplicating work between the task and dyndeps discovery, wasting precious cycles during the build. Makefile syntax at least has a way to express it, even if ninja isn't able to use it. I suppose deps=msvc could allow something like "msvc_deps_output_prefix=writing file:" as a simple way to communicate both inputs and outputs.

 

Long ago I wrote a patch (currently out-of-date I'm sure) to add a new "deps = list" format which is one-literal-file-name-per-line with no escaping, no prefixes, and no output file requirements. This is very easy to generate from a script. Unfortunately, the pull for this patch went nowhere.

I don't see how this meaningfully improves on deps=msvc. Isn't it basically the same, just with an extra file to deal with (and an extra path to coordinate between the tool and build.ninja), rather than using stdout which is already hooked up everywhere?

 

This keeps popping up for our projects and I would like to see if there is enough interest in this to get this patch working again.

Thanks!

Brett

--

[Evan Martin]

On Mon, Jun 14, 2021 at 2:58 AM 'Mathias Stearn' via ninja-build <ninja...@googlegroups.com> wrote:

Long ago I wrote a patch (currently out-of-date I'm sure) to add a new "deps = list" format which is one-literal-file-name-per-line with no escaping, no prefixes, and no output file requirements. This is very easy to generate from a script. Unfortunately, the pull for this patch went nowhere.

I don't see how this meaningfully improves on deps=msvc. Isn't it basically the same, just with an extra file to deal with (and an extra path to coordinate between the tool and build.ninja), rather than using stdout which is already hooked up everywhere?

I think if I were to invent a custom mechanism for transporting information from a subprocess into the ninja process, I'd do it by a pipe and an environment variable.  E.g. ninja spawns the subprocess with fd 4 pointed at a pipe back to the ninja process and the environment variable "NINJA_CHANNEL=4".  However, I have no idea how such a thing works on Windows.  And also sometimes creating intermediate files can be useful for debugging purposes.  So maybe your/Brett's solutions are better...

[jha...@gmail.com]

On Monday, 14 June 2021 at 11:58:01 UTC+2 mat...@mongodb.com wrote:

That said, one thing I've found slightly annoying is that it only covers *inputs*. Some tools also have dynamically discovered outputs, but neither deps=msvc nor deps=gcc has a way to support runtime-discovered outputs. There is now the dyndeps system, but it only supports preflight processing, so it is cumbersome for usecases where it could just be done as part of running the tool, and sometimes requires duplicating work between the task and dyndeps discovery, wasting precious cycles during the build. Makefile syntax at least has a way to express it, even if ninja isn't able to use it. I suppose deps=msvc could allow something like "msvc_deps_output_prefix=writing file:" as a simple way to communicate both inputs and outputs.

There's a PR for dynamic outputs btw: https://github.com/ninja-build/ninja/pull/1953