Proposal to add a new --ninja-outputs-file option to `gn gen`

[David Turner]

I would like to add a new option to the `gn gen` command to generate a JSON file that maps GN labels to lists of Ninja target paths.

This will be used by Fuchsia to improve developer workflows, in particular supporting building targets directly by passing their GN label (e.g. `fx build <label>`) instead of their Ninja output paths, as well as improving certain error messages when build commands break.

The former is important for our long-term migration to Bazel, i.e. we want developers to not rely on Ninja-specific output paths, and in the future, `fx build` will also recognize Bazel target labels in addition to GN one.

A prototype has been implemented, see CL1 and CL2 on Gerrit.

The change does not introduce much complexity, nor does it modify GN behavior, or slows down `gn gen` when the option is not used. The generation takes only 400ms on a large Fuchsia build plan, and produces a file with a very simple schema, i.e.:

{
  "gn_label1": [ "path1", "path2" ],
  "gn_label2": [ "path3" ],
  ...
}

Which is trivial to process (separately) into a faster binary format, to make queries fast in both directions.

Alternatives considered were to rely on the `project.json` file but:

- Generating this file is far slower (up to 8 seconds on top of 13 seconds for regular options).
- The output file is 10x to 20x larger, and parsing it to extract target outputs is slow.
- It does not include all Ninja output paths (e.g. stamp files), that we need in our use case.

Let me know what you think and if you have objections.

- Digit

[Dirk Pranke]

I don't think I fully understand what you're looking for here (or perhaps why).

I understand wanting to be able to say `build <label>`. Is your idea that you would have `fx build <label>` actually call `ninja <output 1> <output 2> ... <output n>`?

Do you expect "gn_label1" to include the toolchain?

Does `gn outputs` give you the list of outputs you need for a given target? But I guess just calling `gn outputs out/default <label> | xargs ninja -C out/default ` or something similar is too slow?

-- Dirk

--
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[David Turner]

On Wed, Feb 21, 2024 at 3:38 PM Dirk Pranke <dpr...@google.com> wrote:

I don't think I fully understand what you're looking for here (or perhaps why).

I understand wanting to be able to say `build <label>`. Is your idea that you would have `fx build <label>` actually call `ninja <output 1> <output 2> ... <output n>`?

Yes, exactly.
 

Do you expect "gn_label1" to include the toolchain?

Not necessarily, as we will also support `fx build` command-line options to change the current GN toolchain suffix that will be applied to following toolchain-less labels, as well as support labels without an explicit name (e.g. `//foo` instead of `//foo:foo`)
So for example:

• `fx build --host //some/tool`
  would be equivalent to `fx build //some/tool:tool\(//build/toochain:host_x64\)`
  and would invoke `ninja tool_output1 ...`
  
  
• `fx build --toolchain=host //some/tool` and `fx build--toolchain=//build/toolchain:host_x64 //some/tool`
  would be equivalent to the previous example as well.
  
  
• `fx build --toolchain=//some:toolchain //some/other:tool`
  would be equivalent to `fx build //some/other:tool\(//some:toolchain\)`
  and would invoke `ninja other_tool_output...`
  
  
• `fx build --host //some/tool --default //another:target //and/another/one`
  would be equivalent to `fx build //some/tool\(//build/toolchain:host_x64\) //another:target //and/another/one:one`
  

Does `gn outputs` give you the list of outputs you need for a given target? But I guess just calling `gn outputs out/default <label> | xargs ninja -C out/default ` or something similar is too slow?

It does, except it does not list stamp files for phony aliases in the default toolchain (e.g. the Ninja ":default" and "default" target paths both correspond to the GN //:default label).
Also `gn outputs` takes 13s on my build plan, while I can get the same information in less than 60ms after converting the ninja_outputs.json file into a simple tabular text file (which is only needed once, and takes about 400ms).

See https://fxrev.dev/994772 for a prototype with multiple backends, so far the simple tabular format beats using a SQlite3 database.

We also want to continue supporting `fx build <ninja_path>` but then print a warning instructing developers which correct GN label to use instead.

Finally, there are cases where we want to print GN labels that correspond to Ninja output paths from action scripts to improve error messages.

All of this requires blazing fast GN label / Ninja path conversions.

Hope this helps,

[Dirk Pranke]

Okay. I mean, I think it's obvious that reading a simple map from a file would be a lot faster than processing the whole build graph (which `gn outputs` has to do) :).

I think this idea is fine. There are places in Chromium where we maintain a manual mapping of some of the targets and I've often thought about getting that directly from GN, so we'd probably use this, too.

I'd definitely like to review the CL(s) when they're ready.

-- Dirk

[Fumitoshi Ukai (鵜飼文敏)]

Doesn't gn already generate gn target as phony target?