gni help: Basing outputs on read_file()

[Ian Barkley-Yeung]

So I want to have a GN template that determines its outputs based on reading a file. Is this possible? If so, what am I doing wrong?

For the sake of a simple test, I have a file, step_one.json, that lists a few files:

iby@iby3:/ssd/iby/chromium/src$ more /ssd/iby/chromium/src/out/Debug/gen/chrome/browser/step_one.json

["/ssd/iby/chromium/src/out/Debug/a.txt", "/ssd/iby/chromium/src/out/Debug/b.txt", "/ssd/iby/chromium/src/out/Debug/c.txt"]

I then create a simple template in step_two.gni:

import("//build/config/chrome_build.gni")

template("step_two") {

  action(target_name) {

    script = "//build/util/step_two.py"

    args = ["--file_list", rebase_path(invoker.filelist, root_build_dir)]

    inputs = [invoker.filelist]

    outputs = []

    foreach(filename, read_file(invoker.filelist, "json")) {

      outputs += [filename]

    }

    forward_variables_from(invoker,

                           [

                             "deps",

                             "public_deps",

                           ])

  }

}

and try to use it in a BUILD.gn file:

step_two("second_step") {

  filelist = "$target_gen_dir/step_one.json"

}

Inevitably, I get an error like:

iby@iby3:/ssd/iby/chromium/src$  autoninja -C out/Debug/ chrome

ninja: Entering directory `out/Debug/'

[0/1] Regenerating ninja files

ERROR at //build/util/step_two.gni:13:19: File is not inside output directory.

      outputs += [filename]

                  ^-------

The given file should be in the output directory. Normally you would specify

"$target_out_dir/foo" or "$target_gen_dir/foo". I interpreted this as

"/ssd/iby/chromium/src/out/Debug/a.txt".

See //chrome/browser/BUILD.gn:132:1: whence it was called.

step_two("second_step") {

^------------------------

See //chrome/browser/media/router/BUILD.gn:80:7: which caused the file to be included.

      "//chrome/browser:browser_process",

      ^---------------------------------

FAILED: build.ninja 

../../buildtools/linux64/gn --root=../.. -q --regeneration gen .

ninja: error: rebuilding 'build.ninja': subcommand failed

The error seems very strange. "/ssd/iby/chromium/src/out/Debug/a.txt" is inside the output directory, so why does it keep complaining that it's not? 

I've also tried having a relative path, with no better luck:

iby@iby3:/ssd/iby/chromium/src$ more /ssd/iby/chromium/src/out/Debug/gen/chrome/browser/step_one.json

["out/Debug/a.txt", "out/Debug/b.txt", "out/Debug/c.txt"]

iby@iby3:/ssd/iby/chromium/src$  autoninja -C out/Debug/ chrome

ninja: Entering directory `out/Debug/'

[0/1] Regenerating ninja files

ERROR at //build/util/step_two.gni:13:19: File is not inside output directory.

      outputs += [filename]

                  ^-------

The given file should be in the output directory. Normally you would specify

"$target_out_dir/foo" or "$target_gen_dir/foo". I interpreted this as

"out/Debug/a.txt".

See //chrome/browser/BUILD.gn:132:1: whence it was called.

step_two("second_step") {

^------------------------

See //chrome/browser/media/router/BUILD.gn:80:7: which caused the file to be included.

      "//chrome/browser:browser_process",

      ^---------------------------------

FAILED: build.ninja 

../../buildtools/linux64/gn --root=../.. -q --regeneration gen .

ninja: error: rebuilding 'build.ninja': subcommand failed

And a whole bunch of other variations.

Is this possible? If so, what am I doing wrong? 

Thanks in advance for any help,

Ian

[Brett Wilson]

It looks like your output files look like "out/Debug/a.txt". But when you assign this to the outputs variable in GN, it treats the files as relative to the current directory. So you're declaring an output of "//chrome/browser/out/Debug/a.txt"

I would recommend having your script always take and output files relative to its current directory which is the build directory. You'll need a rebase_path in there to make it relative to the current directory (or absolute starting with "//").

Brett

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/d6c27230-9339-4fb7-bce2-0f7926be59can%40chromium.org.

[Takuto Ikuta]

I guess using read_file for the generated file won't work as you intended.

It reads the file before running any build action, so clean build fails to find the file.

Or is this only for your local development?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABiGVV93e_f%3DuqKXwWHrXF4SNvoV6YKx3XoKozHKRrR5uRV8vA%40mail.gmail.com.

[Ian Barkley-Yeung]

I guess using read_file for the generated file won't work as you intended.
It reads the file before running any build action, so clean build fails to find the file.
Or is this only for your local development?

This is for a refactor based on the comment chain in https://chromium-review.googlesource.com/c/chromium/src/+/2951165/1/tools/grit/preprocess_if_expr.py. My reviewers want me to break up a single build action into two build actions. 

I had very similar concerns that this isn't possible, and my goal is to make a simple test case to prove this is or isn't possible. However, since I couldn't get it to work at all, even in the "file already exists" case, it was very hard for me to test if it worked in the "file needs to be generated" case.

[Sylvain Defresne]

The reason the check fails is that gn compares the paths read from the file which are absolute (e.g. "/ssd/iby/chromium/src/out/Debug/a.txt") to the value of root_build_dir which is relative to the checkout root (e.g. "//out"). The comparison is a simple string comparison (see EnsureStringIsInOutputDir implementation). This is why the comparison fails. You want your tool to generate a .json file that contains relative paths (i.e. "//out/Debug/a.txt", ...).

However, I'm not sure whether you can generate the filelist at the correct time. My understanding is that you'll generate the filelist in a `step_one` target that uses exec_script to create the file. I'm not sure if gn promises that the exec_script will be executed before the read_file.

-- Sylvain

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPHpbpQx_vN8r2isgZuoLsSDgVQTX-9ozfyx3eKF8XfSsygXtA%40mail.gmail.com.

[Brett Wilson]

On Tue, Jun 15, 2021 at 2:55 AM Sylvain Defresne <sdef...@chromium.org> wrote:

The reason the check fails is that gn compares the paths read from the file which are absolute (e.g. "/ssd/iby/chromium/src/out/Debug/a.txt") to the value of root_build_dir which is relative to the checkout root (e.g. "//out"). The comparison is a simple string comparison (see EnsureStringIsInOutputDir implementation). This is why the comparison fails. You want your tool to generate a .json file that contains relative paths (i.e. "//out/Debug/a.txt", ...).

I would recommend returning files relative to the root_build_dir (the current directory of your script, i.e. "a.txt") and then on that output in GN do:

  rebase_path(files_you_read, ".", root_build_dir)

to make them relative to the current directory.

However, I'm not sure whether you can generate the filelist at the correct time. My understanding is that you'll generate the filelist in a `step_one` target that uses exec_script to create the file. I'm not sure if gn promises that the exec_script will be executed before the read_file. 

I'm not following when these things are executed, can we clarify this? exec_script is strongly discouraged because it affects GN execution performance, and you'll need a toplevel owners review to get on the allowlist.

In the original message nothing was mentioned about exec script. This make me thing the file was generated by an action. If true, that happens at build time, not GN run time, which will be after any read_files are executed.

Brett

[Sylvain Defresne]

I agree.

My understanding is that this would not be possible with action because they are executed after gn is done running, only with exec_script. I assumed the original poster was planning to use exec_script since action cannot work as far as I understand gn+ninja execution model.

 

Brett

[Joe Mason]

I think the issue here is that the original poster wants to use the output of the script to calculate the dependencies of another build step:

Step 1 outputs step_one.json

Step 2 outputs [variable list of files, based on contents of step_one.json]

Other steps may depend on step 2 and would need to know the list of outputs to work out the dependency graph. And I think step 2 itself would need to know its own list of outputs so it can check if they're all up-to-date when building the dependency graph.

So the read_file would need to happen before building the dependency graph, not just before running the action. (Maybe that actually already works, but I'll assume it doesn't.)

One solution is to add an extra fixed output to step 2, which gets stamped whenever it runs. That way the dependency graph just uses that one fake output instead of looking at the actual files that are written. I think you'd also need to run step 2 unconditionally even if step_one.json hasn't changed, so that if one of its outputs gets deleted by an external process it will be regenerated.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABiGVV-uM_HPr8nu6icWprtyYfb4B-A2RXm8UGSxQTJa9rcghQ%40mail.gmail.com.