get_target_outputs across toolchains?

[Charles Nicholson]

I have multiple toolchains in my build, since my products include both native host tools as well as firmware targets.

I have a host toolchain that I use for host compilation but also for actions that generate files: bin2c kinds of things, protoc kinds of things, etc.

I enforce that the current_toolchain is equal to my host toolchain, so that I don't waste build time doing identical work across multiple toolchains.

These generated files are then added to static library "sources" lists.

When there's only one toolchain, it's very easy to just do this:

======

my_action("foo") {

  sources = [ "blah.foo" ] # outputs "${target_gen_dir}/blah.c"

}

static_library("bar") {

  sources = [ "bar.c" ] + get_target_outputs(":foo")

  deps = [ ":foo" ]

}

=======

Moving to multiple toolchains like this angers GN:

======

if (current_toolchain == my_host_toolchain) {

  my_action("foo") {

    sources = [ "blah.foo" ] # outputs "${target_gen_dir}/blah.c"

  }

}

static_library("bar") {

  sources = [ "bar.c" ] + get_target_outputs(":foo(${my_host_toolchain})")

  deps = [ ":foo(${my_host_toolchain)" ]

}

======

get_target_outputs() does not appear to work across toolchains, even in the same BUILD.gn file.

Ideally it would be nice for leaf builds to not have to worry about the names + extensions of the files generated by "my_action"; those are pleasantly handled by get_target_outputs. That doesn't seem possible in this case, though.

Are there any elegant ways to handle this? My inclination is to just create a list containing the inputs, and explicitly transform that into the outputs manually to match the action i'm invoking, but that feels a bit clunky.

Thanks,

Charles

[Dirk Pranke]

On Thu, Jan 20, 2022 at 2:24 PM Charles Nicholson <charles....@gmail.com> wrote:

I have multiple toolchains in my build, since my products include both native host tools as well as firmware targets.

I have a host toolchain that I use for host compilation but also for actions that generate files: bin2c kinds of things, protoc kinds of things, etc.

I enforce that the current_toolchain is equal to my host toolchain, so that I don't waste build time doing identical work across multiple toolchains.

These generated files are then added to static library "sources" lists.

When there's only one toolchain, it's very easy to just do this:

======

my_action("foo") {

  sources = [ "blah.foo" ] # outputs "${target_gen_dir}/blah.c"

}

static_library("bar") {

  sources = [ "bar.c" ] + get_target_outputs(":foo")

  deps = [ ":foo" ]

}

=======

Moving to multiple toolchains like this angers GN:

======

if (current_toolchain == my_host_toolchain) {

  my_action("foo") {

    sources = [ "blah.foo" ] # outputs "${target_gen_dir}/blah.c"

  }

}

static_library("bar") {

  sources = [ "bar.c" ] + get_target_outputs(":foo(${my_host_toolchain})")

  deps = [ ":foo(${my_host_toolchain)" ]

}

======

get_target_outputs() does not appear to work across toolchains, even in the same BUILD.gn file.

That sounds like a bug we should fix to me :).

-- Dirk

 

Ideally it would be nice for leaf builds to not have to worry about the names + extensions of the files generated by "my_action"; those are pleasantly handled by get_target_outputs. That doesn't seem possible in this case, though.

Are there any elegant ways to handle this? My inclination is to just create a list containing the inputs, and explicitly transform that into the outputs manually to match the action i'm invoking, but that feels a bit clunky.

Thanks,

Charles

--
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[Roland McGrath]

get_target_outputs cannot work across toolchains for the same reasons it cannot work for toolchain-tool-based targets or across BUILD.gn files.  All of these cases are equivalent to "across BUILD.gn" files for purposes of the GN evaluation model.  The only reason get_target_outputs works at all is because it knows exactly what you just defined in the same file, and that's why it only works on targets earlier in the same file.  Each separate evaluation of a BUILD.gn file in each toolchain is a wholly separate thing (only the parsing is reused, nothing else).  So it has no way to know what a different toolchain's version of some target would have done, or even if this file evaluated in that toolchain's context will actually define a target by that name at all.  I am not especially happy about this, but it's a deep and fundamental design choice in GN made from the start.

[Charles Nicholson]

Thanks, Roland, that was my recollection from my Alphabet days :)

I'm curious if any elegant-enough patterns have emerge to work around this, or if the action input / output mapping needs to be re-expressed into every leaf. 

Thanks,

Charles

[Roland McGrath]

Basically if you actually need GN to properly know the inputs, there isn't anything to avoid doubly-specifying things.  The best recommendation is to use templates such that the switching into another toolchain and defining the action is always paired with template code that defines the only thing that has to know the precise file names the action produces.  Then it has to be manual between the two evaluations, but it's all inside code from the one template (evaluated in both places) so it can be made to match without a lot of effort.

The only other general solution to related issues is metadata.  But that only suffices if you don't need GN to know about the inputs, or even for Ninja to know directly.  All you can do is use `generated_file` to produce a response file, JSON file, or whatever you like at `gn gen` time that some build action reads to decide what files to actually read, and generates a depfile to tell Ninja after the fact so it knows what incremental rebuilds are required next time around.  For something like compilation, this means all you can do is generate a file of includes and then compile that generated file as one big source file that indirectly uses the other sources.  But that's often inadequate for what one is trying to do.  As was discussed here earlier, there is no solution at all for dynamic outputs, only these very partial solutions for dynamic inputs.  

[Charles Nicholson]

Thanks for the response, Roland. I apologize for not fully understanding what you're suggesting, though.

I like pushing both the host- and non-host-toolchain expressions into the same template gni, but if you're suggesting that the non-host-toolchain flavor somehow call across into the host-toolchain flavor, I'm not sure how to do that.

Could I ask you for a trivial example of what you're suggesting please? I'd appreciate it greatly :)

Thanks,

Charles

[Dirk Pranke]

Yeah, I guess you're right, now that I think about it more. I was thinking initially that you didn't necessarily have to re-evaluate the target in the right context to know the full set of outputs, but on reflection that's obviously not true, because different toolchains can have different GN args that can radically affect what the outputs are.

You are correct that processing everything independently is a core part of the GN architecture, and it's part of the reason why GN can be as fast as it is.

I don't think it's something we couldn't fix or change if we wanted to, but doing so would fundamentally be a big change to GN that would slow it down (by adding dataflow dependencies between parallel tasks that don't exist now) and so it's debatable whether we *should* support it, versus doing the things like you suggest (explicitly/doubly-specifying files to simply not have this problem at the expense of some redundancy and fragility).

-- Dirk

[David Turner]

Le ven. 21 janv. 2022 à 01:07, Charles Nicholson <charles....@gmail.com> a écrit :

Thanks for the response, Roland. I apologize for not fully understanding what you're suggesting, though.

I like pushing both the host- and non-host-toolchain expressions into the same template gni, but if you're suggesting that the non-host-toolchain flavor somehow call across into the host-toolchain flavor, I'm not sure how to do that.

What Roland is saying is that your rules should use a set of conventions to let a target guess the location of another target's output when it is in a different toolchain.

 

Could I ask you for a trivial example of what you're suggesting please? I'd appreciate it greatly :)

In your specific case, this would be something like:

if (current_toolchain == my_host_toolchain) {

  my_action("foo") {

    sources = [ "blah.foo" ] # outputs "${target_gen_dir}/blah.c"

  }

}

static_library("bar") {

  deps = [ ":foo(${my_host_toolchain)" ]

  sources = [ "bar.c" ] + get_label_info(deps[0], "target_gen_dir") + "/blah.c"

}

As you see, the `bar` target needs to know the path and name of the output file for `foo`, doing a non-obvious computation for its directory, and knowing its output name.

Using templates to mask some of these computations can help make your code clearer. For a concrete example, see the compiled_action() template used by the Fuchsia, which guesses the path of a host executable to invoke it in an action.


 

[Roland McGrath]

No, I'm just suggesting that when you're backed into the corner of "the code in one toolchain has to know what the code in the other toolchain did", then writing a single template that generates the code for both toolchains using some shared variables is a good way to make it easier to maintain.  It's not any different from what your starting code looks like, except that you stuff the unsightly parts together into a template rather than just being next to each other in the BUILD.gn file.  And if it's implementing a recurring pattern in your build, you might be able to reuse the template to get the fiddly bits right in only one place.  The key feature you can use in code like that, which appears to "work across toolchains", is `get_label_info(":label($other_toolchain)", "target_gen_dir")` and such, which resolves those "global" variables in the "toolchain context", because they are really just derived from the toolchain's label and don't actually require having evaluated the toolchain definition.

[Wyatt Hepler]

Hi Charles!

Something akin to CMake's generator expressions would be very helpful here. You could imagine GN emitting a stand-in expression for the dependency (e.g. "$<TARGET_OUTPUTS(:foo(${my_host_toolchain}))", to use pseudo CMake), then evaluating those expressions in a separate pass after gn gen. All of the information you need is expressed in Ninja by that point. That wouldn't require any fundamental changes to GN's design, just an additional pass.

Some may cringe to hear this, but Pigweed already does something similar. Pigweed's pw_python_action wrapper can resolve cross-toolchain dependencies to files in script arguments (docs). It uses the generated Ninja files to expand stand-in expressions, and then invokes the script. This feature is essential for Pigweed to operate as a middleware in cross-platform, many-toolchain builds.

Best,
Wyatt

[Roland McGrath]

I don't want to think about the thing you described Pigweed doing.  But I do think that a GN integration with Ninja's dyndeps feature could be a way to support some of the cases that are hard.  I'm just not sure what it would look like or how many of the cases it would actually cover.

[Charles Nicholson]

Thinking this over some more, I'm starting to like the explicitness of doing the mapping in the BUILD.gn file:

======================================

if (current_toolchain == my_host_toolchain) {

  action("foo") {

    sources = [ "blah.foo" ] # emits blah.c

  }

}

static_library("bar") {

  sources = [ "bar.c", "${target_gen_dir}/blah.c" ]

  deps = [ ":foo(${my_host_toolchain})" ]

}

=======================================

I guess in the same way that you (obviously) need to know the path of a generated .h file so that you can write the correct #include directive, it makes sense that you need to explicitly know the path to the generated .c file in this case.

It's a good argument for avoiding complexities in your generated file paths, though, and just preferring target_gen_dir as often as possible.

It would be a really powerful feature, though, if get_target_outputs() worked on _anything_. I'd personally love to see that investment made, even if it's not easy.

Charles

[Roland McGrath]

Note that in this example you need to use `gen_dir = get_label_info(":foo($my_host_toolchain)", "target_gen_dir")` and then `${gen_dir}` in place of `${target_gen_dir}` .

[Charles Nicholson]

Yes, thanks :) (I realized that very quickly after trying it out, ha).