toolchain_args and non-default toolchains

[Nico Weber]

Hi,

I'm either confused about toolchain_args, or think about it the wrong way, or have issues with either its usability or with the usability of forward_variables_from(). Please help me figure out which one it is :)

I have a build that has a normal host toolchain used to build clang, and then a second toolchain that uses the just-built clang to build something else.

There's a gn arg "clang_base_path" that, when set, tells the host toolchain to use the clang at that path as compiler. The build sets a few other args (is_clang, ...) depending on if this arg is set.

For the second toolchain, I figured it'd be a good idea to do `toolchain_args = { clang_base_path = "." }` since that way the other args get a good default. The full setup looks roughly like so:

template("unix_toolchain") {
  toolchain(target_name) {
    cc = "cc"
    if (clang_base_path != "") {
      cc = "$clang_base_path/bin/clang"
    }
    tool("cc") {
      command = "$cc -o {{output}} -c {{source}}
      outputs = [ "{{source_out_dir}}/{{label_name}}.{{source_name_part}}.o" ]
    }

  }

}

unix_toolchain("host_default") {

  toolchain_args = {
    current_os = host_os
    current_cpu = host_cpu
  }
}

unix_toolchain("stage2_toolchain") {

  toolchain_args = {

    clang_base_path = "."
    current_os = host_os
    current_cpu = host_cpu
  }
  deps = [  "//:clang($host_toolchain)" ]
}

Now, for stage2_toolchain, clang_base_path is not set in the unix_toolchain template, but it is set in the normal build files that are evaluated in the context of stage2_toolchain. I'm guessing the reason for that is that stage2_toolchain is defined in the default toolchain, not in itself, so the toolchain_args haven't had an effect yet.

Is that correct? Is that expected? Is that ever what we want? Should secondary toolchains implicitly forward all variables from toolchain_args into the definition of the toolchain, just like they forward the toolchain_args to targets using that toolchain?

Assuming this is correct and intentional, I figured I'd do the forwarding of toolchain_args myself, by adding this in unix_toolchain:

    forward_variables_from(invoker.toolchain_args, "*")

But, unix_toolchain doesn't use current_os or current_cpu as variable, so I get an error about stage2_toolchain setting these in toolchain_args not having an effect.

So instead I could do:

    forward_variables_from(invoker.toolchain_args, ["clang_base_path"])

There are two problems with this:

1. forward_variables_from() has different semantics when forwarding "*" then when forwarding an explicit list. "*" is allowed to clobber existing variables, but the explicit list doesn't have this permission and errors on clobbers. Since clang_base_path is a declare_args, this does clobber and I get a warning. Why do explicit lists have this error? It makes forwarding things other than * fairly verbose and repetitive.

2. I need to explicitly list all toolchain_args I used directly in the toolchain definition, and if I add a use of another toolchain_arg later I need to remember to update this list, else I silently get the default toolchain value, not the secondary toolchain value.

1 can be fixed by doing this instead:

  invoker_toolchain = invoker.toolchain_args

  if (defined(invoker_toolchain.clang_base_path) {

    clang_base_path = invoker_toolchain.clang_base_path

  }

This works! But it doesn't help with 2, and I need to repeat "clang_base_path" 3 times, and I need this for every toolchain_arg that I use directly in the toolchain definition. (In my real project, there's more than just 1.)

Yet another approach would be to set clang_base_path in stage2_toolchain not as a toolchain_arg but as a normal variable, do

    forward_variables_from(invoker, "*", ["toolchain_args"])

in unix_toolchain, and then also in unix_toolchain copy it into toolchain_args. But this runs into the same forward_variables_from() explicit-list ugliness mentioned above since I have to define toolchain_args in unix_toolchain like so:

    toolchain_args = {
      forward_variables_from(invoker.toolchain_args, "*")
      if (defined(invoker.clang_base_path)) {
        clang_base_path = invoker.clang_base_path
      }
    }

Here the failure mode for 2 (if I add a use of a new variable in unix_toolchain) is that I forget to update toolchain_args instead, which means I always use the right value of the variable when defining the toolchain, but if I forget to update the list then I'll get the wrong value in the targets defined with this toolchain.

This seems the least bad approach I can currently use. Maybe I missed some other alternative?

This would be easier if either of (in order of preference):

1. toolchain() behaved as if toolchain_args are defined as variables in the toolchain automatically

2. forward_variables_from(invoker.toolchain_args, "*") wouldn't cause unused variable errors for variables from toolchain_args (since these arguably have a use through the toolchain_args already)

3. forward_variables_from(..., [list]) could clobber values (but this is worse than 1 and 2 since it doesn't help with problem 2 above)

Thanks,

Nico

[Nico Weber]

I found a solution I'm fairly happy with. If I replace

    forward_variables_from(invoker.toolchain_args, "*")

with

    forward_variables_from(invoker.toolchain_args, "*")

    not_needed("*")

then things work pretty well and it solves the two problems I listed.

So thanks for listening :) As often, just writing down everything helped.

The question if toolchain() should behave like this by default remains.

If anyone's interested, here's what this became: https://github.com/llvm/llvm-project/commit/05c5ff5ab0b4d7cec25210bce7444827c25d71b1

[Brett Wilson]

Yes, all toolchains are defined in the default toolchain. This is due to the chicken-and-egg problem of the toolchain definition itself defining the arguments for that toolchain (required for BUILDCONFIG, etc). We can't instantiate the other toolchain until you know what it means. The default toolchain is the only one where you can instantiate stuff without first seeing the toolchain definition for it.

Brett

[Nico Weber]

Right, I guess my question now is: Since toolchain_args are defined for all targets built in that toolchain, should they be defined in the definition of that toolchain too? It seems like it might lead to less surprising behavior.

[Brett Wilson]

On Sun, Jul 5, 2020 at 9:24 AM Nico Weber <tha...@chromium.org> wrote:

Right, I guess my question now is: Since toolchain_args are defined for all targets built in that toolchain, should they be defined in the definition of that toolchain too? It seems like it might lead to less surprising behavior.

I guess I'm confused. How can the toolchain args be defined in a toolchain before we know what that toolchain is? Can you give a more concrete example?

Brett 

[Nico Weber]

The example is in the first email in this thread. I guess my confusion was that all toolchains I know are from a template() and the caller of the template sets toolchain_args, but there's nothing stopping the toolchain from adding stuff to toolchain_args instead of just using the one from invoker, and then the toolchain could set them at the end of the toolchain definition. I now understand why it's not even clear what I proposed, thanks :)

At least in Chromium, no toolchain sets toolchain_args directly though. It's always set passed to a template that then defines a toolchain. So it'd be something like "if a toolchain is in a template and the template only reads toolchain_args off invoker, then when a user of the template passes toolchain_args, its values are transparently copied into normal variables in the scope of the toolchain", but that's admittedly overly magic.

Thanks again for talking me through this :)

[David Turner]

Regarding the current_os / current_cpu variables in toolchain_args, we "solved" this in the Fuchsia build by having our toolchain-defining templates take "toolchain_os" and "toolchain_cpu" parameters. Inside the template, we set toolchain_args.current_os and toolchain_args.current_cpu to invoker.toolchain_cpu and invoker.toolchain_os, respectively, after forwarding the variables from the toolchain_args parameter.

Another surprising thing that you may find useful is that "current_toolchain" is never defined in BUILDCONFIG.gn, which makes it a bit difficult to detect reliably in which toolchain we are when parsing this file. We rely on each toolchain() invocation to set specific toolchain_args variables properly instead, which is a little fragile, but works.

--
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.