How to use Transitions to create a platform agnostic rule?

[Konstantin]

We have some rules ("codegen") which internally run Python scripts and the output of those rules is platform agnostic (the same for any target platform). Right now the output of codegen rules goes to the platform specific output folder, such as x64_windows_opt and running for different target platforms creates separate copies of the output. I need to fix that.

I think the answer here is Transitions, but I could not make it work.

We need to make codegen rules spit output to platform agnostic folder and express in Starlark that any rule consuming codegen outputs should look in that folder.

Could somebody please help with the code snippet for such platform agnostic rules?

Thank you!

Konstantin

[Alex Humesky]

The first thing that comes to mind is to create a generic target platform (e.g. doesn't care about cpu architecture, os, etc), and use an "Incoming edge transition" on your codegen rules to transition to that generic target platform:
https://docs.bazel.build/versions/main/skylark/config.html#incoming-edge-transitions

(and you can still use execution platforms so that the codegen tools select the right python binary for where they're going to run)

(Unfortunately I won't have time today to create a minimal prototype to verify whether what I just recommended actually works :) Hopefully that points you in the right direction though)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/870767c2-281e-4288-ae75-a463d76e9a5an%40googlegroups.com.

[Konstantin]

Like this?

def _host_impl(settings, attr):

    return {"//command_line_option:platforms": ["@local_config_platform//:host"]}

host_transition = transition(

    implementation = _host_impl,

    inputs = [],

    outputs = ["//command_line_option:platforms"],

)

[Alex Humesky]

I believe you would need to define a new platform:

https://docs.bazel.build/versions/main/platforms.html#defining-constraints-and-platforms

Is this the same problem as was discussed in this thread?

https://groups.google.com/g/bazel-discuss/c/eWRINSAQHks/m/4sE943NTAQAJ

Or are there some new differences?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/198d561d-9d23-4e1b-9e42-897c8baac448n%40googlegroups.com.

[Konstantin]

It is the same problem, yes. I have to get back to it because we never dot it working.

[Konstantin]

Why do I need completely custom platform? Can't I coerce everything to "host" with say "opt"?

[Konstantin]

You propose to define a platform without constraints, like this?

platform(

    name = "any",

)

[Konstantin]

And then use it like this:

def _any_impl(settings, attr):

    return {

        "//command_line_option:platforms": ["@tab_toolchains//bazel/platforms:any"],

        "//command_line_option:compilation_mode": "opt",

    }

any_transition = transition(

    implementation = _any_impl,

    inputs = [],

    outputs = [

        "//command_line_option:platforms",

        "//command_line_option:compilation_mode",

    ],

)

[Alex Humesky]

Yes that looks like what I had in mind.

> Why do I need completely custom platform? Can't I coerce everything to "host" with say "opt"?

I believe the advantage would be that then it doesn't matter whether you're running bazel from a linux, mac, windows, x86, arm, etc etc machine. I believe that if you use "host" as the target platform for the codegen, then if you run bazel from a mac, bazel will consider the outputs different from those if you ran bazel from a linux machine. (also, "host" is basically going away, since for most things (especially rules) "exec" is more correct)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7f3b088c-cfd2-4303-8ea3-4858999112dcn%40googlegroups.com.

[Konstantin]

I see the motivation for "any" platform and tried to use this:

def _any_impl(settings, attr):

    return {

        "//command_line_option:platforms": ["@tab_toolchains//bazel/platforms:any"],

        "//command_line_option:compilation_mode": "opt",

    }

any_transition = transition(

    implementation = _any_impl,

    inputs = [],

    outputs = [

        "//command_line_option:platforms",

        "//command_line_option:compilation_mode",

    ],

)

But now I get this error all over:

ERROR: C:/_/z2couxfh/external/tab_toolchains/bazel/macros/BUILD.bazel:21:10: While resolving toolchains for target @tab_toolchains//bazel/macros:true: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.

Confused.

[Alex Humesky]

My guess is that there is a tool dependency somewhere that needs to be in the "exec" configuration

You might try adding --toolchain_resolution_debug=".*" to see what bazel does to select toolchains

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/62d1a534-7aab-4029-a0c6-053cc0fa6bc5n%40googlegroups.com.

[Konstantin]

After all I found one specific codegen type which actually depends on the platform specific targets and that was causing the error above.

This leads me to the next logical question: transitions can read current configuration and change it. But can it store current configuration somewhere for the subsequent transitions to restore it?1 I want inbound transition to change the platform to "any" and then outboard transition to change it back to what it was.

[Fabian Meumertzheim]

You could use a string_list_setting from skylib to write the original
platform into (in your first transition) and reset the platform to its
value (in your second transition).
Keep in mind though that the final configuration after both
transitions will not be fully identical to the original configuration
as Bazel (currently) tracks the set of configuration options that have
been modified by transitions somewhere along the dependency chain. As
a result, you may see "unnecessary" rebuilds for now, but the
situation should improve soon (see
https://github.com/bazelbuild/bazel/issues/14023).

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/aee51f67-bb9f-45ee-93cb-037bd35e4ad6n%40googlegroups.com.

[Konstantin]

Thank you, Fabian, for the very informative answer and referencing the relevant issue!

I think I almost understand the part of your answer regarding " string_list_setting", but not quite yet. All examples I could find show how to get the value of the settings, but I don't see how to SET the value programmatically. If you have time could you please provide a few lines snippet?

[Fabian Meumertzheim]

I'm actually using this pattern in rules_jni: https://github.com/fmeum/rules_jni/blob/42f4f91c364ce545bc57023992baa0a2110483b2/jni/internal/transitions.bzl (see also the BUILD file in that package)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a20392f8-acca-4c10-a5c8-1afad039fe35n%40googlegroups.com.

[Konstantin]

Fabian, I am still looking at your repo, but one problem really bothers me - now when the output folders are named as "x64_windows-opt-ST-0343f2a5cbb9" with some hash at the end HOW we supposed to find the build artifacts? After my experiments with transitions I got a whole bunch of such folder and don't know which is which. But more importantly how the post-build step can find out where to collect the build artifacts from?

[Konstantin]

I found this and it would fit nicely, but it seems  //command_line_option:output directory name was removed since then and I don't see a replacement.

[Fabian Meumertzheim]

Generally speaking, the rule that applies the transition should
package up the build artifacts - it has access to their DefaultInfo
and thus doesn't need to do any path guessing. rules_jni also contains
an example of a rule that applies a (split) transition on platforms
and then packages up the resulting binaries into something that's
available in the top-level config:
https://github.com/fmeum/rules_jni/blob/42f4f91c364ce545bc57023992baa0a2110483b2/jni/internal/cc_jni_library.bzl#L101

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/974e8ee1-6e1d-49e3-b83e-aa2b2f061c70n%40googlegroups.com.

[Konstantin]

I think I understand now. Thank you, Fabian! Your examples are very helpful!