Collecting the artifacts produced by multi-platform build.

[Konstantin]

We have Bazel build which supposed to run for at least four different target platforms. When we build them sequentially the build artifacts end up in folders like "x64_windows-opt", "x86_windows-opt", etc. Those folder names are predictable and allow the installer to collect the artifacts for all platforms from well known folders.

Of course we want to build multiple platforms in parallel, or at the very least build dbg and opt in parallel. To make it happen we use one to many configuration transition, for example:

def _platform_expander_transition_impl(settings, attr):

    return [

        {"//command_line_option:compilation_mode": "opt", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:x64_windows"},

        # {"//command_line_option:compilation_mode": "dbg", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:x64_windows"},

        {"//command_line_option:compilation_mode": "opt", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:x86_windows"},

        # {"//command_line_option:compilation_mode": "dbg", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:x86_windows"},

        # {"//command_line_option:compilation_mode": "opt", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:emscripten"},

        # {"//command_line_option:compilation_mode": "dbg", "//command_line_option:platforms": "@tab_toolchains//bazel/platforms:emscripten"},

    ]

platform_expander_transition = transition(

    implementation = _platform_expander_transition_impl,

    inputs = [],

    outputs = [

        "//command_line_option:compilation_mode",

        "//command_line_option:platforms",

    ],

)

When this transition is applied all configurations are indeed built in parallel. Great!

But here comes a little catch: output folders now named as "x86_windows-opt-ST-5ac270887399 ",

i.e. old name + "-ST-" + some hash. What I need to figure out is how to find (query, discover, guess)

those new output folder names to let the installer collect the artifacts.

Any advice?

Konstantin

[Konstantin]

The naïve trivial solution I had in mind was to look for the folders with well known prefixes, i.e. we don't know exact name for x86/opt folder, but it is probably the one which starts with "x86_windows-opt". Unfortunately there are other transitions in the project and therefore we may end up with multiple output folders all starting with " x86_windows-opt". Of course they have different hash part, but I know no way to figure which is which.

[Alex Humesky]

So the most robust way to do this would seem to be to build the installer with bazel, and have the installer rule depend on the artifacts via the transition, so that it gets all the different versions of the artifacts. It mostly works, but it looks like there's a missing piece. There doesn't seem to be a built-in way for the Starlark rule that receives the different versions of the artifacts to inspect the configuration that the artifacts were built in. (This was discussed some time ago here too: https://groups.google.com/g/bazel-discuss/c/H8lS3JL2jt8/m/pUGXDhpXBAAJ)

I think there's a workaround though, which is to create a little rule that gathers the relevant configuration information and puts that in a provider ("config_hints" below). Then have a dependency on a target of that config info rule via an implicit attribute with the same split transition as the deps of the installer rule. Then that config info target will be evaluated in the configuration of each split (accessible via ctx.split_attr), and its provider will give the relevant configuration information for that split (the keys of the dict ctx.split_attr.<attrname> should match up to other split_attr attributes with the same transition).

For example:

BUILD:

load(":defs.bzl", "installer", "flag", "config_hints", "genfoo")

cc_binary(
  name = "bin_a",
  srcs = ["main_a.c", ":foo"],
)

cc_binary(
  name = "bin_b",
  srcs = ["main_b.c", ":foo"],
)

genfoo(
  name = "foo",
)

installer(
  name = "installer",
  deps = [
    ":bin_a",
    ":bin_b",
  ]
)

flag(
  name = "flag",
  build_setting_default = "0",
)

config_hints(name = "config_hints")

defs.bzl:

# Custom Starlark flag

FlagProvider = provider(fields = ['value'])

def _impl(ctx):
    return FlagProvider(value = ctx.build_setting_value)

flag = rule(
    implementation = _impl,
    build_setting = config.string(flag = True)
)

# Split transition

def _platform_expander_transition_impl(settings, attr):
    return [
        {"//command_line_option:compilation_mode": "opt", "//:flag": "1"},
        {"//command_line_option:compilation_mode": "opt", "//:flag": "2"},
        {"//command_line_option:compilation_mode": "dbg", "//:flag": "1"},
        {"//command_line_option:compilation_mode": "dbg", "//:flag": "2"},
    ]

platform_expander_transition = transition(
    implementation = _platform_expander_transition_impl,
    inputs = [],
    outputs = [
        "//command_line_option:compilation_mode",
        "//:flag",
    ],
)

# Puts configuration data into a provider so that each configuration split
# can be identified in _installer_impl

ConfigHintsProvider = provider(fields = ["compilation_mode", "flag"])

def _config_hints_impl(ctx):
  return ConfigHintsProvider(
      compilation_mode = ctx.var["COMPILATION_MODE"],
      flag = ctx.attr.flag[FlagProvider].value,
  )

config_hints = rule(
  implementation = _config_hints_impl,
  attrs = {
    "flag": attr.label(default="//:flag"),
  },
)

# Generate a file based on //:flag

def _genfoo_impl(ctx):
  foo = ctx.actions.declare_file(ctx.attr.name + ".c")
  ctx.actions.write(
      output = foo,
      content = """
int foo() {
  return 42 * %d;
}
""" % int(ctx.attr.flag[FlagProvider].value))
  return DefaultInfo(files = depset(direct = [foo]))

genfoo = rule(
  implementation = _genfoo_impl,
  attrs = {
    "flag": attr.label(default="//:flag"),
  },
)

# Installer rule

def _installer_impl(ctx):
  installer = ctx.actions.declare_file(ctx.attr.name + ".zip")

  zipper_args = ctx.actions.args()
  zipper_args.add("c", installer.path)

  zipper_inputs = []
  for config_key in ctx.split_attr.deps:
    config_hints = ctx.split_attr._config_hints[config_key][ConfigHintsProvider]
    for dep in ctx.split_attr.deps[config_key]:
      binary = dep.files.to_list()[0] # not usually a good idea to collapse depsets
      zipper_inputs.append(binary)
      path_in_zip = "%s-flag-%s/%s" % (
          config_hints.compilation_mode,
          config_hints.flag,
          binary.basename)
      zipper_args.add(path_in_zip + "=" + binary.path)

  ctx.actions.run(
      inputs = zipper_inputs,
      outputs = [installer],
      executable = ctx.executable._zipper,
      arguments = [zipper_args],
      progress_message = "Creating installer",
      mnemonic = "zipper",
  )

  return DefaultInfo(files = depset(direct = [installer]))

installer = rule(
    implementation = _installer_impl,
    attrs = {
      "deps": attr.label_list(cfg = platform_expander_transition),
      "_config_hints": attr.label(default = ":config_hints", cfg = platform_expander_transition),
      "_zipper": attr.label(default = "@bazel_tools//tools/zip:zipper", cfg = "exec", executable = True),
      "_allowlist_function_transition": attr.label(
          default = "@bazel_tools//tools/allowlists/function_transition_allowlist"),
    },
)

main_a.c:

#include "stdio.h"

int foo();

int main() {
  #ifdef NDEBUG
  printf("main a opt: foo is %d\n", foo());
  #else
  printf("main a dbg: foo is %d\n", foo());
  #endif
  return 0;
}

main_b.c:

#include "stdio.h"

int foo();

int main() {
  #ifdef NDEBUG
  printf("main b opt: foo is %d\n", foo());
  #else
  printf("main b dbg: foo is %d\n", foo());
  #endif
  return 0;
}

And building:

$ bazel build installer
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:installer (37 packages loaded, 573 targets configured).
INFO: Found 1 target...
Target //:installer up-to-date:
  bazel-bin/installer.zip
INFO: Elapsed time: 5.498s, Critical Path: 0.18s
INFO: 38 processes: 13 internal, 25 linux-sandbox.
INFO: Build completed successfully, 38 total actions

$ unzip -lv bazel-bin/installer.zip
Archive:  bazel-bin/installer.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
    9952  Stored     9952   0% 2010-01-01 00:00 3dec8c4d  dbg-flag-1/bin_a
    9952  Stored     9952   0% 2010-01-01 00:00 fbcabb15  dbg-flag-1/bin_b
    9952  Stored     9952   0% 2010-01-01 00:00 5b348f4f  dbg-flag-2/bin_a
    9952  Stored     9952   0% 2010-01-01 00:00 cf802980  dbg-flag-2/bin_b
    7928  Stored     7928   0% 2010-01-01 00:00 35395b2e  opt-flag-1/bin_a
    7928  Stored     7928   0% 2010-01-01 00:00 2fe4fa5c  opt-flag-1/bin_b
    7928  Stored     7928   0% 2010-01-01 00:00 d4ab323b  opt-flag-2/bin_a
    7928  Stored     7928   0% 2010-01-01 00:00 5798472c  opt-flag-2/bin_b
--------          -------  ---                            -------
   71520            71520   0%                            8 files

$ unzip -q -d /tmp/installer bazel-bin/installer.zip

$ for b in $(find /tmp/installer -type f | sort); do echo -n "$b: " ; $b; done
/tmp/installer/dbg-flag-1/bin_a: main a dbg: foo is 42
/tmp/installer/dbg-flag-1/bin_b: main b dbg: foo is 42
/tmp/installer/dbg-flag-2/bin_a: main a dbg: foo is 84
/tmp/installer/dbg-flag-2/bin_b: main b dbg: foo is 84
/tmp/installer/opt-flag-1/bin_a: main a opt: foo is 42
/tmp/installer/opt-flag-1/bin_b: main b opt: foo is 42
/tmp/installer/opt-flag-2/bin_a: main a opt: foo is 84
/tmp/installer/opt-flag-2/bin_b: main b opt: foo is 84

Without the config_hints and just packing up the artifacts directly, you'd get something like this (basically just the contents of bazel-out):

$ unzip -l bazel-bin/installer.zip
Archive:  bazel-bin/installer.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     9136  2010-01-01 00:00   bazel-out/k8-dbg-ST-e55654231bfa/bin/bin_a
     9136  2010-01-01 00:00   bazel-out/k8-dbg-ST-ed4700d5e450/bin/bin_a
     7864  2010-01-01 00:00   bazel-out/k8-opt-ST-0bf6078f5c66/bin/bin_a
     7864  2010-01-01 00:00   bazel-out/k8-opt-ST-3b0a7e3b15d3/bin/bin_a
     9144  2010-01-01 00:00   bazel-out/k8-dbg-ST-e55654231bfa/bin/bin_b
     9144  2010-01-01 00:00   bazel-out/k8-dbg-ST-ed4700d5e450/bin/bin_b
     7872  2010-01-01 00:00   bazel-out/k8-opt-ST-0bf6078f5c66/bin/bin_b
     7872  2010-01-01 00:00   bazel-out/k8-opt-ST-3b0a7e3b15d3/bin/bin_b
---------                     -------
    68032                     8 files

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/32541363-f795-47ae-b65c-0c101abbf5f3n%40googlegroups.com.

[Brian Silverman]

Would ctx.target_platform_has_constraint let you avoid the separate rule? The proposal has an example of how to use it.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAAvaYBkgDsJ8JJdG0MtGOZY-%3Dhhkb8xstemAqKDQ4ykzbRm1QA%40mail.gmail.com.

[Konstantin]

Thank you Alex and Brian! It took me some time to wrap my head around those concepts, but by now I feel pretty much enlightened 😊

 Interestingly I recently started moving in a very similar direction (although not that far) because of the different problem – we have aspect which needs to know some configuration. Naïve attempt to give aspect an attribute and populate it with select() failed with this error: “Unfortunately aspect attributes don't currently support select()”. Bummer. So after some deliberation I proudly came with the idea of another rule which can get necessary configuration with select(), pack it into a custom provider and then make our aspect have an attribute of type attr.label and point it to the instance of that “configuration” rule.

 So Alex, your example was already somewhat familiar to me, but some fragments were revelations. One is this:

        "deps": attr.label_list(cfg = platform_expander_transition),

        "_config_hints": attr.label(default = ":config_hints", cfg = platform_expander_transition),

It is pretty bright idea to apply the same transform to two attributes and then check the provider in one to figure configuration of another. That I did not think about.

 Another fragment is this:

        for config_key in ctx.split_attr.deps:

                config_hints = ctx.split_attr._config_hints[config_key][ConfigHintsProvider]

which actually shows how to get the keys to the split attribute dictionary and then extract split attribute values using those keys. I remember seeing it documented somewhere, but I could not make sense of that documentation. This example helped tremendously.

 Brian, thank you for the hint about ctx.target_platform_has_constraint I did not know about it. At first it did not look very useful, because it is just another way of doing what we already can do with the configuration conveying rule. Also it requires implicit attribute for each configuration parameter we want to check. But then looking at this:

attrs = {

    ...

    '_windows_constraint': attr.label(default = '@bazel_platforms//os:windows'),

  },

 windows_constraint = ctx.attr._windows_constraint[platform_common.ConstraintValueInfo]

 if ctx.target_platform_has_constraint(windows_constraint):

 I grasped the point about the correctness guarantees and it made sense after all.

 I have not implemented it all on my side yet, but I feel pretty confident that I have all the pieces of the puzzle to put it together.

 Thank you again for being so helpful!

Konstantin

[Konstantin]

Now even more interesting complication: besides the top level transition we figure out above, there are other transitions attached to the rule down below the dependency chain and naturally they are built into the folders with unpredictable names. How to locate and collect those?