Selecting a remote execution backend for a rule

[Jonathan Perry]

Hello! Does anyone happen to have a pointer to a worked example of defining a rule which must always execute on some particular subset of buildbarn workers/runners, but whose inputs and outputs are produced/consumed by rules without such a restriction? 

My usecase is that I am trying to incrementally introduce nixos into my environment, and I want to invoke specialised rules on the nixos backends as part of bazel test //…

I have a setup involving exec_compatible_with, but it doesn’t seem to always 100% work (in that with some bazel action graphs the nixos-exec_compatible_with-tagged rules get invoked on a Debian worker), so I’m hoping there’s a fully baked setup which gives me precise control and from which I can learn/copy. 

Thank you!

[zhang kai]

I think you can try exec_properties .

[Jonathan Perry]

Thanks for that. However my question is more about how to do the transitions such that it’s guaranteed to execute on the chosen backend. So I’m really hoping for a full worked example rather than just an alternative to exec_compatible_with. 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/71be246f-bdb4-4f7b-a52f-618afb4925b8n%40googlegroups.com.

[zhang kai]

You can set custom key and value to `exec_properties`. Most remote build servers like buildfarm and buildbarn support using `exec_properties` to schedule actions to matching workers.

For example, if you add following `exec_properties` to a target:

```

xx_rule(
    name = "some_rule",
     exec_properties = {"requireSomeEnv": "True",},

)

```

Then you put `{"requireSomeEnv": "True",}` to your remote workers' `exec_properties`(depending on what remote server you're using the config may be different). Then the `some_rule` will only be executed on the chosen workers.

[Jonathan Perry]

Thank you! Is there a way to make it so that _all_ xx_rule targets get those exec_properties?

My impression was that exec_compatible_with is the ‘right’ way to do it - is that wrong?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9eb639ff-f116-4252-9ad5-e06affae578en%40googlegroups.com.

[zhang kai]

I'm not familiar with `exec_compatible_with`. After some googling, it seems that they can be used combined: https://github.com/bazelbuild/bazel/issues/16743#issuecomment-1320414588 . Seems you can make a toolchain to only execute on specific remote workers this way.

[Fredrik Medley]

The way I think about using the platforms is that the number of (remote) execution platforms are finite and you should be able to define all of them explicitly in Bazel. It is then up to the toolchain resolution to choose a specific execution platform, so `some_rule` needs to define a dependency on at least one toolchain (https://bazel.build/extending/toolchains#writing-rules-toolchains). Otherwise, declaring a macro which sets `exec_properties` or `exec_compatible_with` is a temporary hacky way for trying things out.

I highly recommend watching https://www.youtube.com/watch?v=5Y2WiUafVs0&list=PLxNYxgaZ8RsdH4GCIZ69dzxQCOPyuNlpF&index=18 from BazelCon 2022 where Susan Steinman and John Cater walks through the whole toolchain, platform and exec_group handling. For example, I didn't know that `--toolchain_resolution_debug` can take a target label as argument. Another feature I recently picked up is the `target_settings` attribute of the toolchain rule (https://bazel.build/reference/be/platform#toolchain).

[Jonathan Perry]

Ah nice thanks for the links I will further my education!

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/76620d0f-db9d-491b-bde4-e2c08ae6b0d5n%40googlegroups.com.

[Jonathan Perry]

That was a great talk thanks for the pointer. I’m now way clearer on what’s going on and have solved the proximate problem I had. Thank you!