Interactions between label_flags

[Jonathan Perry]

I have a very woolly and ill formed question. I have added a label_flag into my build, and then use transitions across the codebase to control its value appropriately. Everything is working extremely nicely except for one weirdness - as I add new values for the label_flag, and as I change my build graph to incorporate the resultant targets, I keep on ending up rebuilding totally unrelated targets way upstream of any consumer of my label_flag. In particular, I rebuild the go sdk from scratch. I can’t understand why this is happening and I’m struggling to find insight on how disjoint label_flag declarations interact with each other. 

Does anyone have a pointer to a doc or perhaps the patience to try to explain please?

Thank you!

[John Cater]

The problem here is that bazel is currently fairly conservative about what configuration changes mean the outputs change.

As an example, if you have a simple dependency tree like this:

go_binary(name = "binary", srcs = ..., deps = [":library"],)

go_library(name = "library", srcs = ...)

When you run `bazel build --flags //:binary`, Bazel knows that both `:binary` and `:library` are in the same configuration and have the same flags.

When you add your transition on any flag (not just a label flag) to the top level, the configuration changes. Bazel doesn't know[1] that the `:library` target doesn't care about that changed flag, so it re-analyzes and re-builds it anyway. With something large like the go sdk, that can be very visible.

[Footnote 1]: More details: Bazel actually treats target analysis and action execution separately. Even with the changed configuration, if the action to build `:library`'s outputs doesn't change, Bazel will skip re-executing. The problem is, again, that Bazel doesn't know what flag changes are and are not different, and so a hash of the Starlark flags (like your label flag) ends up in the generated output path, and thus the action is different (it's reading and writing to different paths).

We have a couple of efforts to deal with this:

1. Output path stripping: this aims to reduce the amount of unneeded re-execution by removing unused data from the output path. This probably won't help you with your label_flag, because the flag is actually changing, not just staying at the default.

2. Improved exec transitions: This might help, because parts of the go sdk are tools used in the build, and parts are libraries, and this will at least keep the tools using the same configuration, so that those don't rebuild. The sdk libraries would still, unfortunately, rebuild

3. We keep discussing ways to do true analysis improvements by tracking what flags a target actually cares about, and only re-analyzing the target when those flags change. This turns out to be wildly complicated, and every prototype we've tried has spent more time and memory tracking flag usage than was actually saved. We still have some remaining ideas and intend to persue them, but it's tricky.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU37RRwMqQLda8ie0R0Rf-D2sMquAEodh4XVrw%2Bk%2B%2BJHUFA%40mail.gmail.com.

[Jonathan Perry]

Wow. Clear, fast, extremely helpful. Thank you!

So three questions:

- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?

- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?

- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Thank you again!

[John Cater]

On Thu, Aug 24, 2023 at 2:22 PM Jonathan Perry <jonp...@gmail.com> wrote:

So three questions:

- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?

If you can find appropriate cutpoints, yes. In my example, since both the go_binary and go_library need the go sdk, trying to transition back for the library wouldn't help )and might just cause a further duplication of configurations). It depends a lot on your dependency graph and the structure of your builds.

 

- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?

You can use `bazel cquery 'deps(//target)'` to see all of the configured target dependencies for `//target`, this will list the same target multiple times if it exists in different configurations.

 

- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Not currently, but @Ivo Ristovski List has a proposal active to implement this. I don't know what the timeframe for this is, but it will most likely not be in the next bazel release, Bazel 7.0.

[Jonathan Perry]

Thank you.  I think in my particular case I actually can have useful cut points. I will experiment!

 For question 2 I was hoping for a thing I could add as a test target to be picked up with test //…, but I guess since it’s cquery that won’t be possible?

Thanks for the doc on question 3. 

[Jonathan Perry]

So - this is working exactly as you advertised. I’m trying to find scalable and long-term-supportable ways of cutting the transition propagation to ensure minimal graph splits, and it’s super hard. But the upside of how this all hangs together is so worth it. Bazel is amazing!

I wonder how others deal with this - is there at least a manual cquery I could run that would tell me something like ‘the topmost dependencies underneath some expression which has config value X set where the transitive closure underneath it does not contain a reference to that config value’? And then maybe I can iterate on that to cut and cut and cut…

Man it would be awesome if bazel just figured out for itself, but I appreciate the difficulty of the request!

Thanks again for all the help here. 

[Jonathan Perry]

I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 

I guess the more general question would be - what exactly goes into the computation of the output path described below?

Thanks!

[John Cater]

On Thu, Sep 14, 2023 at 11:22 AM Jonathan Perry <jonp...@gmail.com> wrote:

I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 

Generally we expect them to be the same, but this can be affected a lot by local environment and how that affects execution platforms and toolchains: one build from a Linux machine and another from a macOS machine may end up different because they are using different tools.

 

I guess the more general question would be - what exactly goes into the computation of the output path described below?

This is actually being reworked currently  by @Stephen Twigg: Stephen, do you plan to expand the documentation about output paths as part of this?

[Jonathan Perry]

Thank you again for your response

[Jonathan Perry]

Hello again!

Yet another question on this topic please…

I am trying to come up with a blanket hammer of ‘any external target should transition back to default’. This will give me a reasonable coarse-grained cut on the propagation of the graph split. 

Can you think of any sane way to get that done please? Or anything close (like making it happen only for java_* targets or something)?

Thank you!

[Jonathan Perry]

A couple of concrete possibilities:

- I find a way to patch the built in java_library etc, and add a transition to all of them conditioned on the workspace of the label. (Any suggestions here much appreciated, I’ve been googling and digging into source and can’t find a way that I could have sworn used to exist)

- we have some traversal control functionality in transitions, to allow us to control the propagation of the split. So a transition gets a new field, say traversal_controller or something, which gets the label (and attrs?) of a candidate target, and is allowed to return ‘CONTINUE’ or ‘STOP’. 

[Jonathan Perry]

Any thoughts on the idea of controlling the traversal?  It seems to my naive mind that this might help mitigate the explosive nature of transitions...

Thank you!

[Jonathan Perry]

Sorry for the spam, but one thought here - if we could set a default transition for all rules whose job was to decide whether to flip back to the default, then I think we could implement the equivalent of the STOP functionality with minimal bazel java changes...  but I can imagine that might raise some performance concerns

[John Cater]

This isn't really the angle we're using to approach handling transition fanout, so I don't think anyone has any useful advice. If you find anything interesting feel free to report back.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/ed989db3-ce88-4f32-bd33-c6bc50e79b53n%40googlegroups.com.

[Jonathan Perry]

Ack, thanks, will do!

Where's the best place for me to get familiar with your latest thoughts please?

[Jonathan Perry]

So I think I answered my own question, which is ‘look at this doc: 

https://docs.google.com/document/d/17snvmic26-QdGuwVw55Gl0oOufw9sCVuOAvHqGZJFr4/edit‘

Which then got me to —experimental_output_paths=strip

So I’m trying that

Am I correct?

Thank you as always for your patience. 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4110806d-e6c5-473e-8958-c3b424d138a5n%40googlegroups.com.

[Jonathan Perry]

One thing I can’t tell from this doc is whether I should use ‘strip’ or ‘content’. Naively, ‘content’ seems better (and indeed appears to work, even for my custom rules), but…how should I choose please?

[Fabian Meumertzheim]

I don't think that 'content' is fully implemented. You can get a sense of what is working from this BazelCon talk: https://youtu.be/Et1rjb7ixUU

+Greg Estren 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU348iXF7KR3HFQD4fBfHP7wkLZvh6v7LVHfdfi%2BU6B3S1A%40mail.gmail.com.

[Jonathan Perry]

Brilliant thank you

[Greg Estren]

"content" is not being planned. It's an interesting idea but really design-invasive. We're putting effort in that space into what Fabian linked.

Re: controlling flag propagation, we have all kinds of thoughts on that! We're fleshing out some 2024 ideas of a stronger propagation model for flags, vs. the historic default of "everything propagates everywhere". One concrete subset of this is controlling target -> exec propagation where we've already added more flexibility.

The best place to continue this discussion is probably here. All of this really depends on what flags we're talking about and where exactly we want them to propagate. Starlark flags are different than --platforms is different than --copt, for example. Also whether you want to control this on a flag-by-flag basis vs. a whole collection of flags at once.

[Jonathan Perry]

Thank you for the response.  Would you be interested in a call to discuss our usecase?

[Jonathan Perry]

I understand that a request for a call is presumptuous, but I promise there are reasons. 

I am interested in a use case which:

- uses transitions to control a single label_flag

- consumes that label_flag exclusively within a single repository (which is the repo in which the flag is defined), through a well-known set of rules

- has at least tens of settings for that label_flag

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/cccb5d9c-9124-42bf-8f6e-8acaca30b14bn%40googlegroups.com.

[Jonathan Perry]

Two more I think interesting points

- the label_flag’s value is used in exec and target (specifically it would be consumed by both a java_library dep of a build target as well as by a java_binary used as a tool by a custom rule)

- the label_flag’s value is never used in a control flow expression of a rule’s implementation (dunno if that’s a thing but it feels like it might be?)

[Jonathan Perry]

Greg- is this basically the data you wanted? 

Thanks!

[Jonathan Perry]

Pinging this thread

[Jonathan Perry]

Pinging again - is there any more useful info I can give you?

[Jonathan Perry]

Is there somewhere else I should be pinging? I would really love to give any data that would be useful. Thanks. 

[Jonathan Perry]

Ping. 

[Jonathan Perry]

Ping