Interactions between label_flags

304 views
Skip to first unread message

Jonathan Perry

unread,
Aug 24, 2023, 12:30:27 PM8/24/23
to bazel-discuss
I have a very woolly and ill formed question. I have added a label_flag into my build, and then use transitions across the codebase to control its value appropriately. Everything is working extremely nicely except for one weirdness - as I add new values for the label_flag, and as I change my build graph to incorporate the resultant targets, I keep on ending up rebuilding totally unrelated targets way upstream of any consumer of my label_flag. In particular, I rebuild the go sdk from scratch. I can’t understand why this is happening and I’m struggling to find insight on how disjoint label_flag declarations interact with each other. 

Does anyone have a pointer to a doc or perhaps the patience to try to explain please?

Thank you!

John Cater

unread,
Aug 24, 2023, 1:30:01 PM8/24/23
to Jonathan Perry, bazel-discuss
The problem here is that bazel is currently fairly conservative about what configuration changes mean the outputs change.

As an example, if you have a simple dependency tree like this:
go_binary(name = "binary", srcs = ..., deps = [":library"],)
go_library(name = "library", srcs = ...)

When you run `bazel build --flags //:binary`, Bazel knows that both `:binary` and `:library` are in the same configuration and have the same flags.

When you add your transition on any flag (not just a label flag) to the top level, the configuration changes. Bazel doesn't know[1] that the `:library` target doesn't care about that changed flag, so it re-analyzes and re-builds it anyway. With something large like the go sdk, that can be very visible.

[Footnote 1]: More details: Bazel actually treats target analysis and action execution separately. Even with the changed configuration, if the action to build `:library`'s outputs doesn't change, Bazel will skip re-executing. The problem is, again, that Bazel doesn't know what flag changes are and are not different, and so a hash of the Starlark flags (like your label flag) ends up in the generated output path, and thus the action is different (it's reading and writing to different paths).

We have a couple of efforts to deal with this:
1. Output path stripping: this aims to reduce the amount of unneeded re-execution by removing unused data from the output path. This probably won't help you with your label_flag, because the flag is actually changing, not just staying at the default.
2. Improved exec transitions: This might help, because parts of the go sdk are tools used in the build, and parts are libraries, and this will at least keep the tools using the same configuration, so that those don't rebuild. The sdk libraries would still, unfortunately, rebuild
3. We keep discussing ways to do true analysis improvements by tracking what flags a target actually cares about, and only re-analyzing the target when those flags change. This turns out to be wildly complicated, and every prototype we've tried has spent more time and memory tracking flag usage than was actually saved. We still have some remaining ideas and intend to persue them, but it's tricky.



--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU37RRwMqQLda8ie0R0Rf-D2sMquAEodh4XVrw%2Bk%2B%2BJHUFA%40mail.gmail.com.

Jonathan Perry

unread,
Aug 24, 2023, 2:22:33 PM8/24/23
to John Cater, bazel-discuss
Wow. Clear, fast, extremely helpful. Thank you!

So three questions:
- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?
- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?
- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Thank you again!

John Cater

unread,
Aug 24, 2023, 3:20:47 PM8/24/23
to Jonathan Perry, Ivo Ristovski List, bazel-discuss
On Thu, Aug 24, 2023 at 2:22 PM Jonathan Perry <jonp...@gmail.com> wrote:
So three questions:
- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?

If you can find appropriate cutpoints, yes. In my example, since both the go_binary and go_library need the go sdk, trying to transition back for the library wouldn't help )and might just cause a further duplication of configurations). It depends a lot on your dependency graph and the structure of your builds.
 
- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?

You can use `bazel cquery 'deps(//target)'` to see all of the configured target dependencies for `//target`, this will list the same target multiple times if it exists in different configurations.
 
- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Not currently, but @Ivo Ristovski List has a proposal active to implement this. I don't know what the timeframe for this is, but it will most likely not be in the next bazel release, Bazel 7.0.

Jonathan Perry

unread,
Aug 24, 2023, 3:26:52 PM8/24/23
to John Cater, Ivo Ristovski List, bazel-discuss
Thank you.  I think in my particular case I actually can have useful cut points. I will experiment!

 For question 2 I was hoping for a thing I could add as a test target to be picked up with test //…, but I guess since it’s cquery that won’t be possible?

Thanks for the doc on question 3. 

Jonathan Perry

unread,
Sep 1, 2023, 6:33:36 AM9/1/23
to John Cater, Ivo Ristovski List, bazel-discuss
So - this is working exactly as you advertised. I’m trying to find scalable and long-term-supportable ways of cutting the transition propagation to ensure minimal graph splits, and it’s super hard. But the upside of how this all hangs together is so worth it. Bazel is amazing!

I wonder how others deal with this - is there at least a manual cquery I could run that would tell me something like ‘the topmost dependencies underneath some expression which has config value X set where the transitive closure underneath it does not contain a reference to that config value’? And then maybe I can iterate on that to cut and cut and cut…

Man it would be awesome if bazel just figured out for itself, but I appreciate the difficulty of the request!

Thanks again for all the help here. 

Jonathan Perry

unread,
Sep 14, 2023, 11:22:38 AM9/14/23
to John Cater, Ivo Ristovski List, bazel-discuss
I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 
I guess the more general question would be - what exactly goes into the computation of the output path described below?

Thanks!

John Cater

unread,
Sep 18, 2023, 9:46:58 AM9/18/23
to Jonathan Perry, Stephen Twigg, Ivo Ristovski List, bazel-discuss
On Thu, Sep 14, 2023 at 11:22 AM Jonathan Perry <jonp...@gmail.com> wrote:
I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 

Generally we expect them to be the same, but this can be affected a lot by local environment and how that affects execution platforms and toolchains: one build from a Linux machine and another from a macOS machine may end up different because they are using different tools.
 
I guess the more general question would be - what exactly goes into the computation of the output path described below?

This is actually being reworked currently  by @Stephen Twigg: Stephen, do you plan to expand the documentation about output paths as part of this?

Jonathan Perry

unread,
Sep 21, 2023, 4:14:32 AM9/21/23
to John Cater, Ivo Ristovski List, Stephen Twigg, bazel-discuss
Thank you again for your response

Jonathan Perry

unread,
Jan 17, 2024, 4:11:59 AMJan 17
to John Cater, Ivo Ristovski List, Stephen Twigg, bazel-discuss
Hello again!

Yet another question on this topic please…

I am trying to come up with a blanket hammer of ‘any external target should transition back to default’. This will give me a reasonable coarse-grained cut on the propagation of the graph split. 

Can you think of any sane way to get that done please? Or anything close (like making it happen only for java_* targets or something)?

Thank you!

Jonathan Perry

unread,
Jan 18, 2024, 3:00:04 AMJan 18
to John Cater, Ivo Ristovski List, Stephen Twigg, bazel-discuss
A couple of concrete possibilities:
- I find a way to patch the built in java_library etc, and add a transition to all of them conditioned on the workspace of the label. (Any suggestions here much appreciated, I’ve been googling and digging into source and can’t find a way that I could have sworn used to exist)
- we have some traversal control functionality in transitions, to allow us to control the propagation of the split. So a transition gets a new field, say traversal_controller or something, which gets the label (and attrs?) of a candidate target, and is allowed to return ‘CONTINUE’ or ‘STOP’. 

Jonathan Perry

unread,
Jan 19, 2024, 4:30:45 AMJan 19
to bazel-discuss
Any thoughts on the idea of controlling the traversal?  It seems to my naive mind that this might help mitigate the explosive nature of transitions...

Thank you!

Jonathan Perry

unread,
Jan 19, 2024, 4:49:14 AMJan 19
to bazel-discuss
Sorry for the spam, but one thought here - if we could set a default transition for all rules whose job was to decide whether to flip back to the default, then I think we could implement the equivalent of the STOP functionality with minimal bazel java changes...  but I can imagine that might raise some performance concerns

John Cater

unread,
Jan 19, 2024, 9:05:06 AMJan 19
to Jonathan Perry, bazel-discuss
This isn't really the angle we're using to approach handling transition fanout, so I don't think anyone has any useful advice. If you find anything interesting feel free to report back.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

Jonathan Perry

unread,
Jan 19, 2024, 9:49:15 AMJan 19
to bazel-discuss
Ack, thanks, will do!
Where's the best place for me to get familiar with your latest thoughts please?

Jonathan Perry

unread,
Jan 22, 2024, 1:07:45 PMJan 22
to bazel-discuss
So I think I answered my own question, which is ‘look at this doc: 

Which then got me to —experimental_output_paths=strip

So I’m trying that

Am I correct?

Thank you as always for your patience. 

Jonathan Perry

unread,
Jan 24, 2024, 6:53:01 AMJan 24
to bazel-discuss
One thing I can’t tell from this doc is whether I should use ‘strip’ or ‘content’. Naively, ‘content’ seems better (and indeed appears to work, even for my custom rules), but…how should I choose please?

Fabian Meumertzheim

unread,
Jan 24, 2024, 8:15:02 AMJan 24
to Jonathan Perry, Greg Estren, bazel-discuss
I don't think that 'content' is fully implemented. You can get a sense of what is working from this BazelCon talk: https://youtu.be/Et1rjb7ixUU
+Greg Estren 

Jonathan Perry

unread,
Jan 24, 2024, 8:22:08 AMJan 24
to fab...@meumertzhe.im, Greg Estren, bazel-discuss
Brilliant thank you

Greg Estren

unread,
Jan 25, 2024, 6:04:26 PMJan 25
to Jonathan Perry, fab...@meumertzhe.im, bazel-discuss
"content" is not being planned. It's an interesting idea but really design-invasive. We're putting effort in that space into what Fabian linked.

Re: controlling flag propagation, we have all kinds of thoughts on that! We're fleshing out some 2024 ideas of a stronger propagation model for flags, vs. the historic default of "everything propagates everywhere". One concrete subset of this is controlling target -> exec propagation where we've already added more flexibility.

The best place to continue this discussion is probably here. All of this really depends on what flags we're talking about and where exactly we want them to propagate. Starlark flags are different than --platforms is different than --copt, for example. Also whether you want to control this on a flag-by-flag basis vs. a whole collection of flags at once.

Jonathan Perry

unread,
Jan 26, 2024, 5:51:11 AMJan 26
to bazel-discuss
Thank you for the response.  Would you be interested in a call to discuss our usecase?

Jonathan Perry

unread,
Jan 29, 2024, 8:50:19 AMJan 29
to bazel-discuss
I understand that a request for a call is presumptuous, but I promise there are reasons. 

I am interested in a use case which:
- uses transitions to control a single label_flag
- consumes that label_flag exclusively within a single repository (which is the repo in which the flag is defined), through a well-known set of rules
- has at least tens of settings for that label_flag

Jonathan Perry

unread,
Jan 31, 2024, 3:38:42 AMJan 31
to bazel-discuss
Two more I think interesting points
- the label_flag’s value is used in exec and target (specifically it would be consumed by both a java_library dep of a build target as well as by a java_binary used as a tool by a custom rule)
- the label_flag’s value is never used in a control flow expression of a rule’s implementation (dunno if that’s a thing but it feels like it might be?)

Jonathan Perry

unread,
Feb 1, 2024, 4:28:59 PMFeb 1
to bazel-discuss
Greg- is this basically the data you wanted? 

Thanks!

Jonathan Perry

unread,
Feb 7, 2024, 12:56:25 AMFeb 7
to bazel-discuss
Pinging this thread

Jonathan Perry

unread,
Feb 9, 2024, 3:00:29 AMFeb 9
to bazel-discuss
Pinging again - is there any more useful info I can give you?

Jonathan Perry

unread,
Feb 13, 2024, 8:44:31 AMFeb 13
to bazel-discuss
Is there somewhere else I should be pinging? I would really love to give any data that would be useful. Thanks. 

Jonathan Perry

unread,
Feb 17, 2024, 6:13:54 AMFeb 17
to bazel-discuss
Ping. 

Jonathan Perry

unread,
Feb 28, 2024, 3:42:10 AMFeb 28
to bazel-discuss
Ping
Reply all
Reply to author
Forward
0 new messages