Interactions between label_flags

[Jonathan Perry]

I have a very woolly and ill formed question. I have added a label_flag into my build, and then use transitions across the codebase to control its value appropriately. Everything is working extremely nicely except for one weirdness - as I add new values for the label_flag, and as I change my build graph to incorporate the resultant targets, I keep on ending up rebuilding totally unrelated targets way upstream of any consumer of my label_flag. In particular, I rebuild the go sdk from scratch. I can’t understand why this is happening and I’m struggling to find insight on how disjoint label_flag declarations interact with each other. 

Does anyone have a pointer to a doc or perhaps the patience to try to explain please?

Thank you!

[John Cater]

The problem here is that bazel is currently fairly conservative about what configuration changes mean the outputs change.

As an example, if you have a simple dependency tree like this:

go_binary(name = "binary", srcs = ..., deps = [":library"],)

go_library(name = "library", srcs = ...)

When you run `bazel build --flags //:binary`, Bazel knows that both `:binary` and `:library` are in the same configuration and have the same flags.

When you add your transition on any flag (not just a label flag) to the top level, the configuration changes. Bazel doesn't know[1] that the `:library` target doesn't care about that changed flag, so it re-analyzes and re-builds it anyway. With something large like the go sdk, that can be very visible.

[Footnote 1]: More details: Bazel actually treats target analysis and action execution separately. Even with the changed configuration, if the action to build `:library`'s outputs doesn't change, Bazel will skip re-executing. The problem is, again, that Bazel doesn't know what flag changes are and are not different, and so a hash of the Starlark flags (like your label flag) ends up in the generated output path, and thus the action is different (it's reading and writing to different paths).

We have a couple of efforts to deal with this:

1. Output path stripping: this aims to reduce the amount of unneeded re-execution by removing unused data from the output path. This probably won't help you with your label_flag, because the flag is actually changing, not just staying at the default.

2. Improved exec transitions: This might help, because parts of the go sdk are tools used in the build, and parts are libraries, and this will at least keep the tools using the same configuration, so that those don't rebuild. The sdk libraries would still, unfortunately, rebuild

3. We keep discussing ways to do true analysis improvements by tracking what flags a target actually cares about, and only re-analyzing the target when those flags change. This turns out to be wildly complicated, and every prototype we've tried has spent more time and memory tracking flag usage than was actually saved. We still have some remaining ideas and intend to persue them, but it's tricky.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU37RRwMqQLda8ie0R0Rf-D2sMquAEodh4XVrw%2Bk%2B%2BJHUFA%40mail.gmail.com.

[Jonathan Perry]

Wow. Clear, fast, extremely helpful. Thank you!

So three questions:

- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?

- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?

- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Thank you again!

[John Cater]

On Thu, Aug 24, 2023 at 2:22 PM Jonathan Perry <jonp...@gmail.com> wrote:

So three questions:

- sounds like my best option is to find ways of pruning the effects by retransitioning back to the default at appropriate points, correct?

If you can find appropriate cutpoints, yes. In my example, since both the go_binary and go_library need the go sdk, trying to transition back for the library wouldn't help )and might just cause a further duplication of configurations). It depends a lot on your dependency graph and the structure of your builds.

 

- is there some way for me to add an assertive pure bazel test to make sur so get this right? So like ‘go sdk exists under only one configuration in the transitive closure of this target’?

You can use `bazel cquery 'deps(//target)'` to see all of the configured target dependencies for `//target`, this will list the same target multiple times if it exists in different configurations.

 

- long shot: is there a way for me to inject a transition into someone else’s rule? Specifically, for example, it would be super helpful (but not required!) to transition back on the base_image attr of container_image. 

Not currently, but @Ivo Ristovski List has a proposal active to implement this. I don't know what the timeframe for this is, but it will most likely not be in the next bazel release, Bazel 7.0.

[Jonathan Perry]

Thank you.  I think in my particular case I actually can have useful cut points. I will experiment!

 For question 2 I was hoping for a thing I could add as a test target to be picked up with test //…, but I guess since it’s cquery that won’t be possible?

Thanks for the doc on question 3. 

[Jonathan Perry]

So - this is working exactly as you advertised. I’m trying to find scalable and long-term-supportable ways of cutting the transition propagation to ensure minimal graph splits, and it’s super hard. But the upside of how this all hangs together is so worth it. Bazel is amazing!

I wonder how others deal with this - is there at least a manual cquery I could run that would tell me something like ‘the topmost dependencies underneath some expression which has config value X set where the transitive closure underneath it does not contain a reference to that config value’? And then maybe I can iterate on that to cut and cut and cut…

Man it would be awesome if bazel just figured out for itself, but I appreciate the difficulty of the request!

Thanks again for all the help here. 

[Jonathan Perry]

I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 

I guess the more general question would be - what exactly goes into the computation of the output path described below?

Thanks!

[John Cater]

On Thu, Sep 14, 2023 at 11:22 AM Jonathan Perry <jonp...@gmail.com> wrote:

I have a further question about this please. With respect to the output paths - given the same configuration across two independent invocations, from two different bazel clients, do we expect the output paths to be the same? Specifically, would we expect transitions to break remote caches?  I’m assuming not, but I want to check please. 

Generally we expect them to be the same, but this can be affected a lot by local environment and how that affects execution platforms and toolchains: one build from a Linux machine and another from a macOS machine may end up different because they are using different tools.

 

I guess the more general question would be - what exactly goes into the computation of the output path described below?

This is actually being reworked currently  by @Stephen Twigg: Stephen, do you plan to expand the documentation about output paths as part of this?

[Jonathan Perry]

Thank you again for your response

[Jonathan Perry]

Hello again!

Yet another question on this topic please…

I am trying to come up with a blanket hammer of ‘any external target should transition back to default’. This will give me a reasonable coarse-grained cut on the propagation of the graph split. 

Can you think of any sane way to get that done please? Or anything close (like making it happen only for java_* targets or something)?

Thank you!

[Jonathan Perry]

A couple of concrete possibilities:

- I find a way to patch the built in java_library etc, and add a transition to all of them conditioned on the workspace of the label. (Any suggestions here much appreciated, I’ve been googling and digging into source and can’t find a way that I could have sworn used to exist)

- we have some traversal control functionality in transitions, to allow us to control the propagation of the split. So a transition gets a new field, say traversal_controller or something, which gets the label (and attrs?) of a candidate target, and is allowed to return ‘CONTINUE’ or ‘STOP’. 

[Jonathan Perry]

Any thoughts on the idea of controlling the traversal?  It seems to my naive mind that this might help mitigate the explosive nature of transitions...

Thank you!

[Jonathan Perry]

Sorry for the spam, but one thought here - if we could set a default transition for all rules whose job was to decide whether to flip back to the default, then I think we could implement the equivalent of the STOP functionality with minimal bazel java changes...  but I can imagine that might raise some performance concerns

[John Cater]

This isn't really the angle we're using to approach handling transition fanout, so I don't think anyone has any useful advice. If you find anything interesting feel free to report back.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/ed989db3-ce88-4f32-bd33-c6bc50e79b53n%40googlegroups.com.

[Jonathan Perry]

Ack, thanks, will do!

Where's the best place for me to get familiar with your latest thoughts please?

[Jonathan Perry]

So I think I answered my own question, which is ‘look at this doc: 

https://docs.google.com/document/d/17snvmic26-QdGuwVw55Gl0oOufw9sCVuOAvHqGZJFr4/edit‘

Which then got me to —experimental_output_paths=strip

So I’m trying that

Am I correct?

Thank you as always for your patience. 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/4110806d-e6c5-473e-8958-c3b424d138a5n%40googlegroups.com.

[Jonathan Perry]

One thing I can’t tell from this doc is whether I should use ‘strip’ or ‘content’. Naively, ‘content’ seems better (and indeed appears to work, even for my custom rules), but…how should I choose please?

[Fabian Meumertzheim]

I don't think that 'content' is fully implemented. You can get a sense of what is working from this BazelCon talk: https://youtu.be/Et1rjb7ixUU

+Greg Estren 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU348iXF7KR3HFQD4fBfHP7wkLZvh6v7LVHfdfi%2BU6B3S1A%40mail.gmail.com.

[Jonathan Perry]

Brilliant thank you

[Greg Estren]

"content" is not being planned. It's an interesting idea but really design-invasive. We're putting effort in that space into what Fabian linked.

Re: controlling flag propagation, we have all kinds of thoughts on that! We're fleshing out some 2024 ideas of a stronger propagation model for flags, vs. the historic default of "everything propagates everywhere". One concrete subset of this is controlling target -> exec propagation where we've already added more flexibility.

The best place to continue this discussion is probably here. All of this really depends on what flags we're talking about and where exactly we want them to propagate. Starlark flags are different than --platforms is different than --copt, for example. Also whether you want to control this on a flag-by-flag basis vs. a whole collection of flags at once.

[Jonathan Perry]

Thank you for the response.  Would you be interested in a call to discuss our usecase?

[Jonathan Perry]

I understand that a request for a call is presumptuous, but I promise there are reasons. 

I am interested in a use case which:

- uses transitions to control a single label_flag

- consumes that label_flag exclusively within a single repository (which is the repo in which the flag is defined), through a well-known set of rules

- has at least tens of settings for that label_flag

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/cccb5d9c-9124-42bf-8f6e-8acaca30b14bn%40googlegroups.com.

[Jonathan Perry]

Two more I think interesting points

- the label_flag’s value is used in exec and target (specifically it would be consumed by both a java_library dep of a build target as well as by a java_binary used as a tool by a custom rule)

- the label_flag’s value is never used in a control flow expression of a rule’s implementation (dunno if that’s a thing but it feels like it might be?)

[Jonathan Perry]

Greg- is this basically the data you wanted? 

Thanks!

[Jonathan Perry]

Pinging this thread

[Jonathan Perry]

Pinging again - is there any more useful info I can give you?

[Jonathan Perry]

Is there somewhere else I should be pinging? I would really love to give any data that would be useful. Thanks. 

[Jonathan Perry]

Ping. 

[Jonathan Perry]

Ping

[Jonathan Perry]

Just resurrecting this thread to try to cross-pollinate ideas.  A chunk of the difficulty with transitions as far as I understand it is to do with toolchains and the fact that they generally speaking depend on ambient artifacts from the filesystem.  This means that bazel can't use its normal strategies to guarantee correctness.  Fair enough.

A related problem I've been trying to find a decent solution to has been tight, efficient and correct integration between Nix and bazel.  My thought is that Nix brings a huge number of non-bazelified dependencies to the table, with a large group of maintainers.  The overall philosophy of the two projects are extremely well-aligned.  A tight integration here would allow us to precisely model so much more of our entire build and runtime environment, with the utopia of a correct and minimal remote recompilation of the universe upon an uptick to any dependency at all. 

I think this is achievable (with some effort of course!), and the upsides are compelling.

Of course it doesn't have to be Nix precisely, but perhaps more generally some uptick to various pieces of bazel to incorporate support for 'execution platforms that can safely fulfill dependency requirements from precise inputs', including the propagation of platform requirements through to consumers.

Does this sound inspiring?

[Filip Filmar]

On Sat, Aug 17, 2024 at 1:23 AM Jonathan Perry <jonp...@gmail.com> wrote:

Does this sound inspiring?

Yes. Even better, it's already done.

https://hdlfactory.com/post/2024/04/20/nix-bazel-%EF%B8%8F/

F

[Jonathan Perry]

This is amazing work for sure, and thank you for pointing me at it.

As far as I can tell, efficient and correct remote execution remains a difficulty - correct?

[Jonathan Perry]

So for example, this post explains the shenanigans one has to go through to make the two systems work together - https://www.tweag.io/blog/2024-02-29-remote-execution-rules-nixpkgs/.  Again, incredible work given the constraints, but surely two such well-aligned projects would benefit from a far tighter integration...

[Filip Filmar]

On Sat, Aug 17, 2024 at 1:40 AM Jonathan Perry <jonp...@gmail.com> wrote:

As far as I can tell, efficient and correct remote execution remains a difficulty - correct?

I haven't tried remote builds. I did confirm that the setup works without any prepwork on github actions.

I know that this is not RBE. However, finding a setup that works in a sandboxed environment is probably important for getting RBE to work.

I don't know why this couldn't be extended to work with RBE. Let me know if you do.  Here's why I think that. The workspace is fully declared in build files and nix scripts. The ephemeral nix install lives in cache. The only quirk today is that the ephemeral nix lives in its own directory, not in an external repo, but it's still in cache at a repeatable spot. In contrast to what's described in that article, where it's a system-wide nix install which lives a life independent of bazel cache.

What difficulties with remote exec and correctness did you have in mind?

F

 

[Jonathan Perry]

I'm thinking that, for one thing, the sheer size of the zip for each external dep, with no sharing between them, will be prohibitive.  

However I would have to try it to actually stand behind that.

I take your point though that there are things we can do around the edges.  However I still maintain that we are missing a trick in bazel by not allowing for deep integration with like-minded platforms.  I'd love to get engagement from the bazel maintainers to take your and tweag's amazing work and push it way further down the stack.

[Filip Filmar]

On Sat, Aug 17, 2024 at 2:12 AM Jonathan Perry <jonp...@gmail.com> wrote:

I'm thinking that, for one thing, the sheer size of the zip for each external dep, with no sharing between them, will be prohibitive.  

I am fairly convinced that you will have this issue with any RBE setup that does not rely on a shared cache.

So it's not clear to me why it would be uniquely difficult in this setup.

 

 I'd love to get engagement from the bazel maintainers [...]

Can't speak to bazel maintainers, but I personally appreciated that something like this was possible without doing surgery on bazel.

And if this doesn't work for you, you can invert the entire thing and package all your deps in a docker container instead of having them live "around" your build environment; then invoke individual build steps there. See: https://github.com/filmil/bazel-rules-bid That should work out of the box if you can run docker on your build farm and then docker is your cache.

F

[Jonathan Perry]

On Sat, Aug 17, 2024 at 10:22 AM Filip Filmar <fil...@gmail.com> wrote:

On Sat, Aug 17, 2024 at 2:12 AM Jonathan Perry <jonp...@gmail.com> wrote:

I'm thinking that, for one thing, the sheer size of the zip for each external dep, with no sharing between them, will be prohibitive.  

I am fairly convinced that you will have this issue with any RBE setup that does not rely on a shared cache. 

So it's not clear to me why it would be uniquely difficult in this setup.

Because when bazel understands the dependency graph, it downloads each one once.   As a concrete example, imagine you have 2 toolchains that both use the JDK: you would zip the entire JDK twice and distribute it twice.

 

 I'd love to get engagement from the bazel maintainers [...]

Can't speak to bazel maintainers, but I personally appreciated that something like this was possible without doing surgery on bazel.

Totally!  I'm not dismissing the work so far at all, as I keep saying, it's amazing. 

And if this doesn't work for you, you can invert the entire thing and package all your deps in a docker container instead of having them live "around" your build environment; then invoke individual build steps there. See: https://github.com/filmil/bazel-rules-bid That should work out of the box if you can run docker on your build farm and then docker is your cache.

The performance of invoking a docker run for every build step would be completely prohibitive.  Further this doesn't solve the problem of "I added a dependency on a rule that uses toolchain X", as it can't transparently and reproducibly add the dependency to the docker image, docker being what it is.

F

Wow, you have done a ton of interesting work in this space.  Thank you for that and thank you for engaging in this thread. 

[Jonathan Perry]

No, the idea is that they are two separate toolchains but with shared internal deps - understood by nix but not by bazel.

On Sat, Aug 17, 2024 at 10:48 AM Manuel <naranjo...@gmail.com> wrote:

El sáb, 17 ago 2024, 11:30, Jonathan Perry <jonp...@gmail.com> escribió:

On Sat, Aug 17, 2024 at 10:22 AM Filip Filmar <fil...@gmail.com> wrote:

On Sat, Aug 17, 2024 at 2:12 AM Jonathan Perry <jonp...@gmail.com> wrote:

I'm thinking that, for one thing, the sheer size of the zip for each external dep, with no sharing between them, will be prohibitive.  

I am fairly convinced that you will have this issue with any RBE setup that does not rely on a shared cache. 

So it's not clear to me why it would be uniquely difficult in this setup.

Because when bazel understands the dependency graph, it downloads each one once.   As a concrete example, imagine you have 2 toolchains that both use the JDK: you would zip the entire JDK twice and distribute it twice.

I'm jumping quite late in the thread, and I'm a bit lazy to read it all TBH, but wouldn't this 2 toolchains be the same from a hashing perspective? If that's the case then client will push to the cache once and the RBE workers will pull it twice, they may not need to pull it if they have downloaded before. RBE is dependant on Remote caching, you can't have RBE without RC 

[Filip Filmar]

On Sat, Aug 17, 2024 at 2:30 AM Jonathan Perry <jonp...@gmail.com> wrote:

Because when bazel understands the dependency graph, it downloads each one once.   As a concrete example, imagine you have 2 toolchains that both use the JDK: you would zip the entire JDK twice and distribute it twice. [...]  The performance of invoking a docker run for every build step would be completely prohibitive. Further this doesn't solve the problem of "I added a dependency on a rule that uses toolchain X", as it can't transparently and reproducibly add the dependency to the docker image, docker being what it is.

I have not found the above issues to be a problem at least in what I was trying to do. Do you have a specific project and specific requirements that you are tackling?

If that's the case then client will push to the cache once and the RBE workers will pull it twice, they may not need to pull it if they have downloaded before. RBE is dependant on Remote caching, you can't have RBE without RC  

No, the idea is that they are two separate toolchains but with shared internal deps - understood by nix but not by bazel. 

Do you have a specific need where this is a practical issue?

F

[Jonathan Perry]

On Sat, Aug 17, 2024 at 11:59 PM Filip Filmar <fil...@gmail.com> wrote:

On Sat, Aug 17, 2024 at 2:30 AM Jonathan Perry <jonp...@gmail.com> wrote:

Because when bazel understands the dependency graph, it downloads each one once.   As a concrete example, imagine you have 2 toolchains that both use the JDK: you would zip the entire JDK twice and distribute it twice. [...]  The performance of invoking a docker run for every build step would be completely prohibitive. Further this doesn't solve the problem of "I added a dependency on a rule that uses toolchain X", as it can't transparently and reproducibly add the dependency to the docker image, docker being what it is.

I have not found the above issues to be a problem at least in what I was trying to do. Do you have a specific project and specific requirements that you are tackling?

Yep.  I can't really get into it much, but suffice it to say that the project is:

 - Large and growing

 - a monorepo

 - used to build and deploy services to several different environments with different toolchain needs

 - entirely built by remote execution

If that's the case then client will push to the cache once and the RBE workers will pull it twice, they may not need to pull it if they have downloaded before. RBE is dependant on Remote caching, you can't have RBE without RC  

No, the idea is that they are two separate toolchains but with shared internal deps - understood by nix but not by bazel. 

Do you have a specific need where this is a practical issue?

F

Sure - gcloud and python.  Or intellij and...well the universe.  Or python and all-the-things-you-can-call-with-python.

I guess I'm barking up the wrong tree here.  I'll go away and try your solution.  Thanks again for your engagement and your work in this area.

[Manuel]

El sáb, 17 ago 2024, 11:30, Jonathan Perry <jonp...@gmail.com> escribió:

On Sat, Aug 17, 2024 at 10:22 AM Filip Filmar <fil...@gmail.com> wrote:

On Sat, Aug 17, 2024 at 2:12 AM Jonathan Perry <jonp...@gmail.com> wrote:

I'm thinking that, for one thing, the sheer size of the zip for each external dep, with no sharing between them, will be prohibitive.  

I am fairly convinced that you will have this issue with any RBE setup that does not rely on a shared cache. 

So it's not clear to me why it would be uniquely difficult in this setup.

Because when bazel understands the dependency graph, it downloads each one once.   As a concrete example, imagine you have 2 toolchains that both use the JDK: you would zip the entire JDK twice and distribute it twice.

I'm jumping quite late in the thread, and I'm a bit lazy to read it all TBH, but wouldn't this 2 toolchains be the same from a hashing perspective? If that's the case then client will push to the cache once and the RBE workers will pull it twice, they may not need to pull it if they have downloaded before. RBE is dependant on Remote caching, you can't have RBE without RC