Incompatible changes policy

341 views
Skip to first unread message

Lukács T. Berki

unread,
Aug 29, 2018, 3:14:22 AM8/29/18
to bazel-discuss
Hey there,

Continuing the thread on bazel-dev (which I should arguably have started here in the first place), let's discuss about how we should roll out incompatible changes to Bazel, that is, ones that have a change to break some builds.

Summarizing the above thread:
  1. Breaking changes are bad because no one wants to have their code broken. Thus, the less frequent breaking changes are, the better.
  2. We need to make some breaking changes to Bazel. The more frequently we can make breaking changes, the faster we can develop Bazel (this, of course, directly contradicts (1)
  3. We have a lot of functionality we want to remove from Bazel that are mostly misfeatures it inherited from Blaze, its Google-internal version
  4. Starlark already has an incompatible changes policy, to which they hold themselves (but the rest of Bazel does not)
  5. If we do breaking changes, we should at least have a way to warn people before in a way that is as well-focused as possible (i.e. these warnings are only seen by people who want it and who can act on it)
  6. Unlike at Google, we can't know all the places that would be broken by any given changes (unless we give in to the siren song of on-by-default telemetry, and even then, we can't catch every case)
  7. The easier it is to keep up to date with the latest Bazel version, the more likely it is that all of your dependencies will be compatible with it and thus won't stop you from upgrading

Trying to take into account all the replies on that thread, here is a proposal:
  1. When we want to do an incompatible change, we always put it behind a flag. Release N would add the flag and default to the old behavior and release N+1 would remove the flag and enforce the new behavior (note that this is different from the current Starlark policy and allows us to remove the old functionality a release earlier)
  2. We publish a list of flags we want to flip in any given release.
  3. We add some functionality to Bazel (for example, behind a flag) that would tell which places need to be updated to stay compatible (if possible for the flag in question). Its output would be, for example, like this: "the next Bazel version will flip the flag --incompatible_kitten_paws. These locations need to be updated: alice/BUILD:123, bob/charlie.bzl:45 and @dave//erica:fred.bzl (@dave is maintained by geo...@harvey.com)
  4. We publish a list of functionality that we would like to remove before we call Bazel 1.0 (we might also make the removal of this functionality a sufficient condition for calling Bazel 1.0, but that's a different discussion)
  5. At first, we allow incompatible changes in every Bazel version. A while later, once the functionality stabilizes, start ratcheting this down so that we have incompatible changes less and less frequently (next step could be, for example, every three months). Eventually, we only allow incompatible changes in major releases.
How does this sound?







--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Lukács T. Berki

unread,
Aug 29, 2018, 7:56:45 AM8/29/18
to bazel-discuss, Dmitry Babkin
Dmitry kindly reminded me of a hole in the above plan: namely, how do we keep ci.bazel.build green after the change in the default behavior its Bazel HEAD? In particular, we can't mandate that the code be updated right after the release containing the flag (without the default behavior change) is cut because projects on ci.bazel.build things need to work with the last released Bazel (defaulting to the old behavior) and Bazel at HEAD (defaulting to the new behavior).

My best idea is that we try to make it so that --incompatible_* flags only remove features. Then, projects on ci.bazel.build can migrate to whatever alternative we come up with for the removed features and thus will work with both the last release and HEAD. Failing that, we'll have to make it so that the presence or the absence of the feature can be detected (for example, by looking for a particular Skylark symbol).


Laurent Le Brun

unread,
Aug 29, 2018, 9:05:58 AM8/29/18
to Lukács T. Berki, bazel-discuss, Dmitry Babkin
Projects on the CI should work with:
 - Bazel at head
 - the latest release of Bazel
 - the latest release of Bazel with --all_incompatible_changes

It's possible that they fail with Bazel at head with --all_incompatible_changes, but we should make sure they are updated quickly.

--incompatible_ flags are not just for removing features, but also for changing behavior. However, it should be made so that users can have code compatible with both the legacy and the new behaviors.


--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOu%2B0LWecfCHLBcHDG%3D85CKe3KLSLsk2PFmKOm4pqeH-cG2rWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
Laurent

Tony Aiuto

unread,
Aug 29, 2018, 5:18:39 PM8/29/18
to Laurent Le Brun, Lukács T. Berki, bazel-...@googlegroups.com, Dmitry Babkin
IIRC, we were considering taking popular community rule sets into Bazel CI, so we can see a broader range of potential breakages.
I think we will have to allow those rules to fail longer with --all_incompatible_changes, so that the rule owner has a chance to update.
The open question is if their failure to do so in some reasonable time frame would block a Bazel release.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFtxR%2BmkJp-rMDSWxX-7bMK4XxhLFKgyhmtbFxzeg5oYuJ0HDQ%40mail.gmail.com.

Laurent Le Brun

unread,
Aug 30, 2018, 12:31:50 PM8/30/18
to Tony Aiuto, Lukács T. Berki, bazel-discuss, Dmitry Babkin
I'd like to point out that all significant changes must have a design doc. All design docs are supposed to have a backward compatibility section. (https://bazel.build/designs/index.html)

For 1. I'd like to make it a bit stronger and consistent with the Starlark policy (https://docs.bazel.build/versions/master/skylark/backward-compatibility.html). Release N+1 should have the flag switched off. This provides a workaround for users who didn't notice the breakage earlier. At least, we make sure they can upgrade and continue using Bazel. This helps with our desire to have all users using the latest version of Bazel.

I understand that it can add too much burden in some cases. So I suggest we keep the rule used by Starlark (and experimented for more than a year) as the default, but authors can request exceptions in their design doc.


For 5. this looks reasonable to me. We should try to reduce the frequency. For example, we should try to have at least one backward compatible release in the next few months. This will give us experience and more confidence, before we can get to Bazel 1.0.

Overall, your proposal looks good, Lukács!

ittai zeidman

unread,
Aug 31, 2018, 1:54:50 AM8/31/18
to bazel-discuss
Should we do release patches if there is a problem with a flag?
One of the java common constructs was deprecated without giving a solution to a common use case.
We (rules_scala) waited with the upgrading to the release for some time and then needed to turn off the incompatible checks when a user needed the newest one.
The issue is now sorted but now we are behind since I think we have several compatibility checks broken and it’s hard to tackle all of them.
My point is that maybe this kind of issue (bug?) should cause a patch release with the removal of the deprecation.
Wdyt?

Lukács T. Berki

unread,
Aug 31, 2018, 5:23:14 AM8/31/18
to Laurent Le Brun, Tony Aiuto, bazel-discuss, Dmitry Babkin
On Thu, Aug 30, 2018 at 6:31 PM, Laurent Le Brun <laur...@google.com> wrote:
I'd like to point out that all significant changes must have a design doc. All design docs are supposed to have a backward compatibility section. (https://bazel.build/designs/index.html)

For 1. I'd like to make it a bit stronger and consistent with the Starlark policy (https://docs.bazel.build/versions/master/skylark/backward-compatibility.html). Release N+1 should have the flag switched off. This provides a workaround for users who didn't notice the breakage earlier. At least, we make sure they can upgrade and continue using Bazel. This helps with our desire to have all users using the latest version of Bazel.
I would argue that if one wants to keep up to date with Bazel, they should be expected to have some situational awareness. In particular, the breakage when the flag is removed in release N+2 could have been noticed at the least at the following points:
  1. At the announcement of release N+1
  2. When running release N+1 with the "pretend all incompatible flags are flipped and tell me if anything is wrong" flag
  3. With the release candidate N+2
I don't think adding another layer of defense helps a lot, does it? It does add an "emergency hatch" (i.e. "crap, my build broken, I want to get it back up") but it comes at the cost of semantics fragmentation (what if one of your dependencies requires the flag to be flipped?).

 

I understand that it can add too much burden in some cases. So I suggest we keep the rule used by Starlark (and experimented for more than a year) as the default, but authors can request exceptions in their design doc.
Mind you, even though I'm arguing against a less conservative approach, it's possible that what Starlark is doing now is better. But adopting that policy is a big change for everyone not working on Starlark and I have already heard a lot of pushback. How about we go gradually and adopt my approach, the less conservative one first, and if that is still not enough, we reconsider? That would also give us time to remove the all the cruft we want to remove before Bazel 1.0 more quickly.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

--
Laurent

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

Liam Miller-Cushon

unread,
Sep 7, 2018, 1:56:45 PM9/7/18
to bazel-discuss
How does this work for startup options? I don't think --all_incompatible_changes is sufficient there?

Paul Draper

unread,
Sep 9, 2018, 8:27:04 PM9/9/18
to bazel-discuss
--all_incompatible_changes", the owners have to fix it quickly (or 
report why it's difficult or not feasible). 

FYI, a common reason not to fix it immediately, is....compatibility.

(1) Downstream projects are reliant on upstream projects to update. As the community expands, this dependency chain get deeper, and --all_incompatible_changes takes longer to propagate. Similar discussion: https://groups.google.com/forum/#!topic/bazel-discuss/WeeIqzty2bE

Rule libraries have additional challenges, since restrictive Bazel requirements on their users is a fresh version of dependency hell.  IMO any well-maintained rule library should maintain compatibility for at least the last 2-3 months of Bazel.

(2) Changes happen suddenly. For example, IIRC `--incompatible_disallow_data_transition` was introduced in 0.16.0 (July 31) which disallows `cfg = "data"`. But in 0.14.1 (Jun 8), `cfg = "data"` was not only compatible but in fact required. And it is not easy to change this based on Bazel version.

(3) Testing with incompatible flags across Bazel versions can be challenging. It is the choice between (a) listing --incompatible_* and (b) --all_incompatible_changes and excepting --incompatible_* . Both have difficulties since Bazel breaks if there is a --incompatible_* flag it does not recognize. For rules_scala_annex, I chose (b), plus version-detection in tools/bazel and selectively loading version-specific RC files https://github.com/andyscott/rules_scala_annex/tree/master/tools

I say these things to point out (the perhaps obvious truth) that uncompromising `--all_incompatible_changes` is identical to no compatibility flags at all. 

Laurent Le Brun

unread,
Sep 10, 2018, 9:54:45 AM9/10/18
to Paul Draper, bazel-discuss
Data point: the change in https://github.com/bazelbuild/bazel/issues/5827 is causing a significant breakage, although we provided much more time than any policy we've ever discussed here.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/704022c8-2e10-474e-aaf8-a280daa6e575%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Laurent

Lukács T. Berki

unread,
Sep 10, 2018, 10:33:04 AM9/10/18
to Laurent Le Brun, Paul Draper, bazel-discuss, Dmitry Lomov
@Paul Draper: in general, the larger the community is, the more difficult these changes will be for the exact reason you mention. What I'm hoping is that by adopting a policy where incompatible changes can only be done after a short migration period, people will get used to the idea (both people who develop Bazel and thus have to deal with flags and migration periods and both people who use Bazel and thus have to update their code and nag their dependencies to update). And then, once these things work, we can dial down the speed of changes, eventually culminating in a rule "we don't do breaking changes except when bumping major version number". 

dslomov@ made a good point, namely that there are two kinds of incompatible changes: the ones where it's feasible to build replacement functionality, migrate to the new functionality, then remove the old one (e.g. replacing one API with another) and ones where we can't have such a migration period (e.g. changing the behavior of some Skylark function or as it happened recently, one of our sandboxes).

In the first case, it's perfectly reasonable to have a version of Bazel where both the new and the old functionality are available without flags (i.e. neither a --experimental_* nor a --incompatible_* flag is required), but we can't have the same with the second one. Of course, it follows that the second type of changes should be minimized, but we can't reduce those to zero. 

There is also the problem that the more flags we have, the more unexpected ways they can interact and thus the less coverage we can have. Thus, that's an argument for minimizing their number. 

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.


--
Laurent

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFtxR%2BmXg8cvA%2BTJq0JwqDe0z1vY25kZDRJgdscqja4SEziApg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

Tony Aiuto

unread,
Sep 10, 2018, 6:13:37 PM9/10/18
to bazel-discuss
What makes this problem hard is that the "Bazel which end users care about" is not just the Bazel core code at bazelbuild/bazel.

From a practical point of view, you start with the core Bazel, the Starlark language and their integration (the whole of which we will just call Bazel), then you immediately add in rules_{many} for the set of languages you use. Large projects tend to quickly add languages which they might not even know they are dependent on - and fundamentally do not care about. For example, a Java project might require C++ compilation to build a code generator, from a third party component, while no developer in the organization uses C++ for anything else.

If I put myself in the shoes of a Scala developer, I don't want to make the fine distinction between the release cycles and policies of Bazel, rules_java and rules_scala. I simply need the union of all my binaries and rule sets to work together.

I can easily see this scenario
  • I depend on rules_a and rules_b
  • rules_a is very active. They stay on top of Bazel changes as soon as they are mare. They set --all_incompatible_changes in their .bazelrc files.
  • rules_b has a less active maintainer. They build and test with the "out-of-box" Bazel config. That is, they never use --all_incompatible_changes.
Then we can look at even a minor change: Load must appear at top of file. This is a "migration" kind of rule, where you have to change all the .bzl files and someday you can turn it off.

The rules_a team is going to account for this immediately.
The rules_b team is going to sit on this until it fails on some future Bazel release.

If we wanted to place blame, we could say the rules_b team is at fault. That is not productive. From the end user's point of view, some rules stop working at some Bazel release. They have to check rules_b for updates, and hope that the problem is fixed in their release. They think, "Why does the Bazel ecosystem let this happen?"

The point of having a release policy for changes is to attempt to see problems ahead of time. It should maximize the amount of time we can give a user or rule maintainer to spot an upcoming problem and take corrective action. This is why I keep arguing that deprecation warnings are a good thing. In the scenario above, there should be a deprecation warning on load() not being at the top. Maybe the end user can not correct rules_b, but they at least can see there is a looming problem with it. Even better, the lazy rules_b team will see the warning (since they presumably actually use the rules they maintain).

For behavior changes we will certainly need a staged release cycle, with an --experimental_new_behavior=false, then =true, then removed. But with enough rule sets and different levels of support from the rule maintainers, it is inevitable that some user will be in the situation where they depend on rule sets such that one requires the flag to be false, while the other requires it to be true. No policy that the Bazel team can implement on its own can prevent that. What we could do is:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.
  • all code hosted on github/bazelbuild also must have some sort of integration test in buildkite (and thus test with all the new experiments)
Of course, this does little for organizations that build their own rules or depend on rule sets provided by other products. For them, I fear, long cycles between the time a breaking change is available and when it is required is the best solution. We might do it in just 3 releases (flag=false, flag=true, remove flag), but those might be Bazel N, N+4, N+8. I can hear myself screaming that this is going to slow me down to much. Taking 8 releases to removed the old code is horrible. What we should add to design review is a risk and failure recovery analysis.  For a change like load() positioning, N,N+1,N+2 is fine. The fix to .bzl files should be trivial. For a subtle semantic change, perhaps the release deltas should be larger than 1. This gives us the flexibility to do low risk things more quickly than high risk. We would also want to pre-announce the change plans on a regular basis, outside the scope of the design review. We can't expect users to read design proposals. So, release notes will talk about a new experimental flag, but also give an idea of how many releases/months/quarters until that flag changes value and eventually goes away.

Philipp Wollermann

unread,
Sep 10, 2018, 6:38:01 PM9/10/18
to Tony Aiuto, bazel-discuss
On Tue, Sep 11, 2018 at 12:13 AM 'Tony Aiuto' via bazel-discuss <bazel-...@googlegroups.com> wrote:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.

On our Buildkite CI we currently don't test with this flag at all, ever. I personally don't know anyone who has ever used that flag (but I guess it might be popular with rule authors, considering that Starlark is where most of the --incompatible flags came from until now?).

Laurent recently filed a bug that we should start doing so for downstream projects, so we're about to fix the CI situation. :) However, it's not completely clear to me how it should look like:

- Should we test Bazel itself also with --all_incompatible_flags? For the actual Java unit tests and integration tests it "shouldn't" matter, because we should just have tests for the behavior with and without the flag. However, for some changes, "Bazel building itself" or "Bazel running its test suite" is actually part of making sure that it works - think about a flag that affects Bazel's resource scheduling or a flag that affects our test strategy. So maybe yes?

- If yes: On presubmit, too? This would mean we have to provision twice the hardware resources as we currently do. Expensive (omg, more iMac Pros), but OTOH people *really* don't like it when their changes are rolled back even though the presubmit passed (even more so when the breakage comes from a downstream project that they don't even know how to build / test on their local machine, making it hard to debug the breakage).

- Should we test all downstream projects with --all_incompatible_flags? I guess so and the linked issue says so, too. However note that this will double the number of results on this already overloaded UI: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/433 - one could argue that before we add even more breakages on top of that pile, we should - at least once, for the love of Bazel - try to get the current downstream projects pipeline green. AFAIK since inception of our Buildkite CI, this pipeline has never been green.

Cheers,
Philipp

Ofer Bartal

unread,
Sep 10, 2018, 6:45:30 PM9/10/18
to Philipp Wollermann, Tony Aiuto, bazel-discuss
Here are some ideas which may help with parts of this:
* We can't see or change our clients code, but for some breaking changes, we might be able to offer an automated fix. Take Java as an example, error_prone, which is already integrated into Bazel, can automatically create patches to fix code (see example here). If we break an API in a way that we can statically analyse, we might be able to offer a fix, and then "allow ourselves" to expect faster fixes from Bazel users.
* If people don't notice our deprecation warnings until they actually break them, we could use this "trick": Make it an error, and describe in the error that it can be turned into a warning (until a later release when it'll turn back to error) by adding a flag to bazelrc. 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BAhZojfMg7D5D7GzaD_a%3DLcpAg0aSC4kPOCn_EZ_xDFTeYBrg%40mail.gmail.com.

Laurent Le Brun

unread,
Sep 10, 2018, 7:18:53 PM9/10/18
to Ofer Bartal, Philipp Wollermann, Tony Aiuto, bazel-discuss
> For a change like load() positioning, N,N+1,N+2 is fine.

Actually, no. For an example of a trivial change, see https://github.com/bazelbuild/bazel/issues/5827
I tried to give a lot of notice, but that didn't even worked. Maybe advertising --all_incompatible_changes would help. Maybe testing it on the CI, and filing bugs against downstream projects will be enough.

But still: once proto rules are updated, everything that depends on them needs to update their version of the proto rules (and so on, recursively). To make things more bearable, we could synchronize the breaking changes. Some Bazel releases have to be backward-compatible, so that we don't require the recursive updates every month.

And yes, we have to improve tooling. Tooling to automatically detect and fix the syntactic issues (we're working on it), and tooling to help update the WORKSPACE files.


To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CA%2BndrjrJ_mKyugnPKSew36SctzFWQvCoaBiWh2oB7O%2BuhoioKQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--
Laurent

ittai zeidman

unread,
Sep 10, 2018, 11:23:44 PM9/10/18
to Laurent Le Brun, Ofer Bartal, Philipp Wollermann, Tony Aiuto, bazel-discuss
+1 re PACKAGE_NAME example- I was the one who submitted the fix to protobuf on master since, at the time, rules_scala ran with this flag on.
It wasn’t a good experience to try and understand who’s broken and then checkout protobuf, understand how to fix them and wait for someone on their side to take notice (which took a bit of time).
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFtxR%2BnZOTq0kygGPK8vXaG%2BuZoA37LJZ5tqy8-j37zb%3DYH6Jg%40mail.gmail.com.

Lukács T. Berki

unread,
Sep 11, 2018, 3:36:37 AM9/11/18
to ittai zeidman, Laurent Le Brun, Ofer Bartal, Philipp Wollermann, Tony Aiuto, bazel-discuss
The incompatible flag that removed PACKAGE_NAME has been there since ages, and there was *still* breakage on ci.bazel.build which is an indication that people either don't update their code or they don't update their dependencies, even when given a lot of time. In other words, time does not help. Then what is the utility of being generous with it? In this case what would have helped is a version of ci.bazel.build that runs with a set of incompatible changes and the understanding that projects there are expected to fix themselves and bug their dependencies to do so. As long as we get that from any policy, we are at a good place. 

Of course, it would be the best if we had breaking changes less frequently, but realistically, we cannot do that now. What we *can* do now is to instigate a policy that doesn't constrain the speed at which we can remove bad functionality much, and then once we see the number of backward-incompatible changes decreasing, we can make that policy stricter and stricter, eventually culminating in "no backward incompatible changes, ever". 

In other words, I'm proposing a covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down.

How does that sound?



On Tue, Sep 11, 2018 at 5:23 AM, ittai zeidman <itt...@gmail.com> wrote:
+1 re PACKAGE_NAME example- I was the one who submitted the fix to protobuf on master since, at the time, rules_scala ran with this flag on.
It wasn’t a good experience to try and understand who’s broken and then checkout protobuf, understand how to fix them and wait for someone on their side to take notice (which took a bit of time).
On Tue, 11 Sep 2018 at 2:18 'Laurent Le Brun' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
> For a change like load() positioning, N,N+1,N+2 is fine.

Actually, no. For an example of a trivial change, see https://github.com/bazelbuild/bazel/issues/5827
I tried to give a lot of notice, but that didn't even worked. Maybe advertising --all_incompatible_changes would help. Maybe testing it on the CI, and filing bugs against downstream projects will be enough.

But still: once proto rules are updated, everything that depends on them needs to update their version of the proto rules (and so on, recursively). To make things more bearable, we could synchronize the breaking changes. Some Bazel releases have to be backward-compatible, so that we don't require the recursive updates every month.

And yes, we have to improve tooling. Tooling to automatically detect and fix the syntactic issues (we're working on it), and tooling to help update the WORKSPACE files.


On Tue, 11 Sep 2018 at 00:45, 'Ofer Bartal' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
Here are some ideas which may help with parts of this:
* We can't see or change our clients code, but for some breaking changes, we might be able to offer an automated fix. Take Java as an example, error_prone, which is already integrated into Bazel, can automatically create patches to fix code (see example here). If we break an API in a way that we can statically analyse, we might be able to offer a fix, and then "allow ourselves" to expect faster fixes from Bazel users.
* If people don't notice our deprecation warnings until they actually break them, we could use this "trick": Make it an error, and describe in the error that it can be turned into a warning (until a later release when it'll turn back to error) by adding a flag to bazelrc. 

On Tue, Sep 11, 2018 at 1:38 AM 'Philipp Wollermann' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
On Tue, Sep 11, 2018 at 12:13 AM 'Tony Aiuto' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.

On our Buildkite CI we currently don't test with this flag at all, ever. I personally don't know anyone who has ever used that flag (but I guess it might be popular with rule authors, considering that Starlark is where most of the --incompatible flags came from until now?).

Laurent recently filed a bug that we should start doing so for downstream projects, so we're about to fix the CI situation. :) However, it's not completely clear to me how it should look like:

- Should we test Bazel itself also with --all_incompatible_flags? For the actual Java unit tests and integration tests it "shouldn't" matter, because we should just have tests for the behavior with and without the flag. However, for some changes, "Bazel building itself" or "Bazel running its test suite" is actually part of making sure that it works - think about a flag that affects Bazel's resource scheduling or a flag that affects our test strategy. So maybe yes?

- If yes: On presubmit, too? This would mean we have to provision twice the hardware resources as we currently do. Expensive (omg, more iMac Pros), but OTOH people *really* don't like it when their changes are rolled back even though the presubmit passed (even more so when the breakage comes from a downstream project that they don't even know how to build / test on their local machine, making it hard to debug the breakage).

- Should we test all downstream projects with --all_incompatible_flags? I guess so and the linked issue says so, too. However note that this will double the number of results on this already overloaded UI: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/433 - one could argue that before we add even more breakages on top of that pile, we should - at least once, for the love of Bazel - try to get the current downstream projects pipeline green. AFAIK since inception of our Buildkite CI, this pipeline has never been green.

Cheers,
Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.


--
Laurent

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFtxR%2BnZOTq0kygGPK8vXaG%2BuZoA37LJZ5tqy8-j37zb%3DYH6Jg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOfK4wWLChzyM627QnYgH3RbQatVfdoc_TP%3DTAvi6xgVZkX-PA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

Dmitry Lomov

unread,
Sep 11, 2018, 9:28:54 AM9/11/18
to Lukács T. Berki, ittai zeidman, Laurent Le Brun, of...@google.com, Philipp Wollermann, Tony Aiuto, bazel-discuss
Folks, this is a great discussion - I am back from my vacation and I have just had time to digest this thread.
To provide some background, I have shared a "Radical Stability for Bazel" document proposing some of the measures discussed here internally.  I now think that sharing this internally was a mistake, or at least I should have sharied it with the larger community earlier - apologies for that. Kudos to Lukacs for starting the public discussion on this, and many thanks for thoughtful contributions from everyone.

To respond to some concerns raised in this thread:
1. Beyond Bazel itself, the rules should be stable too: that is very true, but I think we need to start with Bazel itself. I think that it is very difficult to require any stability from rulesets before Bazel itself is stable. Clarifying stability and breaking change assumptions for rulesets would be the next step (for example, that can be a part of Bazel Rules Curation). It is good to think about that next step already though! - so it is very valuable.
2. No matter what we do, people will not update their projects: I believe this is not a permanent state of the world, but rather a consequence of our own behavior - we have not been upfront and methodical about communicating breaking changes, so our users could not follow up in any kind of methodical way. 

Summarizing and expanding on various proposals here, to get to a more stable place for Bazel, we need four things:
* Policy. We are all converging on 
  1.  everything not under experimental flag is "stable"
  2. changes to stable functionality require incompatible_change flag
  3. for every incompatible_change flag there is a migration window release:
    1. If possible, a release when both deprecated and new functionality is available without any flags (respective --experimental flag is on by default, respective --incompatible_change flag is off by default and both are available)
    2. Failing that, an incompatible_change flag that is off by default and a migration receipt to update code in such a way that it works with that flag both on and off.
    3. Cases where 2) is unfeasible should be exceptional - this will be communicated specially with a lot of warning.
  4. we want to eventually introduce a stability window - a number of releases during which no incompatible changes are allowed. We can start with a size of that window being 1 (i.e. every release can have incompatible changes) and increase that size later.
All of the above rhymes with Lukacs' "covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down."

* Implementation. Bazel contributors need to follow this policy! Instrumental to that, we need an implementation of flags framework in Bazel such that it is easy to add a new flag and access a new flag from any part of the code, and there is a centralized list of such flags.

Going forward, where it feasible we can provide people with migration tools, based on https://github.com/bazelbuild/buildtools.

* Communication. The policy should be published to the website, and more importantly, we need to have a public, easy-to-access list of incompatible changes, their migration plans, and time frames for the change. For that, I suggest that every incompatible_change flag that we add is associated with GitHub issue (with appropriate label) that tracks it progress. That issue can contain the migration plan and can also be used to communicate time frames. On top of that, incompatible changes should be part of release notes and communicated more broadly when appropriate, but having a systematic canonical list, up-to-date and accessible at any time, is critical.

* Verification. We need to setup testing for incompatible changes on ci.bazel.build. The purpose of that testing is two-fold:
  1.  to catch accidental incompatible changes. For that we need to have a set of projects (both rules and end-user projects), fixed in time for a stability window period, that we will test with every release for compatibility. We will update for the next "stable" release. 
  2. to ensure that downstream projects are migrated withing the migration window. Roughly that means "test downstream project at HEAD with latest release  --incompatible_change_x=true for all incompatible changes x that we want to turn on in the next release"
Scenario 1 means: Bazel@HEAD, donwstream@fixed version; scenario 2 means Bazel@release + incompatible changes, downstream@HEAD.
Our current downstream testing is Bazel@HEAD, downstream@HEAD and this is frankly very noisy.

Does this sound about right? Anything on high level that I am missing?
The next step I think is to go ahead and address all four of these areas - there is a fair amount of detail to figure out here.

Cheers,
Dmitry

On Tue, Sep 11, 2018 at 9:36 AM 'Lukács T. Berki' via bazel-discuss <bazel-...@googlegroups.com> wrote:
The incompatible flag that removed PACKAGE_NAME has been there since ages, and there was *still* breakage on ci.bazel.build which is an indication that people either don't update their code or they don't update their dependencies, even when given a lot of time. In other words, time does not help. Then what is the utility of being generous with it? In this case what would have helped is a version of ci.bazel.build that runs with a set of incompatible changes and the understanding that projects there are expected to fix themselves and bug their dependencies to do so. As long as we get that from any policy, we are at a good place. 

Of course, it would be the best if we had breaking changes less frequently, but realistically, we cannot do that now. What we *can* do now is to instigate a policy that doesn't constrain the speed at which we can remove bad functionality much, and then once we see the number of backward-incompatible changes decreasing, we can make that policy stricter and stricter, eventually culminating in "no backward incompatible changes, ever". 

In other words, I'm proposing a covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down.

How does that sound?


On Tue, Sep 11, 2018 at 5:23 AM, ittai zeidman <itt...@gmail.com> wrote:
+1 re PACKAGE_NAME example- I was the one who submitted the fix to protobuf on master since, at the time, rules_scala ran with this flag on.
It wasn’t a good experience to try and understand who’s broken and then checkout protobuf, understand how to fix them and wait for someone on their side to take notice (which took a bit of time).
On Tue, 11 Sep 2018 at 2:18 'Laurent Le Brun' via bazel-discuss <bazel-...@googlegroups.com> wrote:
> For a change like load() positioning, N,N+1,N+2 is fine.

Actually, no. For an example of a trivial change, see https://github.com/bazelbuild/bazel/issues/5827
I tried to give a lot of notice, but that didn't even worked. Maybe advertising --all_incompatible_changes would help. Maybe testing it on the CI, and filing bugs against downstream projects will be enough.

But still: once proto rules are updated, everything that depends on them needs to update their version of the proto rules (and so on, recursively). To make things more bearable, we could synchronize the breaking changes. Some Bazel releases have to be backward-compatible, so that we don't require the recursive updates every month.

And yes, we have to improve tooling. Tooling to automatically detect and fix the syntactic issues (we're working on it), and tooling to help update the WORKSPACE files.


On Tue, 11 Sep 2018 at 00:45, 'Ofer Bartal' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Here are some ideas which may help with parts of this:
* We can't see or change our clients code, but for some breaking changes, we might be able to offer an automated fix. Take Java as an example, error_prone, which is already integrated into Bazel, can automatically create patches to fix code (see example here). If we break an API in a way that we can statically analyse, we might be able to offer a fix, and then "allow ourselves" to expect faster fixes from Bazel users.
* If people don't notice our deprecation warnings until they actually break them, we could use this "trick": Make it an error, and describe in the error that it can be turned into a warning (until a later release when it'll turn back to error) by adding a flag to bazelrc. 

On Tue, Sep 11, 2018 at 1:38 AM 'Philipp Wollermann' via bazel-discuss <bazel-...@googlegroups.com> wrote:
On Tue, Sep 11, 2018 at 12:13 AM 'Tony Aiuto' via bazel-discuss <bazel-...@googlegroups.com> wrote:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.

On our Buildkite CI we currently don't test with this flag at all, ever. I personally don't know anyone who has ever used that flag (but I guess it might be popular with rule authors, considering that Starlark is where most of the --incompatible flags came from until now?).

Laurent recently filed a bug that we should start doing so for downstream projects, so we're about to fix the CI situation. :) However, it's not completely clear to me how it should look like:

- Should we test Bazel itself also with --all_incompatible_flags? For the actual Java unit tests and integration tests it "shouldn't" matter, because we should just have tests for the behavior with and without the flag. However, for some changes, "Bazel building itself" or "Bazel running its test suite" is actually part of making sure that it works - think about a flag that affects Bazel's resource scheduling or a flag that affects our test strategy. So maybe yes?

- If yes: On presubmit, too? This would mean we have to provision twice the hardware resources as we currently do. Expensive (omg, more iMac Pros), but OTOH people *really* don't like it when their changes are rolled back even though the presubmit passed (even more so when the breakage comes from a downstream project that they don't even know how to build / test on their local machine, making it hard to debug the breakage).

- Should we test all downstream projects with --all_incompatible_flags? I guess so and the linked issue says so, too. However note that this will double the number of results on this already overloaded UI: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/433 - one could argue that before we add even more breakages on top of that pile, we should - at least once, for the love of Bazel - try to get the current downstream projects pipeline green. AFAIK since inception of our Buildkite CI, this pipeline has never been green.

Cheers,
Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.


--
Laurent

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFtxR%2BnZOTq0kygGPK8vXaG%2BuZoA37LJZ5tqy8-j37zb%3DYH6Jg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOfK4wWLChzyM627QnYgH3RbQatVfdoc_TP%3DTAvi6xgVZkX-PA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOu%2B0LW1B6r%2BWLdf--eM6V%2BmOUNKQsV4KnxUrLgbchRZABTCwg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--
Google Germany GmbH
Erika-Mann-Straße 33, 80636 München, Germany

Jon Brandvein

unread,
Sep 11, 2018, 10:27:39 AM9/11/18
to bazel-discuss
I like the sound of a lot of that, but which steps of the flag dance require waiting for the end of a migration window? If we consider that someone's workflow can be broken by 1) adding a new incompatible change with default false (for users of --all_incompatible_changes), 2) flipping the default to true (for all other users), and 3) removing the flag (hard breakage), and if we use N = 3 as the stable release window, then for a well-timed feature/cleanup that's (3 steps) * (N-1) + 1 = 7 releases = 7 months before you can delete code.

Centralizing our communication and documentation around incompatible changes sounds like an easy win. Making it more convenient to access these flags in the implementation is also necessary.

Dmitry Lomov

unread,
Sep 12, 2018, 4:41:00 AM9/12/18
to Jon Brandvein, bazel-discuss
On Tue, Sep 11, 2018 at 4:27 PM 'Jon Brandvein' via bazel-discuss <bazel-...@googlegroups.com> wrote:
I like the sound of a lot of that, but which steps of the flag dance require waiting for the end of a migration window? If we consider that someone's workflow can be broken by 1) adding a new incompatible change with default false (for users of --all_incompatible_changes), 2) flipping the default to true (for all other users), and 3) removing the flag (hard breakage), and if we use N = 3 as the stable release window, then for a well-timed feature/cleanup that's (3 steps) * (N-1) + 1 = 7 releases = 7 months before you can delete code.

Well, the key thing with migration windows (where people can migrate their code using bazel without flags for both old and new way of doing things) is that no one should run their dev/production build with --incompatible_change_* or --exeperimental_* flags. These flags are there to facilitate experimentation and migration, but all real builds (including builds for a dev organization at large) should be without flags. Thus, an addition/removal of incompatible_change/experimental flag is not per se a breaking change, and the old code can be removed immediately after stability window closes.
 

For more options, visit https://groups.google.com/d/optout.

Lukács T. Berki

unread,
Sep 14, 2018, 6:45:13 AM9/14/18
to Dmitry Lomov, ittai zeidman, Laurent Le Brun, Ofer Bartal, Philipp Wollermann, Tony Aiuto, bazel-discuss
On Tue, Sep 11, 2018 at 3:28 PM, Dmitry Lomov <dsl...@google.com> wrote:
Folks, this is a great discussion - I am back from my vacation and I have just had time to digest this thread.
To provide some background, I have shared a "Radical Stability for Bazel" document proposing some of the measures discussed here internally.  I now think that sharing this internally was a mistake, or at least I should have sharied it with the larger community earlier - apologies for that. Kudos to Lukacs for starting the public discussion on this, and many thanks for thoughtful contributions from everyone.

To respond to some concerns raised in this thread:
1. Beyond Bazel itself, the rules should be stable too: that is very true, but I think we need to start with Bazel itself. I think that it is very difficult to require any stability from rulesets before Bazel itself is stable. Clarifying stability and breaking change assumptions for rulesets would be the next step (for example, that can be a part of Bazel Rules Curation). It is good to think about that next step already though! - so it is very valuable.
2. No matter what we do, people will not update their projects: I believe this is not a permanent state of the world, but rather a consequence of our own behavior - we have not been upfront and methodical about communicating breaking changes, so our users could not follow up in any kind of methodical way. 

Summarizing and expanding on various proposals here, to get to a more stable place for Bazel, we need four things:
* Policy. We are all converging on 
  1.  everything not under experimental flag is "stable"
  2. changes to stable functionality require incompatible_change flag
  3. for every incompatible_change flag there is a migration window release:
    1. If possible, a release when both deprecated and new functionality is available without any flags (respective --experimental flag is on by default, respective --incompatible_change flag is off by default and both are available)
    2. Failing that, an incompatible_change flag that is off by default and a migration receipt to update code in such a way that it works with that flag both on and off.
    3. Cases where 2) is unfeasible should be exceptional - this will be communicated specially with a lot of warning.
  4. we want to eventually introduce a stability window - a number of releases during which no incompatible changes are allowed. We can start with a size of that window being 1 (i.e. every release can have incompatible changes) and increase that size later.
This sounds reasonable if I'm reading this optimistically -- just to be sure that we are on the same page, can you describe what process it would take to do a breaking change both in the case where it's possible for the old and the new behavior to live in the same release without a flag (e.g. replacing an API with another) and in the case where that's not possible (e.g. changing the behavior a sandbox)?

 
All of the above rhymes with Lukacs' "covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down."

* Implementation. Bazel contributors need to follow this policy! Instrumental to that, we need an implementation of flags framework in Bazel such that it is easy to add a new flag and access a new flag from any part of the code, and there is a centralized list of such flags.

Going forward, where it feasible we can provide people with migration tools, based on https://github.com/bazelbuild/buildtools.

* Communication. The policy should be published to the website, and more importantly, we need to have a public, easy-to-access list of incompatible changes, their migration plans, and time frames for the change. For that, I suggest that every incompatible_change flag that we add is associated with GitHub issue (with appropriate label) that tracks it progress. That issue can contain the migration plan and can also be used to communicate time frames. On top of that, incompatible changes should be part of release notes and communicated more broadly when appropriate, but having a systematic canonical list, up-to-date and accessible at any time, is critical.
I'm somewhat skeptical about the proposed amount of red tape (we have enough of that already -- a GH issue, a migration plan, a web link, and then what else? Consider how much overhead that is for a trivial single-line bugfix), but I agree with the general idea of clear expectations, both of us by our users and by us of our users. I'm thinking of a covenant like this:
  1. Bazel will inform users of impending incompatible changes and endeavors not to make too many of them or too quickly (with the latter being ratcheted down over time)
  2. Bazel will roll back / cherry-pick fixes for unintended breakages
  3. Users will run release candidates and inform us of breakages
  4. *maybe* users will put representative code on ci.bazel.build so that we get informed about breakages immediately and not after an average delay of two weeks (if we have monthly releases)


* Verification. We need to setup testing for incompatible changes on ci.bazel.build. The purpose of that testing is two-fold:
  1.  to catch accidental incompatible changes. For that we need to have a set of projects (both rules and end-user projects), fixed in time for a stability window period, that we will test with every release for compatibility. We will update for the next "stable" release. 
  2. to ensure that downstream projects are migrated withing the migration window. Roughly that means "test downstream project at HEAD with latest release  --incompatible_change_x=true for all incompatible changes x that we want to turn on in the next release"
 
Scenario 1 means: Bazel@HEAD, donwstream@fixed version; scenario 2 means Bazel@release + incompatible changes, downstream@HEAD.
Our current downstream testing is Bazel@HEAD, downstream@HEAD and this is frankly very noisy.
What's "downstream@HEAD" and "downstream@fixed version"? I thought that we could get away with three setups: we'd always use the latest version of a downstream project (e.g. that uses Bazel) and test that against Bazel@HEAD + Bazel@HEAD with --incompatible_* flags. That would give data both on accidental breakages and planned breakages that are not fixed yet.


Does this sound about right? Anything on high level that I am missing?
The next step I think is to go ahead and address all four of these areas - there is a fair amount of detail to figure out here.

Cheers,
Dmitry

On Tue, Sep 11, 2018 at 9:36 AM 'Lukács T. Berki' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
The incompatible flag that removed PACKAGE_NAME has been there since ages, and there was *still* breakage on ci.bazel.build which is an indication that people either don't update their code or they don't update their dependencies, even when given a lot of time. In other words, time does not help. Then what is the utility of being generous with it? In this case what would have helped is a version of ci.bazel.build that runs with a set of incompatible changes and the understanding that projects there are expected to fix themselves and bug their dependencies to do so. As long as we get that from any policy, we are at a good place. 

Of course, it would be the best if we had breaking changes less frequently, but realistically, we cannot do that now. What we *can* do now is to instigate a policy that doesn't constrain the speed at which we can remove bad functionality much, and then once we see the number of backward-incompatible changes decreasing, we can make that policy stricter and stricter, eventually culminating in "no backward incompatible changes, ever". 

In other words, I'm proposing a covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down.

How does that sound?


On Tue, Sep 11, 2018 at 5:23 AM, ittai zeidman <itt...@gmail.com> wrote:
+1 re PACKAGE_NAME example- I was the one who submitted the fix to protobuf on master since, at the time, rules_scala ran with this flag on.
It wasn’t a good experience to try and understand who’s broken and then checkout protobuf, understand how to fix them and wait for someone on their side to take notice (which took a bit of time).
On Tue, 11 Sep 2018 at 2:18 'Laurent Le Brun' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
> For a change like load() positioning, N,N+1,N+2 is fine.

Actually, no. For an example of a trivial change, see https://github.com/bazelbuild/bazel/issues/5827
I tried to give a lot of notice, but that didn't even worked. Maybe advertising --all_incompatible_changes would help. Maybe testing it on the CI, and filing bugs against downstream projects will be enough.

But still: once proto rules are updated, everything that depends on them needs to update their version of the proto rules (and so on, recursively). To make things more bearable, we could synchronize the breaking changes. Some Bazel releases have to be backward-compatible, so that we don't require the recursive updates every month.

And yes, we have to improve tooling. Tooling to automatically detect and fix the syntactic issues (we're working on it), and tooling to help update the WORKSPACE files.


On Tue, 11 Sep 2018 at 00:45, 'Ofer Bartal' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
Here are some ideas which may help with parts of this:
* We can't see or change our clients code, but for some breaking changes, we might be able to offer an automated fix. Take Java as an example, error_prone, which is already integrated into Bazel, can automatically create patches to fix code (see example here). If we break an API in a way that we can statically analyse, we might be able to offer a fix, and then "allow ourselves" to expect faster fixes from Bazel users.
* If people don't notice our deprecation warnings until they actually break them, we could use this "trick": Make it an error, and describe in the error that it can be turned into a warning (until a later release when it'll turn back to error) by adding a flag to bazelrc. 

On Tue, Sep 11, 2018 at 1:38 AM 'Philipp Wollermann' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
On Tue, Sep 11, 2018 at 12:13 AM 'Tony Aiuto' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.

On our Buildkite CI we currently don't test with this flag at all, ever. I personally don't know anyone who has ever used that flag (but I guess it might be popular with rule authors, considering that Starlark is where most of the --incompatible flags came from until now?).

Laurent recently filed a bug that we should start doing so for downstream projects, so we're about to fix the CI situation. :) However, it's not completely clear to me how it should look like:

- Should we test Bazel itself also with --all_incompatible_flags? For the actual Java unit tests and integration tests it "shouldn't" matter, because we should just have tests for the behavior with and without the flag. However, for some changes, "Bazel building itself" or "Bazel running its test suite" is actually part of making sure that it works - think about a flag that affects Bazel's resource scheduling or a flag that affects our test strategy. So maybe yes?

- If yes: On presubmit, too? This would mean we have to provision twice the hardware resources as we currently do. Expensive (omg, more iMac Pros), but OTOH people *really* don't like it when their changes are rolled back even though the presubmit passed (even more so when the breakage comes from a downstream project that they don't even know how to build / test on their local machine, making it hard to debug the breakage).

- Should we test all downstream projects with --all_incompatible_flags? I guess so and the linked issue says so, too. However note that this will double the number of results on this already overloaded UI: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/433 - one could argue that before we add even more breakages on top of that pile, we should - at least once, for the love of Bazel - try to get the current downstream projects pipeline green. AFAIK since inception of our Buildkite CI, this pipeline has never been green.

Cheers,
Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.


--
Laurent

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

ittai zeidman

unread,
Sep 14, 2018, 6:54:21 AM9/14/18
to Lukács T. Berki, Dmitry Lomov, Laurent Le Brun, Ofer Bartal, Philipp Wollermann, Tony Aiuto, bazel-discuss
Wouldn’t that setup mean that these rules need to either version themselves as well (including dependencies) or only support latest (sometimes unreleased) bazel?
On Tue, Sep 11, 2018 at 9:36 AM 'Lukács T. Berki' via bazel-discuss <bazel-...@googlegroups.com> wrote:
The incompatible flag that removed PACKAGE_NAME has been there since ages, and there was *still* breakage on ci.bazel.build which is an indication that people either don't update their code or they don't update their dependencies, even when given a lot of time. In other words, time does not help. Then what is the utility of being generous with it? In this case what would have helped is a version of ci.bazel.build that runs with a set of incompatible changes and the understanding that projects there are expected to fix themselves and bug their dependencies to do so. As long as we get that from any policy, we are at a good place. 

Of course, it would be the best if we had breaking changes less frequently, but realistically, we cannot do that now. What we *can* do now is to instigate a policy that doesn't constrain the speed at which we can remove bad functionality much, and then once we see the number of backward-incompatible changes decreasing, we can make that policy stricter and stricter, eventually culminating in "no backward incompatible changes, ever". 

In other words, I'm proposing a covenant between the users and the developers of Bazel that says "We give you a flag and a transition period and in exchange we get to remove functionality at a given speed", with "given speed" erring on the "fast" side first, then it being gradually ratcheted down.

How does that sound?


On Tue, Sep 11, 2018 at 5:23 AM, ittai zeidman <itt...@gmail.com> wrote:
+1 re PACKAGE_NAME example- I was the one who submitted the fix to protobuf on master since, at the time, rules_scala ran with this flag on.
It wasn’t a good experience to try and understand who’s broken and then checkout protobuf, understand how to fix them and wait for someone on their side to take notice (which took a bit of time).
On Tue, 11 Sep 2018 at 2:18 'Laurent Le Brun' via bazel-discuss <bazel-...@googlegroups.com> wrote:
> For a change like load() positioning, N,N+1,N+2 is fine.

Actually, no. For an example of a trivial change, see https://github.com/bazelbuild/bazel/issues/5827
I tried to give a lot of notice, but that didn't even worked. Maybe advertising --all_incompatible_changes would help. Maybe testing it on the CI, and filing bugs against downstream projects will be enough.

But still: once proto rules are updated, everything that depends on them needs to update their version of the proto rules (and so on, recursively). To make things more bearable, we could synchronize the breaking changes. Some Bazel releases have to be backward-compatible, so that we don't require the recursive updates every month.

And yes, we have to improve tooling. Tooling to automatically detect and fix the syntactic issues (we're working on it), and tooling to help update the WORKSPACE files.


On Tue, 11 Sep 2018 at 00:45, 'Ofer Bartal' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Here are some ideas which may help with parts of this:
* We can't see or change our clients code, but for some breaking changes, we might be able to offer an automated fix. Take Java as an example, error_prone, which is already integrated into Bazel, can automatically create patches to fix code (see example here). If we break an API in a way that we can statically analyse, we might be able to offer a fix, and then "allow ourselves" to expect faster fixes from Bazel users.
* If people don't notice our deprecation warnings until they actually break them, we could use this "trick": Make it an error, and describe in the error that it can be turned into a warning (until a later release when it'll turn back to error) by adding a flag to bazelrc. 

On Tue, Sep 11, 2018 at 1:38 AM 'Philipp Wollermann' via bazel-discuss <bazel-...@googlegroups.com> wrote:
On Tue, Sep 11, 2018 at 12:13 AM 'Tony Aiuto' via bazel-discuss <bazel-...@googlegroups.com> wrote:
  • all tests on buildkite are always run with --all_incompatible_changes. I hope this is true already.

On our Buildkite CI we currently don't test with this flag at all, ever. I personally don't know anyone who has ever used that flag (but I guess it might be popular with rule authors, considering that Starlark is where most of the --incompatible flags came from until now?).

Laurent recently filed a bug that we should start doing so for downstream projects, so we're about to fix the CI situation. :) However, it's not completely clear to me how it should look like:

- Should we test Bazel itself also with --all_incompatible_flags? For the actual Java unit tests and integration tests it "shouldn't" matter, because we should just have tests for the behavior with and without the flag. However, for some changes, "Bazel building itself" or "Bazel running its test suite" is actually part of making sure that it works - think about a flag that affects Bazel's resource scheduling or a flag that affects our test strategy. So maybe yes?

- If yes: On presubmit, too? This would mean we have to provision twice the hardware resources as we currently do. Expensive (omg, more iMac Pros), but OTOH people *really* don't like it when their changes are rolled back even though the presubmit passed (even more so when the breakage comes from a downstream project that they don't even know how to build / test on their local machine, making it hard to debug the breakage).

- Should we test all downstream projects with --all_incompatible_flags? I guess so and the linked issue says so, too. However note that this will double the number of results on this already overloaded UI: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/433 - one could argue that before we add even more breakages on top of that pile, we should - at least once, for the love of Bazel - try to get the current downstream projects pipeline green. AFAIK since inception of our Buildkite CI, this pipeline has never been green.

Cheers,
Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.


--
Laurent

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

david.o...@gmail.com

unread,
Sep 15, 2018, 6:22:39 AM9/15/18
to bazel-discuss
On Wednesday, August 29, 2018 at 9:14:22 AM UTC+2, Lukács T. Berki wrote:
> Hey there,
>
>
> Continuing the thread on bazel-dev (which I should arguably have started here in the first place), let's discuss about how we should roll out incompatible changes to Bazel, that is, ones that have a change to break some builds.
>
>
> Summarizing the above thread:
> Breaking changes are bad because no one wants to have their code broken. Thus, the less frequent breaking changes are, the better.We need to make some breaking changes to Bazel. The more frequently we can make breaking changes, the faster we can develop Bazel (this, of course, directly contradicts (1)We have a lot of functionality we want to remove from Bazel that are mostly misfeatures it inherited from Blaze, its Google-internal versionStarlark already has an incompatible changes policy, to which they hold themselves (but the rest of Bazel does not)
> If we do breaking changes, we should at least have a way to warn people before in a way that is as well-focused as possible (i.e. these warnings are only seen by people who want it and who can act on it)Unlike at Google, we can't know all the places that would be broken by any given changes (unless we give in to the siren song of on-by-default telemetry, and even then, we can't catch every case)The easier it is to keep up to date with the latest Bazel version, the more likely it is that all of your dependencies will be compatible with it and thus won't stop you from upgrading
>
>
> Trying to take into account all the replies on that thread, here is a proposal:
> When we want to do an incompatible change, we always put it behind a flag. Release N would add the flag and default to the old behavior and release N+1 would remove the flag and enforce the new behavior (note that this is different from the current Starlark policy and allows us to remove the old functionality a release earlier)We publish a list of flags we want to flip in any given release.We add some functionality to Bazel (for example, behind a flag) that would tell which places need to be updated to stay compatible (if possible for the flag in question). Its output would be, for example, like this: "the next Bazel version will flip the flag --incompatible_kitten_paws. These locations need to be updated: alice/BUILD:123, bob/charlie.bzl:45 and @dave//erica:fred.bzl (@dave is maintained by geo...@harvey.com)We publish a list of functionality that we would like to remove before we call Bazel 1.0 (we might also make the removal of this functionality a sufficient condition for calling Bazel 1.0, but that's a different discussion)At first, we allow incompatible changes in every Bazel version. A while later, once the functionality stabilizes, start ratcheting this down so that we have incompatible changes less and less frequently (next step could be, for example, every three months). Eventually, we only allow incompatible changes in major releases.
> How does this sound?

I would like to raise javac/byte code compatibility issue
for java rules and ignore for a moment all incompatible
flags and Starlark features.

Many Java based Bazel projects rely on Java 8 byte code
generation, as it is currently the default supported JDK
version. In fact, all is fine when Bazel 0.14.0 is used.

Since Bazel 0.16, even though still Java 8 byte code is
generated per default, we cannot build Gerrit's stable
branches any more, because Bazel's switch to JDK9 javac,
where quite some packages were removed, most notably:

javax.annotation.Generated

This broke outdated versions of annotation process like
Google's auto-value and Dagger 2 libraries.

Currently, on Bazel@HEAD there is no an easy and supported
way to enforce usage of JDK8 javac. You need still this:[1]
and this:[2]. And invoke Bazel@HEAD with something like:

$ cd gerrit
$ git checkout v2.15
$ bazel build --host_javabase=@bazel_tools//tools/jdk:absolute_javabase \
--define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java \
--host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
--java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla \
:release

Yes it is known, that switching to Vanilla Java Builder would
mean the loss of the following features:

* Error Prone
* Strict Java Deps
* Header Compilation
* Reduced Classpath Optimization

Yet, that would allow us to build the code base without
bumping the transitive dependencies to be buildable on JDK9 javac.
(This may or may not be trivial as others have mentioned in that
thread especially when other rulesets are involved like currently
non-maintained rules_closure.)

Still even with those mentioned CLs merged on Bazel@HEAD, there
still incompatibility issues like this one: [3] because of
changing the way where bazel.rc is read from (moving it from
%workspace%/tools/bazel.rc to %workspace%/.bazelrc).

This raises the question whether trying hard to preserve backwards
Java compatibility would be jeopardized by other incompatible changes.
So the probably the only answer would be to build older stable branches
with older versions of Bazel.

The way it worked with Buck was: in .buckversion the SHA1 was checked
in into the repository and once we build a specific commit, Buck
updated itself and built from the source this SHA1 code base of
itself:

$ cd gerrit
$ git checkout stable-2.12
$ buck build gerrit-server:server
Buck is at 79d36de9f5284f6e833cca81867d6088a25685fb, but should be
1b03b4313b91b634bd604fc3487a05f877e59dee.
Buck is updating itself. To disable this, add a '.nobuckcheck'
file to your project root. In general, you should only disable
this if you are developing Buck.
Buck does not appear to have been built -- building Buck!
All done, continuing with build.
Using watchman.
[...]
[-] PROCESSING BUCK FILES...FINISHED 1.2s [100%]
[-] BUILDING...FINISHED 17.0s [100%] (250/250 JOBS, 250 UPDATED, 100.0% CACHE MISS)

This raises the question what is the right way to achieve
similar functionality for developers and CIs environments
with Bazel?

Can different Bazel versions be installed in parallel on the
same machine (Docker image)? Say developers would like to bisect
a problem starting from Gerrit master (Bazel >= 0.17.1) and going
back to older stable releases, that can only be built with Bazel
< 0.16.0, so that they would to install different Bazel versions...

[1] https://github.com/bazelbuild/bazel/issues/6012
[2] https://github.com/bazelbuild/bazel/pull/5984
[3] http://paste.openstack.org/show/730135

Liam Miller-Cushon

unread,
Sep 15, 2018, 4:02:30 PM9/15/18
to david.o...@gmail.com, bazel-discuss
On Sat, Sep 15, 2018 at 3:22 AM <david.o...@gmail.com> wrote:
This raises the question whether trying hard to preserve backwards
Java compatibility would be jeopardized by other incompatible changes.
So the probably the only answer would be to build older stable branches
with older versions of Bazel.
... 
This raises the question what is the right way to achieve
similar functionality for developers and CIs environments
with Bazel?

Thanks for raising this, I think it's an important question.

The incompatible changes policy options discussed in this thread aren't strong enough to guarantee that old release branches will build with the latest version of Bazel. The 'Radical Stability' doc discusses 3 month migration windows for incompatible changes, and release branches could be a lot older than that.

I agree that having some way to version the Bazel binary with your repository is necessary to support that use-case.

I'm not aware of any inherent problems with having multiple versions of the Bazel binary installed, so the question might just be how to make it convenient to do that for development and CI builds.

Philipp Wollermann

unread,
Sep 18, 2018, 4:49:53 AM9/18/18
to david.o...@gmail.com, bazel-discuss
Hi David,

would you like it if Bazel had a similar mechanism to Buck's .buckversion file?
What were your good and bad experiences with that approach?

I think it wouldn't be hard to add support for this and I personally think it's a good approach. It would give users:
- Easy versioning of the used Bazel version with their code base (so you wouldn't run into issues like "my users upgraded Bazel and now they can no longer build the latest stable Gerrit release").
- Easy and testable upgrades for Bazel - just bump the version in your .bazelversion in a PR, run a presubmit check on CI, if necessary, apply any necessary migrations and rules upgrades in the same CL to make the upgrade atomic and then submit with confidence. Your repo would never be in a state where it doesn't build / test due to an incompatible Bazel or rules version.

What do you think?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c5e82e78-b410-4a10-94ad-8cf96f024c7a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München


Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

Klaus Aehlig

unread,
Sep 18, 2018, 5:48:25 AM9/18/18
to Philipp Wollermann, david.o...@gmail.com, bazel-discuss
> would you like it if Bazel had a similar mechanism to Buck's .buckversion
> file?

I think, the whole complexity comes in with external dependencies---and
the fact that bazel wants to build them from source. So what do we do
with an external repository then? Do we switch to that bazel version
when entering there? What about rules from external repositories---which
version of bazel is to be used there? What about providers passed through
repository boundaries?

Can someone explain what the semantics of .buckversion is in this case?

Note that this is not a theoretical question; a major point in the
discussion about the migration window was the question about what to
do, if one of my dependencies needs a very new version of bazel and
another dependency hasn't updated yet.

Thanks,
Klaus

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

david.o...@gmail.com

unread,
Sep 18, 2018, 6:31:25 AM9/18/18
to bazel-discuss
Hi Philipp,

Am Dienstag, 18. September 2018 10:49:53 UTC+2 schrieb Philipp Wollermann:
> Hi David,
>
>
> would you like it if Bazel had a similar mechanism to Buck's .buckversion file?
> What were your good and bad experiences with that approach?

It worked for gerrit project. I have impression that Buck is
smaller project as Bazel with compilation time 10-15 seconds.
So that it was acceptable.



> I think it wouldn't be hard to add support for this and I personally think it's a good approach. It would give users:
> - Easy versioning of the used Bazel version with their code base (so you wouldn't run into issues like "my users upgraded Bazel and now they can no longer build the latest stable Gerrit release").
> - Easy and testable upgrades for Bazel - just bump the version in your .bazelversion in a PR, run a presubmit check on CI, if necessary, apply any necessary migrations and rules upgrades in the same CL to make the upgrade atomic and then submit with confidence. Your repo would never be in a state where it doesn't build / test due to an incompatible Bazel or rules version.
>
>
> What do you think?

It depends on compilation time. Bazel build from source is much
longer, including building external dependencies from source as well:
Protobuff, gRPC. Waiting many minutes until Bazel was built from
scratch is less attractive during bisect session.

IIRC in recent Buck versions, they removed that
self-update feature. So that may be someone from
Buck project could comment why that feature was removed?

Another option we have is to be able to install multiple
Bazel versions simultaneously and find a way to switch
between them. On linux update-alternatives could be an option.

Straight forward Bazel multiple version management would be
just download 10 different versions and install the right one.

The problem I ran into yesterday, during bisecing one Gerrit problem,
was just that: I forgot what version of Bazel we used one year ago,
I started from installing 0.10 und went until 0.6.1 and it worked.

In recent gerrit versions we are tracking the minimum required version
that was used in WORKSPACE file:

versions.check(minimum_bazel_version = "0.14.0")

In older gerrit versions that information is lost ;-(

Philipp Wollermann

unread,
Sep 18, 2018, 7:37:28 AM9/18/18
to Klaus Aehlig, david.o...@gmail.com, bazel-discuss
On Tue, Sep 18, 2018 at 11:48 AM Klaus Aehlig <aeh...@google.com> wrote:
> would you like it if Bazel had a similar mechanism to Buck's .buckversion
> file?

I think, the whole complexity comes in with external dependencies---and
the fact that bazel wants to build them from source. So what do we do
with an external repository then? Do we switch to that bazel version
when entering there? What about rules from external repositories---which
version of bazel is to be used there? What about providers passed through
repository boundaries?

I'd be pragmatic here and say we ignore the .bazelversion file in external repos and just use the one from your own workspace.

This "should" be fine, because:
- Project owners will only pick versions of external dependencies that work with their overall project (which includes their Bazel version then). So they can pick a version that works fine with their current version of Bazel. Or they could vendor and patch the dependency to work with a newer version of Bazel if required (and hopefully contribute that back to the upstream source). That's all nothing new and project maintainers are used to this style of development - you pin your deps to a version that works for you, then upgrade as necessary. In Google we basically do the same.
- Rule / library authors will want to ensure that their stuff works with as many Bazel versions still in use as possible. If we give them good CI that allows you to test every commit against all still relevant Bazel versions and if we do good work in providing people the mechanisms to target different Bazel versions with their rule / library, then this should work. Again, this is nothing new - people out there already have to maintain compatibility with old versions of language interpreters / compilers / build tools.

FWIW, I see what you'd like to see here (depending on binary artifacts) and I agree that this would also be a cool solution to this, albeit probably hard to implement with Bazel? (But you're the expert on external deps and have lots of experience with how other tools do it, so I trust your vision here!)

Note that this is not a theoretical question; a major point in the
discussion about the migration window was the question about what to
do, if one of my dependencies needs a very new version of bazel and
another dependency hasn't updated yet.

I'd say: Then you either fix the dependency yourself (either of them) or you just don't depend on it. If I'm developing my web app with Django on Python 3.x, but that totally awesome REST backend framework that I want to depend on only supports Python 2.x, well... then I just won't use it and instead use something else. Or maybe I'll fix it to be compatible with Python 3 and contribute the patch back (or people start to depend on my fork instead).

Philipp Wollermann

unread,
Sep 18, 2018, 7:45:29 AM9/18/18
to david.o...@gmail.com, bazel-discuss
On Tue, Sep 18, 2018 at 12:31 PM <david.o...@gmail.com> wrote:
It depends on compilation time. Bazel build from source is much
longer, including building external dependencies from source as well:
Protobuff, gRPC. Waiting many minutes until Bazel was built from
scratch is less attractive during bisect session.

We could easily have binaries available for download for every commit in our GitHub repo (we already do for each tag / release), so Bazel wouldn't have to build it from source - it could just download the matching version from our CI.
 
IIRC in recent Buck versions, they removed that
self-update feature. So that may be someone from
Buck project could comment why that feature was removed?

I checked their issue tracker and found this explanation: https://github.com/facebook/buck/issues/1733

"if its important for your team to synchronize buckversions, you should consider moving to prebuilt buck binaries instead. Buck publishes releases on github now, you can also fetch buck built at any git sha from jitpack like https://jitpack.io/com/github/facebook/buck/<buck-repo-git-sha>/buck-<buck-repo-git-sha>.pex

The logic to download buck should be administered via a wrapper script ( example ). The logic was removed from buck originially as it adds overhead to always check the buck version against the git in the source copy."

(That's basically what I considered doing, too.)

Notably, in Google the "blaze" command that you run to build stuff is actually a wrapper script that exec's a checked-in "blaze" binary. Which is hard to do in Git (without LFS at least) due to repository bloat over time, but using a ".bazelversion" file would achieve the same thing.
 
Another option we have is to be able to install multiple
Bazel versions simultaneously and find a way to switch
between them. On linux update-alternatives could be an option.

As Bazel is just a self-contained binary, people can already install multiple versions of it by renaming the downloaded binary to e.g. "bazel-0.16.1".
If this is a popular thing to do, we could make it easier to install multiple versions via our apt-get repository or by letting the official installer install it via a versioned filename (and maybe installing "bazel" as a symlink to it).

I think the problem here is that this still puts the cost on the owner of figuring out which Bazel version to use with a given project.

We often get issue reports of people trying to build e.g. TensorFlow with a too old or too new version of Bazel, but they don't get any meaningful errors, they simply run into failing builds. Using a .bazelversion file would help these users by automatically using a tested, compatible version of Bazel.

Straight forward Bazel multiple version management would be
just download 10 different versions and install the right one.

Exactly.

Austin Schuh

unread,
Sep 18, 2018, 12:12:00 PM9/18/18
to Philipp Wollermann, david.o...@gmail.com, bazel-discuss
On Tue, Sep 18, 2018 at 4:45 AM 'Philipp Wollermann' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Notably, in Google the "blaze" command that you run to build stuff is actually a wrapper script that exec's a checked-in "blaze" binary. Which is hard to do in Git (without LFS at least) due to repository bloat over time, but using a ".bazelversion" file would achieve the same thing.

Note, today the tools/bazel script can be used to recreate this behavior pretty easily.  Our script is a script which looks in a cache folder for the correct version of a downloaded bazel, downloads it if it doesn't exist, and runs it.  This lets us version bazel with the repo without git repo bloat.  It would be pretty easy to script looking for a .bazelversion file.

Austin

gert...@gmail.com

unread,
Sep 18, 2018, 6:36:27 PM9/18/18
to bazel-discuss
On Tuesday, 18 September 2018 13:45:29 UTC+2, Philipp Wollermann wrote:
> As Bazel is just a self-contained binary, people can already install multiple versions of it by renaming the downloaded binary to e.g. "bazel-0.16.1".

I was about to post that here. Be aware about 'bazel' just a being a script calling a 'bazel-real' binary, though.

> If this is a popular thing to do, we could make it easier to install multiple versions via our apt-get repository or by letting the official installer install it via a versioned filename (and maybe installing "bazel" as a symlink to it).

If you go for that approach (I'd say yey!), please also consider any of the two following approaches:

A:
Rename the packages to include the Bazel version (e.g. bazel-0.17, bazel-0.16), each providing the latest point release. Change the 'bazel' package to simply become an empty metapackage with the dependency on the latest 'bazel-X.Y' package. This will allow multiple bazel versions to be installed. Then add support for 'update-alternatives' on Debian/Ubuntu to select the right/default version for the default 'bazel' command. This allows users to change the bazel version without reinstalling and actually keep multiple versions installed.

B:
Change the current hosted APT repository to retain older versions of the 'bazel' package - I believe only the latest version is available currently ('apt policy bazel'). Then users can manually reinstall the version they want, and use APT pinning configurations or something to blacklist breakages (e.g. consider anything 0.16.x broken, but upgrade to 0.17.x+ when available.).

Laurent Le Brun

unread,
Sep 19, 2018, 6:04:05 AM9/19/18
to gert...@gmail.com, bazel-discuss
Hi,

I really it's a great idea to have .bazelversion or a similar mechanism to everyone builds the code base with the same version of Bazel. This is super useful for reproducible builds, and it can simplify maintainers work a lot. Multiple companies do that. It's much easier for engineers if they use the same build system as the CI. And with remote execution, it means more cache hits (a Bazel update might change the exact command sent).


However, I don't think this is enough.

1. Scenario: A company has a big code base. There are multiple things to fix to update to the next Bazel version. Changing all the instances in the whole code base in one commit (the commit modifying the Bazel version) is not feasible. So we need a migration window. If the function foo is renamed foo2, we need to:
   a. A Bazel update introduces a foo2
   b. Company updates code to use foo2
   c. Another Bazel update removes foo.

2. On top of that, we also need flags. In the previous scenario, the company will probably want to switch the flag to disable foo as soon as they've migrated away. This is important for example if there are multiple migrations to do (you cannot do them all at the same time). Switching the flag prevents regressions in the code base (some engineers might not know about it, or might use the old way by accident).

3. Some migrations can take a lot of time. An example was the migration from Python-preprocessing to Skylark/Starlark. It took ~2 years at Google and a lot of time at Facebook. A complicated migration time means that Bazel has to support the legacy behavior for a long time. So it's important to have a switch (testing with the next release candidate is definitely not enough).


Providing a way to test the new behavior in advance provides a nicer experience for anyone will to follow the best practices. As Bazel evolves fast, each release fixes a bunch of issues, we want users to be able to update smoothly to the latest version. If we don't do that, we may have users stuck to older versions. And that means they'll file bugs against those old versions, which we don't want to maintain.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f257e30a-dba0-4cf2-b816-1cc11776b571%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Laurent

Dmitry Lomov

unread,
Sep 19, 2018, 6:10:58 AM9/19/18
to gert...@gmail.com, bazel-discuss
Fixing bazel at specific versions is a solution to a different problem than what Radical Stability and this thread trying to address.
As several replies here pointed out, there are plenty of already existing mechanisms to pin bazel to specific versions, and of course the projects using Bazel need to do so if they want to achieve full reproducibility.

However, the problem that we do face is that we do not make breaking changes in Bazel just for the sake of breaking changes. We do them because Bazel does not yet do all the things it needs to do, and does not do the things it does well enough. So we need to change, adapt, and evolve Bazel. And so the ecosystem around Bazel needs to change, adapt, and evolve. This includes rulesets for different languages/ecosystems, popular libraries that use Bazel (such as Tensorflow), and services and external software that rely of Bazel APIs (Buildfarm, IntelliJ plugins and so on).

So all the proposals for a migration window, stability window, accounting for incompatible changes and so on are on the table to facilitate the evolution, facilitate the change. We must make life easier and give structure to those in the community who are willing to follow the Bazel evolution, to actually do so. We absolutely should not require upgrading Bazel all the time for people who do not need that, but we should encourage and support people who want to and need to do that. 

This is especially true for maintainers of "well-connected nodes" in the ecosystem - those that many other users of Bazel rely on. Bazel is nothing without its ecosystem of rulesets, libraries that support building themselves with Bazel, and other tooling, and without a clear incompatible change policy, and means to stay informed and follow those changes, we are selling these people short.

Pinning to specific version of Bazel is easy (and useful in some cases - I think people should absolutely do so for full reproduciblity), but if that is our only recommendation, we will not be able to evolve the ecosystem, the Bazel landscape will be fragmented beyond recognition, and Bazel ecosystem will wither and die.

Dmitry


--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f257e30a-dba0-4cf2-b816-1cc11776b571%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--

Lukács T. Berki

unread,
Sep 19, 2018, 7:46:30 AM9/19/18
to Dmitry Lomov, gert...@gmail.com, bazel-discuss
On Wed, Sep 19, 2018 at 12:10 PM, 'Dmitry Lomov' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Fixing bazel at specific versions is a solution to a different problem than what Radical Stability and this thread trying to address.
As several replies here pointed out, there are plenty of already existing mechanisms to pin bazel to specific versions, and of course the projects using Bazel need to do so if they want to achieve full reproducibility.

However, the problem that we do face is that we do not make breaking changes in Bazel just for the sake of breaking changes. We do them because Bazel does not yet do all the things it needs to do, and does not do the things it does well enough. So we need to change, adapt, and evolve Bazel. And so the ecosystem around Bazel needs to change, adapt, and evolve. This includes rulesets for different languages/ecosystems, popular libraries that use Bazel (such as Tensorflow), and services and external software that rely of Bazel APIs (Buildfarm, IntelliJ plugins and so on).

So all the proposals for a migration window, stability window, accounting for incompatible changes and so on are on the table to facilitate the evolution, facilitate the change. We must make life easier and give structure to those in the community who are willing to follow the Bazel evolution, to actually do so. We absolutely should not require upgrading Bazel all the time for people who do not need that, but we should encourage and support people who want to and need to do that. 

This is especially true for maintainers of "well-connected nodes" in the ecosystem - those that many other users of Bazel rely on. Bazel is nothing without its ecosystem of rulesets, libraries that support building themselves with Bazel, and other tooling, and without a clear incompatible change policy, and means to stay informed and follow those changes, we are selling these people short.

Pinning to specific version of Bazel is easy (and useful in some cases - I think people should absolutely do so for full reproduciblity), but if that is our only recommendation, we will not be able to evolve the ecosystem, the Bazel landscape will be fragmented beyond recognition, and Bazel ecosystem will wither and die.
This is an important point -- are you envisioning the Bazel ecosystem as one that works like continuous integration, i.e. people are expected to stick to the very close to the latest version of whatever library they depend on, or do they expect to use old-ish versions? If the latter, incompatible changes are much harder to make. 


Dmitry


On Wed, Sep 19, 2018 at 12:36 AM <gert...@gmail.com> wrote:
On Tuesday, 18 September 2018 13:45:29 UTC+2, Philipp Wollermann  wrote:
> As Bazel is just a self-contained binary, people can already install multiple versions of it by renaming the downloaded binary to e.g. "bazel-0.16.1".

I was about to post that here. Be aware about 'bazel' just a being a script calling a 'bazel-real' binary, though.

> If this is a popular thing to do, we could make it easier to install multiple versions via our apt-get repository or by letting the official installer install it via a versioned filename (and maybe installing "bazel" as a symlink to it).

If you go for that approach (I'd say yey!), please also consider any of the two following approaches:

A:
Rename the packages to include the Bazel version (e.g. bazel-0.17, bazel-0.16), each providing the latest point release. Change the 'bazel' package to simply become an empty metapackage with the dependency on the latest 'bazel-X.Y' package. This will allow multiple bazel versions to be installed. Then add support for 'update-alternatives' on Debian/Ubuntu to select the right/default version for the default 'bazel' command. This allows users to change the bazel version without reinstalling and actually keep multiple versions installed.

B:
Change the current hosted APT repository to retain older versions of the 'bazel' package - I believe only the latest version is available currently ('apt policy bazel'). Then users can manually reinstall the version they want, and use APT pinning configurations or something to blacklist breakages (e.g. consider anything 0.16.x broken, but upgrade to 0.17.x+ when available.).

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAObu7DEpOdu7-sWuCdjmEpi2Q-p44rsbWC1SJ9Zs5r7%3DGLd11w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Dmitry Lomov

unread,
Sep 19, 2018, 8:08:01 AM9/19/18
to Lukács T. Berki, Gert van Dijk, bazel-discuss
On Wed, Sep 19, 2018 at 1:46 PM Lukács T. Berki <lbe...@google.com> wrote:


On Wed, Sep 19, 2018 at 12:10 PM, 'Dmitry Lomov' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Fixing bazel at specific versions is a solution to a different problem than what Radical Stability and this thread trying to address.
As several replies here pointed out, there are plenty of already existing mechanisms to pin bazel to specific versions, and of course the projects using Bazel need to do so if they want to achieve full reproducibility.

However, the problem that we do face is that we do not make breaking changes in Bazel just for the sake of breaking changes. We do them because Bazel does not yet do all the things it needs to do, and does not do the things it does well enough. So we need to change, adapt, and evolve Bazel. And so the ecosystem around Bazel needs to change, adapt, and evolve. This includes rulesets for different languages/ecosystems, popular libraries that use Bazel (such as Tensorflow), and services and external software that rely of Bazel APIs (Buildfarm, IntelliJ plugins and so on).

So all the proposals for a migration window, stability window, accounting for incompatible changes and so on are on the table to facilitate the evolution, facilitate the change. We must make life easier and give structure to those in the community who are willing to follow the Bazel evolution, to actually do so. We absolutely should not require upgrading Bazel all the time for people who do not need that, but we should encourage and support people who want to and need to do that. 

This is especially true for maintainers of "well-connected nodes" in the ecosystem - those that many other users of Bazel rely on. Bazel is nothing without its ecosystem of rulesets, libraries that support building themselves with Bazel, and other tooling, and without a clear incompatible change policy, and means to stay informed and follow those changes, we are selling these people short.

Pinning to specific version of Bazel is easy (and useful in some cases - I think people should absolutely do so for full reproduciblity), but if that is our only recommendation, we will not be able to evolve the ecosystem, the Bazel landscape will be fragmented beyond recognition, and Bazel ecosystem will wither and die.
This is an important point -- are you envisioning the Bazel ecosystem as one that works like continuous integration, i.e. people are expected to stick to the very close to the latest version of whatever library they depend on, or do they expect to use old-ish versions? If the latter, incompatible changes are much harder to make. 

It is up to all of us together to help ecosystem shape in the way that works best for everyone, and for Bazel vision.
As is probably clear from my messages, I am very much pro embracing change (in a structured, controlled manner)



Dmitry


On Wed, Sep 19, 2018 at 12:36 AM <gert...@gmail.com> wrote:
On Tuesday, 18 September 2018 13:45:29 UTC+2, Philipp Wollermann  wrote:
> As Bazel is just a self-contained binary, people can already install multiple versions of it by renaming the downloaded binary to e.g. "bazel-0.16.1".

I was about to post that here. Be aware about 'bazel' just a being a script calling a 'bazel-real' binary, though.

> If this is a popular thing to do, we could make it easier to install multiple versions via our apt-get repository or by letting the official installer install it via a versioned filename (and maybe installing "bazel" as a symlink to it).

If you go for that approach (I'd say yey!), please also consider any of the two following approaches:

A:
Rename the packages to include the Bazel version (e.g. bazel-0.17, bazel-0.16), each providing the latest point release. Change the 'bazel' package to simply become an empty metapackage with the dependency on the latest 'bazel-X.Y' package. This will allow multiple bazel versions to be installed. Then add support for 'update-alternatives' on Debian/Ubuntu to select the right/default version for the default 'bazel' command. This allows users to change the bazel version without reinstalling and actually keep multiple versions installed.

B:
Change the current hosted APT repository to retain older versions of the 'bazel' package - I believe only the latest version is available currently ('apt policy bazel'). Then users can manually reinstall the version they want, and use APT pinning configurations or something to blacklist breakages (e.g. consider anything 0.16.x broken, but upgrade to 0.17.x+ when available.).

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891
Reply all
Reply to author
Forward
0 new messages