Preventing analysis cache discards

[Alex Eagle]

Slack thread context: https://bazelbuild.slack.com/archives/CA31HN1T3/p1668109501597109

Curious if anyone has thought about a more robust way to detect/prevent analysis cache discards. It's a classic Bazel "foot-gun" that "you forgot to pass --mumble_mumble to your bazel info command and that's why your subsequent bazel build is taking minutes."

There are some buried features like --max_config_changes_to_show which help a savvy user to debug the analysis cache discard, but a novice user or even a savvy one supporting lots of Bazel deployments can't keep up with a reactive "fix it when it's a problem" approach.

Has anyone thought harder about this problem? I'm imagining some way to say "put my Bazel in a mode where analysis cache discards or server restarts are an error" so that we can enable that mode on CI for example. As a strawman, you could imagine a marker file in the output base indicating this behavior.

-Alex

[Konstantin]

Alex, I just want to acknowledge the problem you have raised and mention that unfortunately I observed situations when Bazel was dropping Analysis cache without even declaring which configuration change caused it. 

Also it could be more beneficial in the long term to work on reducing the number of cases when Analysis cache is dropped. I'd love to see it stay for opt/dbg switch and for the custom command line flags changes. I understand it would grow substantially to include both configurations but still in many cases it is preferred to rebuilding it after every seemingly insignificant change.

Konstantin

[Alex Eagle]

We had several fruitful discussions about this at BazelCon so let me provide some notes for those who couldn't attend!

1) Let's make "common" mean what you expect in bazelrc, so it's easier to ensure `query` will always get the configuration-related options: https://github.com/bazelbuild/bazel/issues/3054

2) Something like --disallow_analysis_cache_discards https://github.com/bazelbuild/bazel/issues/16804 can help to fail-fast
3) Tony says that the ideal long-term state is to move configuration into targets, since the ultimate goal of multi-configuration builds suggests there would be no more flags that could invalidate configuration and therefore the analysis cache. Requires a great deal of willpower and years of google3 Large Scale Changes so we certainly need some other remediations in the short term.
4) Accidental restarts of the Bazel server and invalidations of external repositories are also problems.