Setting up optimal CI builds

[Fredrik Medley]

Our presubmit checks is spenting a lot of time in Bazel, to find out that everything is remotely cached by Buildbarn. The analysis cache is unfortunately discarded because we run Bazel multiple times with different flags. How should one setup the CI system to avoid spending minutes on checking for cache hits?

We have already cut the time in Bazel by more than 50% by applying https://github.com/moroten/bazel/commits/optimize-remote-execution as described in https://github.com/bazelbuild/bazel/issues/10875. My latest investigation only points to dependency graph management within Bazel and just a minority of the time is spent on actually checking for cache hit.

Ulf Jack answered in https://github.com/bazelbuild/bazel/issues/12113 that they don't discard their analysis cache between builds. That would definitely cut our build times. How can this be done in practice?

We would like to get down to presubmit times of below one minute, inspired by Eric Burnett's talk "One minute Presubmits" at http://build-london.com/. Bazel has taken us down from the 1 hour range to 10 minutes, but right now Bazel is on the critical path.

[Austin Schuh]

If you pick different output bases for each of your invocations, you can avoid invalidating your action cache at the cost of more memory usage.  The end result is one bazel instance per output base.

I haven't figured out a solution yet, but with build without the bytes, I'm unable to break the sub 3 minute mark on one of my builds.  Turning it off increases the fresh build time and decreases the incremental build time.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/907bb266-6596-4c4c-b786-091603b83c42n%40googlegroups.com.

[Fredrik Medley]

It reminds me of my one year old thread about keeping the analysis cache, https://groups.google.com/g/bazel-discuss/c/vgn9uyyIrIM/m/4trNkRS0AQAJ. The memory footprint and multiple external repository folders due to multiple Bazel instances is a bit scary. Do others user multiple Bazel instances with different --output_base?

Yes, we are using "build without the bytes".