Sharing bazel cache (among multiple Jenkins workspaces)

[rober...@verbsurgical.com]

Hi--

We are using Jenkins, where a build node is configured with multiple executors, and the bazel project's job is configured with "Execute concurrent builds if necessary".

The problem is that when a build node runs multiple instances of a job, each instance of the job gets its own workspace, which I believe means each instance of the job ends up with its own bazel cache (since path to cache is based on path to workspace).

Is "--output_base" intended to let me configure these jobs (all executed by the same "jenkins" user) to share a bazel cache?

Each of my bazel configurations maps to a separate jenkins job, so I think --output_base means each instance of a job will share both a cache and a bazel server (so I think each job must have a different --output_base, if their bazel configurations are different). I think I am primarily interested in just sharing the cache artifacts (e.g., avoid recompiles), but I guess sharing a bazel server is also good.

Thanks,

--Rob

[Brian Silverman]

Yes --output_base will let you have all the jobs share a cache (and output tree) if you set it the same for all the jobs, but I'm pretty sure you don't want to do that. If you do, all of the jobs will share the same bazel server, and will serialize commands, which kind of defeats the purpose of running jobs concurrently. Bazel's filesystem cache and output tree are not designed for concurrent use, and they won't let you do that easily (I'm sure you could force it, but you don't want to).

I think you want each job to have a small set of output bases, and use a different one from that set for each instance of the job you run concurrently. That will share the caches among each job, but still allow concurrency. FWIW, I've found that locks in the kernel are the biggest performance limitation for machines with even 16 cores, so I only run one job (which encompasses a few different configurations) on each Jenkins build machine. That should be better with the new sandbox, but it's something to keep in mind.

I do use --output_base to run multiple bazel jobs at the same time from the same source tree (on each build machine). The script which starts up the bazel commands passes a different --output_base based on the --compiler/--cpu/--compilation_mode/etc flags for each one, which results in good caching. Important tip if you're going to do this: use --symlink_prefix=/ or target wildcards will recurse into the convenience symlinks when they happen to point into a different output base, and then give strange errors occasionally when the files change out from underneath it.

Each output base (explicitly set or the default one based on the path to the workspace and the current user) will run its own server, and have a completely separate output tree and cache. I'm not completely sure what you mean by "bazel configuration", but if you mean bazel flags (like --compilation_mode) then a single server and cache will cache the latest version of each output for each configuration separately. Those are "target" configurations made of --compilation_mode, --compiler, --cpu, and --platform_suffix, and the single "host" configuration. I've run into some issues with the flags to the host C++ toolchain changing based on the flags, which then triggers rebuilds of most of the cached outputs every time you switch; not sure if you have anything like this, but that's part of the reason I use a separate --output_base for each combination of flags on the CI machines.

The real solution here is the remote caching and execution code which is coming soon (I'm really not sure what the current status is; might already be ready to use for this). When it's ready to use for at least sharing a cache between multiple bazel servers, possibly on multiple machines, and I have time, I plan to set it up to solve this same problem. Until then, a cache for each output configuration on each machine isn't too bad.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a4a4e5a9-a309-44a8-8d54-f42bb41d30cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.