Bazel caches getting frequently corrupted on CircleCI builds

804 views
Skip to first unread message

Shivansh Rai

unread,
Mar 25, 2022, 1:39:34 AM3/25/22
to bazel-discuss
Hi,

We use Bazel for managing our internal Go repositories, and use CircleCI for builds with
Bazel cache enabled. Since last week we have started seeing the following error on
every build:
```
FATAL: corrupt installation: file '/root/.cache/bazel/_bazel_root/install/c87283ec3a7822eea44f4cecb6db792e/A-server.jar' is missing or modified.  Please remove '/root/.cache/bazel/_bazel_root/install/c87283ec3a7822eea44f4cecb6db792e' and try again.
```

As a workaround, we had to disable Bazel cache on CircleCI, but that in turn leads to
very long build times. I came across a similar issue from 2015 for Scala builds:
https://github.com/bazelbuild/bazel/issues/471.

Our CircleCI builds use the following Bazel version:
```
$ bazel version
Build label: 5.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Jan 19 14:08:54 2022 (1642601334)
Build timestamp: 1642601334
Build timestamp as int: 1642601334
```

Any pointers on this would be appreciated, thanks!
Regards,
Shivansh

Haixin Tie

unread,
May 10, 2024, 1:26:15 PMMay 10
to bazel-discuss
I encountered the same issue, happening on a BitBucket pipeline every time. Having to disable the build cache for now. Any solution recommendations would be appreciated.

FATAL: corrupt installation: file '/home/builder/.cache/bazel/_bazel_builder/install/7e26c0f1b3f6b658dee2b9a9a80ffbb2/A-server.jar' is missing or modified.  Please remove '/home/builder/.cache/bazel/_bazel_builder/install/7e26c0f1b3f6b658dee2b9a9a80ffbb2' and try again.

Regards,
HaiXin

Haixin Tie

unread,
May 13, 2024, 10:24:50 AMMay 13
to bazel-discuss
Google groups are dropping messages for some reason. Multiple replies to this thread were not captured here. Pasting them here for posterity:

On Sat, May 11, 2024 at 2:45 PM HaiXin Tie <thx...@gmail.com> wrote:
Thanks, Manuel! After changing --disk_cache to another location the error went away.

Regards,
HaiXin


On Fri, May 10, 2024 at 1:45 PM Manuel <naranjo...@gmail.com> wrote:
. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

Edbert Chan

unread,
May 13, 2024, 5:23:38 PMMay 13
to bazel-discuss
We are seeing this issue as well.

The error is thrown from a file validation check that Bazel does to ensure no files have been tampered with.
https://github.com/bazelbuild/bazel/blob/1d20ea76113e49f7fb607ef92231f3665797b8e6/src/main/cpp/util/file_posix.cc#L458

Someone submitted a hack for Scala that touches the files:
https://github.com/bazelbuild/bazel/commit/cb2e03322be3f34b28df22ea17fe05e16fd2466d

It is not clear how to get around this as the original issue and idea is 5 years old. For now, since the fix is simple and there is little downside risk, we can go ahead with this and deal with the core issue at another time.

Try running this before you run bazel:

future_date=$(date +%s -d "+10 years")
find $FOLDER_CACHE/install -type f -exec touch --date=@$future_date {} +

carpen...@gmail.com

unread,
May 15, 2024, 9:46:43 AMMay 15
to bazel-discuss
On windows we see bazel sets the timestamp to of these files to be in the future and it checks they are still in the future as an integrity check. When you use ReFS for example this is put back by the OS and you get this error. 

Could something be adjusting the timestamps on the files?

Manuel

unread,
May 15, 2024, 10:48:09 AMMay 15
to Haixin Tie, bazel-discuss
Stop caching the ~/.bazel, configure repository cache and disk cache, and keep that around instead

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f7b8d9a7-6751-419a-ada5-a38d42667fdan%40googlegroups.com.

HaiXin Tie

unread,
May 15, 2024, 10:48:09 AMMay 15
to Manuel, bazel-discuss
Thanks, Manuel! After changing --disk_cache to another location the error went away.

Regards,
HaiXin


On Fri, May 10, 2024 at 1:45 PM Manuel <naranjo...@gmail.com> wrote:
. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

El vie, 10 may 2024, 22:05, HaiXin Tie <thx...@gmail.com> escribió:
~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,
HaiXin

Manuel

unread,
May 15, 2024, 10:48:09 AMMay 15
to HaiXin Tie, bazel-discuss
. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

El vie, 10 may 2024, 22:05, HaiXin Tie <thx...@gmail.com> escribió:
~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,
HaiXin


On Fri, May 10, 2024 at 11:16 AM Manuel <naranjo...@gmail.com> wrote:

HaiXin Tie

unread,
May 15, 2024, 10:48:09 AMMay 15
to Manuel, bazel-discuss
~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,
HaiXin


On Fri, May 10, 2024 at 11:16 AM Manuel <naranjo...@gmail.com> wrote:
Reply all
Reply to author
Forward
0 new messages