Bazel caches getting frequently corrupted on CircleCI builds

[Shivansh Rai]

Hi,

We use Bazel for managing our internal Go repositories, and use CircleCI for builds with
Bazel cache enabled. Since last week we have started seeing the following error on
every build:
```
FATAL: corrupt installation: file '/root/.cache/bazel/_bazel_root/install/c87283ec3a7822eea44f4cecb6db792e/A-server.jar' is missing or modified.  Please remove '/root/.cache/bazel/_bazel_root/install/c87283ec3a7822eea44f4cecb6db792e' and try again.
```

As a workaround, we had to disable Bazel cache on CircleCI, but that in turn leads to
very long build times. I came across a similar issue from 2015 for Scala builds:
https://github.com/bazelbuild/bazel/issues/471.

Our CircleCI builds use the following Bazel version:
```
$ bazel version
Build label: 5.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Jan 19 14:08:54 2022 (1642601334)
Build timestamp: 1642601334
Build timestamp as int: 1642601334
```

Any pointers on this would be appreciated, thanks!
Regards,
Shivansh

[Haixin Tie]

I encountered the same issue, happening on a BitBucket pipeline every time. Having to disable the build cache for now. Any solution recommendations would be appreciated.

FATAL: corrupt installation: file '/home/builder/.cache/bazel/_bazel_builder/install/7e26c0f1b3f6b658dee2b9a9a80ffbb2/A-server.jar' is missing or modified.  Please remove '/home/builder/.cache/bazel/_bazel_builder/install/7e26c0f1b3f6b658dee2b9a9a80ffbb2' and try again.

Regards,

HaiXin

[Haixin Tie]

Google groups are dropping messages for some reason. Multiple replies to this thread were not captured here. Pasting them here for posterity:

On Sat, May 11, 2024 at 2:45 PM HaiXin Tie <thx...@gmail.com> wrote:

Thanks, Manuel! After changing --disk_cache to another location the error went away.

Regards,

HaiXin

On Fri, May 10, 2024 at 1:45 PM Manuel <naranjo...@gmail.com> wrote:

. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

[Edbert Chan]

We are seeing this issue as well.

The error is thrown from a file validation check that Bazel does to ensure no files have been tampered with.
https://github.com/bazelbuild/bazel/blob/1d20ea76113e49f7fb607ef92231f3665797b8e6/src/main/cpp/util/file_posix.cc#L458

Someone submitted a hack for Scala that touches the files:
https://github.com/bazelbuild/bazel/commit/cb2e03322be3f34b28df22ea17fe05e16fd2466d

It is not clear how to get around this as the original issue and idea is 5 years old. For now, since the fix is simple and there is little downside risk, we can go ahead with this and deal with the core issue at another time.

Try running this before you run bazel:

future_date=$(date +%s -d "+10 years")
find $FOLDER_CACHE/install -type f -exec touch --date=@$future_date {} +

[carpen...@gmail.com]

On windows we see bazel sets the timestamp to of these files to be in the future and it checks they are still in the future as an integrity check. When you use ReFS for example this is put back by the OS and you get this error. 

Could something be adjusting the timestamps on the files?

[Manuel]

Stop caching the ~/.bazel, configure repository cache and disk cache, and keep that around instead

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f7b8d9a7-6751-419a-ada5-a38d42667fdan%40googlegroups.com.

[HaiXin Tie]

Thanks, Manuel! After changing --disk_cache to another location the error went away.

Regards,

HaiXin

On Fri, May 10, 2024 at 1:45 PM Manuel <naranjo...@gmail.com> wrote:

. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

El vie, 10 may 2024, 22:05, HaiXin Tie <thx...@gmail.com> escribió:

~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,

HaiXin

[Manuel]

. cache/bazel is where Bazel will expand all the internals, better to go with cache/bazel-disk or something.

The issue you're hitting is deep in the guts, it happens when you cache the entire cache dir, Bazel gets confused by the mtime of those files which doesn't match expectations 

El vie, 10 may 2024, 22:05, HaiXin Tie <thx...@gmail.com> escribió:

~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,

HaiXin

On Fri, May 10, 2024 at 11:16 AM Manuel <naranjo...@gmail.com> wrote:

[HaiXin Tie]

~/.cache/bazel is used as the disk-cache, with the following line in .bazelrc:

build --disk_cache=~/.cache/bazel

Regards,

HaiXin

On Fri, May 10, 2024 at 11:16 AM Manuel <naranjo...@gmail.com> wrote: