Remote Cache with Google Cloud Storage provides no net benefit

[Steve Munday]

Hi there!

We have been using Bazel for a while and I recently moved us into Circle CI.  We are running Android builds with a significant C++ native codebase inside a Docker container.

Because Circle creates a new Docker environment for each build, I implemented Bazel Remote Cache so that we weren't doing a full clean build every time.  So we now have an added benefit of using a remote cache, but a drawback of increased network bandwidth.  

• The net benefit has been about zero – our build times in Circle CI before remote cache were ~13 minutes and afterwards about the same.  
  
• When I ran this the first time the build time was ~37 minutes, which was expected as Bazel filled the remote cache.  
• Total actions for a clean build is about 1750, and our remote cache has roughly 5000 objects in it, all dated from the first run.

Am I doing something wrong here?  How can I optimize our remote cache to actually see some benefit?

Thanks in advance, 

Steve

My .bazelrc file:

build --spawn_strategy=remote --genrule_strategy=remote

build --strategy=Javac=remote --strategy=Closure=remote

build --remote_http_cache=https://storage.googleapis.com/<bucket-name>

build --google_credentials=${HOME}/gcloud-service-key.json

my tools/bazel.rc file:

build --copt=-Wall --copt=-Wextra --cxxopt=-std=c++11
build --copt=-Wno-sign-compare --copt=-Wno-unused-parameter
build --spawn_strategy=standalone 
build --android_crosstool_top=@androidndk//:toolchain-libcpp

build --fat_apk_cpu=armeabi,armeabi-v7a

[Shimin Guo]

We use the following flags

            --remote_http_cache=https://<GCS URL> \

            --experimental_remote_spawn_cache \

            --google_credentials=<credentials.json>

We see a pretty significant speedup. Our code is mostly go. Also our build machines are in the same GCP zone as the GCS bucket.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/26081486-c788-4386-baa5-7e46696dec68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fedor Korotkov]

Steven, if you are not using Circle CI Enterprise then you builds are executed in AWS so network latency is not on your side. You can try to use machine type so your build will be executed in a preemptible VM in GCP to make sure it's the latency issue.

On a side note, when I was tuning Cirrus CI's HTTP Cache for Bazel I noticed that Bazel uploads and downloads a lot of very small metadata files. In some cases even establishing http connections for each request were taking substantial amount of time. That's why I had to add a local proxy running on localhost that has a persistent connection for loading and uploading artifacts.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/26081486-c788-4386-baa5-7e46696dec68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOBFCepg_WBiT8HoOjXRz0j%2Bd9Nce9rcyrreOS4nGKgqf2sGWA%40mail.gmail.com.

[st...@interaxon.ca]

We are not using Circle CI enterprise so the latency between GCP and AWS might be the cause. I did not know Circle uses both for their build workers that’s a good suggestion. Our docker image has a lot of stuff in it so switching to machine type is not trivial.

[Jakob Buchgraber]

Hi Steve,

I think there's a good chance that the latency to GCS is too high and the bandwidth too low. The initial 37min build time is a good indication that you are spending tens of minutes uploading to GCS. Latency and bandwidth are absolutely critical for good remote caching performance.

I think GCS is not ideal for you in this case. A developer from CircleCI has developed a remote cache for Bazel [1] that can use AWS S3 as a backend. I believe this might be a better fit for you. The Angular project is using it for their CircleCI builds.

Please let me know how it goes!

Cheers,

Jakob

[1] https://github.com/notnoopci/bazel-remote-proxy

[Steve Munday]

Thanks @Jakob, this looks amazing and exactly what I need.  

It also looks like this bazel-remote-proxy has implemented using Circle CI built-in cache as a backend for this proxy so I don't even have to create an S3 bucket.  This is great because all of our cloud infrastructure is on GCP so I'd like to avoid setting up an account on S3.  I'm testing this now but getting errors right now.  I'll post an issue in Github, but this is what I'm seeing:

ERROR: /home/circleci/<PATH>/BUILD:286:1: Executing genrule //libmuse:muse_bridge_h failed (Exit 34). Note: Remote connection/protocol failed with: execution failed