FYI: prow using bazel with remote build execution

[Erick Fejta]

Hello kubernetes community,

tl;dr - we are using distributed bazel builds in addition to distributed caching; no developer impact expected. Contact us if you see strange errors, or want to try this out on your repo.

Feel free to stop reading if you do not use prow presubmits or bazel. The rest of the email should be irrelevant.

During April the kubernetes/test-infra repo converted its bazel presubmits to do work remotely using remote build execution or RBE (docs, presentation). This means bazel moves build and test work out of the prow pod and into a remote cluster of build machines.

We have now also migrated some kubernetes/kubernetes presubmits to use RBE. Currently pull-kubernetes-bazel-test and pull-kubernetes-bazel-build (more to follow).

When this happens the build log will include remote in the status message, for example:

[8,884 / 10,334] 289 / 825 tests; GoLink staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps/kunstruct/linux_amd64_race_stripped/go_default_test; 12s remote ... (366 actions, 365 running)

We expect local and distributed builds to behave the same way. Please let us know if you believe you are seeing errors unique to RBE so we can investigate.

Also, please reach out (#sig-testing or #bazel on slack, or a test-infra issue) if:

• Your repo already uses bazel and prow and want to turn on RBE, or
  
• You would like help migrating your repo and its jobs to use bazel, RBE and prow.

PS: the initial goals here are to

• reduce build times, and
• pull-test-infra-bazel ~6m with remote execution and caching, ~8m using only remote caching and
• pull-kubernetes-bazel-test ~7m with remote execution and caching, ~9m using only remote caching
• better distribute compute load
• e2e jobs no longer need to allocate 8 cores for 60m just to speed up the initial 10m build. Tying up these cores for the next 50m while mostly waiting for http responses is wasteful.
• This remains a work in progress

Thanks and happy developing!