FYI: prow using bazel with remote build execution

[Erick Fejta]

Hello kubernetes community,

tl;dr - we are using distributed bazel builds in addition to distributed caching; no developer impact expected. Contact us if you see strange errors, or want to try this out on your repo.

Feel free to stop reading if you do not use prow presubmits or bazel. The rest of the email should be irrelevant.

During April the kubernetes/test-infra repo converted its bazel presubmits to do work remotely using remote build execution or RBE (docs, presentation). This means bazel moves build and test work out of the prow pod and into a remote cluster of build machines.

We have now also migrated some kubernetes/kubernetes presubmits to use RBE. Currently pull-kubernetes-bazel-test and pull-kubernetes-bazel-build (more to follow).

When this happens the build log will include remote in the status message, for example:

[8,884 / 10,334] 289 / 825 tests; GoLink staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps/kunstruct/linux_amd64_race_stripped/go_default_test; 12s remote ... (366 actions, 365 running)

We expect local and distributed builds to behave the same way. Please let us know if you believe you are seeing errors unique to RBE so we can investigate.

Also, please reach out (#sig-testing or #bazel on slack, or a test-infra issue) if:

• Your repo already uses bazel and prow and want to turn on RBE, or
  
• You would like help migrating your repo and its jobs to use bazel, RBE and prow.

PS: the initial goals here are to

• reduce build times, and
• pull-test-infra-bazel ~6m with remote execution and caching, ~8m using only remote caching and
• pull-kubernetes-bazel-test ~7m with remote execution and caching, ~9m using only remote caching
• better distribute compute load
• e2e jobs no longer need to allocate 8 cores for 60m just to speed up the initial 10m build. Tying up these cores for the next 50m while mostly waiting for http responses is wasteful.
• This remains a work in progress

Thanks and happy developing!

[Clayton Coleman]

Is this running RBE on google infra, or something in the community test-infra?

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAMMDcuHf7Y8tq2KpdNbwYQ8eFSGqs%3DRwi%2BPua9fVprZquk6M_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Erick Fejta]

Google. 

Right now it uses the same GCP

 project as prow, which is still on Google infra. 

It's easy to change the GCP project to whatever, just need to make sure that project has an RBE cluster and give the service account the right I AM grants to that project.

Thus far no one on the test-infra oncall rotation has any access to community owned projects. If we get it we can move it over there. 

[Davanum Srinivas]

Erick,

wg-k8s-infra would happily create a cluster and grant access to a community owned cluster. Can someone in the oncall rotation please open an issue in k/k8s.io repo with details? (how big etc ... and the list of people who need access)

Thanks,

Dims

You received this message because you are subscribed to the Google Groups "kubernetes-sig-testing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-te...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-testing/CAMMDcuG2tHJaEbs32kyX5%2BfGrejJcaux1Eq07BwTxwchx_JyHQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Davanum Srinivas :: https://twitter.com/dims

[Erick Fejta]

oncall should have access to a project, not a cluster in that project. An RBE cluster is owned by the project but is not a GKE cluster (creation scripts here, which will require whitelisting either the user making the call or else the project for the alpha feature)

[Davanum Srinivas]

Ack. let's please details here : https://github.com/kubernetes/k8s.io/issues/249