Build AOSP 11 with Google RBE service

[John Delight]

Hi Android Building,

I am following the site (https://groups.google.com/g/android-building/c/EJradKEiPl0) to build AOSP 11 with Google RBE service. I have created the service account and enabled the "Remote Build Execution API" service in Google Cloud. After setup env variables, the make always show a TLS CA cert issue. Could you show me the correct way to get the TLS CA cert for Google RBE service?Thanks!

Below is env variables setup:

    export RBE_service=remotebuildexecution.googleapis.com:443
    export RBE_tls_ca_cert=rbe.crt
    export RBE_use_rpc_credentials=true
    export GOOGLE_APPLICATION_CREDENTIALS=loyal-theater-204209-c0471ad7c674.json

    ...

I got rbe.cert using command "openssl s_client -connect remotebuildexecution.googleapis.com:443 -showcerts" and converted to DER format, and loyal-theater-204209-c0471ad7c674.json is my service account key file.

Afte make, it always output below error, seems the TLS CA cert is wrong:

E0626 12:33:13.387181   10559 bootstrap.go:91] Unable to start reproxy: F0626 12:32:53.444195   10569 main.go:122] Error connecting to remote execution client: Could not create TLS config: failed to load TLS CA certificates from rbe.crt

I tried several ways to get the TLS CA cert, neither works, it always show above error. Really appreciated your help on it. Thank! 

Regards,

John

[Amith Dsouza]

Hi John,

Google's RBE service is not generally available for building Android Platform, please take a look at alternative options listed here - https://bazel.build/community/remote-execution-services.

Regards,

[John Delight]

Thanks Amith for the clarification. According to build logs in below link, you are using RBE in daily build. Is it possible to let people outside your company to have a try to see the potential of reducing the AOSP building time using RBE?

https://ci.android.com/builds/submitted/8795921/aosp_arm64-userdebug/latest/view/logs/build.log

In our company the AOSP 11 building time is 1.5hr which is still slow, as daily build and dev build are heavily depending on it. I have seen the large potential of RBE to improve the build efficiency,  really appreciate if you could allow me to have a try on it. Thanks!

Btw, I have checked services in your link (https://bazel.build/community/remote-execution-services), seems they are all for Bazel project, but AOSP is based on Android.bp not Bazel. It will be much helpful if you could point me to a workable 3rd-party solution. Many thanks!

Regards,

John 

[Dan Willemsen]

In our company the AOSP 11 building time is 1.5hr which is still slow, as daily build and dev build are heavily depending on it. I have seen the large potential of RBE to improve the build efficiency,

That's your clean build time, right? Are you using incremental builds when possible? Many of our CI builds were incremental builds first, which sped things up substantially, then remote execution got added to speed things up further. RE without incremental builds still isn't as good as RE with incremental builds in terms of speed (our releases are still non-incremental builds, and incremental builds are not feasible all the time, so we still do both).

Btw, I have checked services in your link (https://bazel.build/community/remote-execution-services), seems they are all for Bazel project, but AOSP is based on Android.bp not Bazel. It will be much helpful if you could point me to a workable 3rd-party solution. Many thanks!

 

While Bazel was the original user of the remote execution (RE) API, it is a general API usable by multiple different clients, just like it's able to be implemented by multiple different services. I don't know which services claim support or have been tested with Android builds, but in theory it should be possible to make many of them work.

- Dan

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/89acfd24-3e87-45dd-b6e1-2b10e9545328n%40googlegroups.com.

[John Delight]

Hi Dan,

Yes, 1.5hr is the clear build time in my company, it is used in CI daily build, dev build and Gerrit-trigger build. Just want to clarify with you the actual meaning of "incremental build" you mentioned, is it keeping the "out" directory from last build or meaning ccache or remote caching?

For incremental build (keeping the "out" directory from last build), previously we had met some kinds of compilation errors, so finally we decided to not use it.

We have already used CCACHE, and want to try RE + RC (remote caching) to reduce the building time further.

Do you have some benchmark data for your RE + RC service? Meanwhile I will try some 3rd-party RE services to see whether workable.

Many thanks for your support.

Regards,

John

[Faqiang Zhu]

Hi John,

Have you verified any 3rd-paty RE services work well?

I tried the BuildGrid, based on it's documentation, with bazel to distribute build action to BuildGrid can work. and then try to use the BuildGrid with Android build, but just got below errors:

10:26:18 Unable to start RBE reproxy
FAILED: RBE bootstrap failed with: exit status 10
E0216 10:26:18.180877 3164959 bootstrap.go:96] Unable to start reproxy: "E0216 10:26:13.389380 3164973 main.go:205] Failed to initialize remote-execution client: couldn't create RPC creds for https://www.googleapis.com/auth/cloud-platform: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information., authentication type (identity) used=\"application default credentials\"\n"

Try restarting the build after running the following command:
    gcloud auth application-default login --no-launch-browser --disable-quota-project

This makes me really puzzled, because I'm trying to use buildgrid deployed in my local env, not on the google cloud,  and the fail log seems to be related to google could authentication.

what 3rd-party RE services have you tried? is there any one can work well?