how to distribute android 10/11 ?

1,855 views
Skip to first unread message

alexander zhu

unread,
Mar 1, 2020, 5:39:35 PM3/1/20
to Android Building
Hi,

I faced one problem, i have tried to solve it many days.


AOSP compile on one powerful physical server(36U128G) is fast . Because my company has so many people need to compile AOSP on the server,so it is busy all the time , and I do not have money to buy more.But the VM I can request more.So I want to use VM on the cloud to compile AOSP.

I have some distribute compile experiences in other c/c++ projects. So I'm thinking is there any way to compile AOSP in distribute way?exp: use many VMs for one AOSP build.


I tried distcc , but failed . AOSP switching to clang, but distcc support clang not ok.https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926884


So, I want to know ,AOSP in google , how to solve the long build time ?use more powerful physical server? or you have your own distribute compile solution?

Orion Hodson

unread,
Mar 2, 2020, 2:52:27 AM3/2/20
to android-...@googlegroups.com
Hi Alexander

Internally Android developers use goma. The thread below has pointers to the client and server. It is built on GCP.

Kind regards
Orion

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/e3dd71e8-e452-48e2-8686-c51e359cbf5c%40googlegroups.com.

Dan Willemsen

unread,
Mar 3, 2020, 4:57:32 PM3/3/20
to Android Building, Igor Foox
The build team does not recommend using Goma -- it does not work particularly well with the Android build, and we expect to be removing support for it in master in the coming months (support will most likely still be in this years release, but not next years).

RBE is the replacement, but it's still evolving at the moment. Many of our CI builds have switched to it (they never used Goma), but only a handful of developers have used it so far. If you're interested in this, I'd highly recommend asking your TAM about it (or if you don't have one, contact me and I can try to route your request to the right team).

For what we're doing (other than Goma/RBE), the majority of our CI builds are done on n1-standard-64 machines with SSD persistent disks. Depending on your limitations/requirements, the newer C2/N2 machine families may be more efficient. For local developers, we're generally buying fast desktops -- 2x Intel Xeon GOLD 6154 processor (18-core Skylake, up to 3.70 GHz) with 192 GB RAM and a few TB of SSDs.

- Dan

Dan Willemsen

unread,
Mar 3, 2020, 5:00:40 PM3/3/20
to Android Building, Igor Foox
Oh, and for developer builds (and many frequent CI builds), we're almost always using incremental builds, so the clean build times are somewhat less limiting. If you're often doing clean builds or very large builds, that's probably one of the first things to look at.

- Dan

alexander zhu

unread,
Mar 18, 2020, 9:39:31 PM3/18/20
to Android Building
hi,Dan

I'm so glad to get your message.specially, the "buying fast desktops "parts . I did the same thing . please give my request to the right team. So many thanks to you . I will continue to follow the RBE things. 

在 2020年3月4日星期三 UTC+8上午6:00:40,Dan Willemsen写道:

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-...@googlegroups.com.

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-...@googlegroups.com.

alexander zhu

unread,
Mar 19, 2020, 12:07:39 PM3/19/20
to Android Building
hi,
Dan,

does RBE is your GCP service?  i want to know how it works


在 2020年3月4日星期三 UTC+8上午6:00:40,Dan Willemsen写道:
Oh, and for developer builds (and many frequent CI builds), we're almost always using incremental builds, so the clean build times are somewhat less limiting. If you're often doing clean builds or very large builds, that's probably one of the first things to look at.

- Dan


For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-...@googlegroups.com.

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-...@googlegroups.com.

Duslabo N

unread,
Mar 19, 2020, 12:44:25 PM3/19/20
to android-...@googlegroups.com, Igor Foox
Cool,

Does anybody did benchmarking of aosp clean build on AMD Reyzen threadripper?

Regards,
Duslabo

alexander zhu

unread,
Mar 20, 2020, 2:46:43 AM3/20/20
to Android Building
hi,

Dan,

I got one ppt, which is 'Engineering productivity talk' in 2020,which pages mention you. I read it and go through the pages and overview(https://cloud.google.com/sdk/gcloud/reference/alpha/remote-build-execution). It looks like the result is what I'm looking for.

result.png




I have some quetions:

1. does the result use Android R master? what is the number of “-j” you use? because I faced one problem, which is the larger the "-j" is the java OOM error comes out.In my case , 32U128G server,I only can set -j72. if the number bigger than this number there is big chance to face the JAVA OOM(system is out of resource).
2. Does RBE build the binary(.o or .so or jar) to put it into remote cache, and next time some one build ,the new one will use the binary instead of real compiling ? how does this work? each action has one md5 ?
3. how to solve java action in build ? I tried distcc ,in some modules it can work, but java part. 
4. so many details I want to know .I want to build one thing like this to solve my Android build so slow problem

在 2020年3月20日星期五 UTC+8上午12:07:39,alexander zhu写道:

唐木村

unread,
Feb 3, 2021, 12:30:32 PM2/3/21
to Android Building
@Dan Willemsen

>>> The build team does not recommend using Goma -- it does not work particularly well with the Android build, and we expect to be removing support for it in master in the coming months (support will most likely still be in this years release, but not next years).

whether goma not work well for android build ??? and now 2021 year support better ???

Dan Willemsen

unread,
Feb 16, 2021, 5:14:19 PM2/16/21
to Android Building
GOMA support has been removed on the AOSP master branch, and has been replaced with RBE.

- Dan

Eli Chang

unread,
Feb 24, 2021, 3:29:00 AM2/24/21
to Android Building
Hi Dan,

I'm very interested in RBE, but no TAM. Could you tell me the specific method of building RBE on AOSP? 

Is RBE part of Bazel? Is there any document for reference?

thanks

在2020年3月4日星期三 UTC+8 上午5:57:32<Dan Willemsen> 写道:

Ulf Adams

unread,
Apr 28, 2021, 10:54:57 AM4/28/21
to android-...@googlegroups.com
Hi Eli,

RBE stands for Remote Build Execution. It is not built into Bazel, but there are a couple of open source as well as commercial options, documented here: https://docs.bazel.build/versions/4.0.0/remote-execution.html

Full disclosure: I work for one of the commercial vendors in this space and we contributed some patches in the past to improve AOSP builds. Last I checked, we could successfully build AOSP in ~20 minutes in the best case scenario.

Cheers,

-- Ulf

EngFlow GmbH
Fischerweg 51, 82194 Gröbenzell, Germany
Amtsgericht München, HRB 255664
Geschäftsführer (Managing Director): Ulf Adams


Ruben Wu

unread,
May 14, 2021, 6:37:26 PM5/14/21
to Android Building
Hi Dan,
While I understand goma support is being deprecated/removed on AOSP master, is there a way to still be able to use Goma on Android 12 since I already have a goma client and server setup to work with Android 10.

Best,
Ruben

Dan Willemsen

unread,
May 14, 2021, 6:40:44 PM5/14/21
to Android Building
While I understand goma support is being deprecated/removed on AOSP master, is there a way to still be able to use Goma on Android 12 since I already have a goma client and server setup to work with Android 10.

No, it was removed before Android 12 was branched, so it's not in Android 12 either. You could always attempt to revert those removals in your own tree, but I don't know how complex that could be. RBE works with significantly more of the build actions than GOMA ever did.

- Dan

Ruben Wu

unread,
May 18, 2021, 4:48:21 PM5/18/21
to Android Building
Thanks for the info, Dan! Is there a place I can reference on how to turn on RBE on Android 11/12? I understand a remote build execution engine/service is needed, but how do I configure Android 11 to send jobs to that service?

Thanks,
Ruben

Eli Chang

unread,
May 26, 2021, 12:35:55 PM5/26/21
to Android Building
Thanks for your reply, and that`s very impressive. I already read the document and some examples are implemented, but there is still a long way to applied on AOSP. 

We will focus on that and looking forward to your updates.

Ulf Adams

unread,
May 27, 2021, 2:26:45 PM5/27/21
to android-...@googlegroups.com
Hi Ruben!

Enabling RBE once you have a remote execution cluster running requires approximately these steps:

    source build/make/rbesetup.sh
    ulimit -n 20000

    export RBE_service=<hostname>:<port, typically 443>
    export RBE_tls_ca_cert=certificate.crt <-- depends on what TLS cert is in use on the service
    export RBE_use_rpc_credentials=false <-- depends on how you do client authentication
    export RBE_JAVA_POOL=default
    export RBE_METALAVA_POOL=default
    export RBE_ABI_DUMPER=1
    export RBE_ABI_LINKER=1
    export RBE_CLANG_TIDY=1
    export RBE_CXX_LINKS=1
    export RBE_D8=1
    export RBE_JAVAC=1
    export RBE_METALAVA=1
    export RBE_R8=1
    export RBE_SIGNAPK=1
    export RBE_TURBINE=1
    export RBE_ZIP=1

    export RBE_ABI_DUMPER_EXEC_STRATEGY=remote_local_fallback
    export RBE_ABI_LINKER_EXEC_STRATEGY=remote_local_fallback
    export RBE_CXX_EXEC_STRATEGY=remote_local_fallback
    export RBE_CXX_LINKS_EXEC_STRATEGY=remote_local_fallback
    export RBE_CLANG_TIDY_EXEC_STRATEGY=remote_local_fallback
    export RBE_D8_EXEC_STRATEGY=remote
    export RBE_JAR_EXEC_STRATEGY=remote_local_fallback
    export RBE_JAVAC_EXEC_STRATEGY=remote
    export RBE_SIGNAPK_EXEC_STRATEGY=remote_local_fallback
    export RBE_TURBINE_EXEC_STRATEGY=remote_local_fallback
    export RBE_R8_EXEC_STRATEGY=remote
    export RBE_ZIP_EXEC_STRATEGY=remote_local_fallback
    export RBE_METALAVA_EXEC_STRATEGY=remote_local_fallback

You can also adjust the number of remote actions like so:
    export NINJA_REMOTE_NUM_JOBS=8

You may need to run a Docker registry, or you can pull images from GCR.

And then you can use the `m` tool as usual
    mmma external/sqlite/
    m

Hope that helps,

-- Ulf

EngFlow GmbH
Fischerweg 51, 82194 Gröbenzell, Germany
Amtsgericht München, HRB 255664
Geschäftsführer (Managing Director): Ulf Adams

Ruben Wu

unread,
Jul 31, 2021, 9:44:32 AM7/31/21
to Android Building
Hi Ulf,

Thanks for the reply, could you elaborate on RBE_use_rpc_credentials? If this needs to be true, how do I set the RPC credentials?

Also, is there any plans to open source the RBE client code (whatever generates the reproxy and rewrapper), I'm curious on the possible flags and functionality of the RBE client?

Best,
Ruben

Eli Chang

unread,
Aug 3, 2021, 11:49:01 PM8/3/21
to Android Building
Hi  Ulf,

Did  RBE means remote build executor?

And if so, by what Ruben said, we also need the RBE service. How can I  configure the RBE service?

Best,
Eli

Reply all
Reply to author
Forward
0 new messages