android building server hardware recommendation

[Ping Yin]

Can anyone share your build server hardware configuration and the corresponding build time?

I want to build ICS in less than 10 minutes. With current server, it will cost 30 minutes (with ccache). Any recommendation?

My current server config is

16G memory 4G x 4

ssd disk

Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz

Ping Yin

[Jean-Baptiste Queru]

In order to target sub-10 minute build times (which might not even be
possible), you're going to need some very beefy hardware.

My recommendation would be to have everything at the very least on
SSDs, but possibly in ramdisks instead. Also, you're going to need
several multi-core processors of the latest generation.

Unless you really need very short build latencies, you might find that
you can get a higher throughput for the same investment by having more
cheaper machines.

JBQ

> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Ping Yin]

On Wed, Dec 28, 2011 at 1:59 AM, Jean-Baptiste Queru <j...@android.com> wrote:
>
> My recommendation would be to have everything at the very least on
> SSDs, but possibly in ramdisks instead. Also, you're going to need
> several multi-core processors of the latest generation.
>
> Unless you really need very short build latencies, you might find that
> you can get a higher throughput for the same investment by having more
> cheaper machines.

Thanks. I want to set up a continuous integration system, so i need
the build feedback as quick as possible. What i want is short build
time (maybe 20 minutes is more realistic) with cheap hardware. For
cheap price, i can only use 16G memory (most of desk-level motherboard
only have 4 memory slots). 32G memory (8G x 4) is too expensive
relative to 16G memory. (4G - 20$, 8G - 120$)

Can you share your server config and the build time?

[Jean-Baptiste Queru]

I'm not sure what our build servers use, as those are managed by
another team. Our build times vary from about 15 minutes (fastest
linux boxes) to a bit under 2 hours (slowest Mac builds). Those are
internal Google builds, so they contain a lot more than AOSP.

My primary workstation is a dual E5620, 24GB of RAM, single low-end
HD. Back-to-back AOSP builds are sub-20 minutes with a warm ccache.
It's clearly I/O-limited.

I agree on the price of 8GB memory sticks, right now it seems more
cost-effective to get a fast HD and/or an SSD. I used a 2TB WD Caviar
Black for a while and it was working great for me. Apparently ccache
benefits a lot from being on an SSD, which means that you can do with
a small one (64GB).

JBQ

[Ping Yin]

On Thu, Dec 29, 2011 at 1:04 AM, Jean-Baptiste Queru <j...@android.com> wrote:
> I'm not sure what our build servers use, as those are managed by
> another team. Our build times vary from about 15 minutes (fastest
> linux boxes) to a bit under 2 hours (slowest Mac builds). Those are
> internal Google builds, so they contain a lot more than AOSP.
>
> My primary workstation is a dual E5620, 24GB of RAM, single low-end
> HD. Back-to-back AOSP builds are sub-20 minutes with a warm ccache.
> It's clearly I/O-limited.
>
> I agree on the price of 8GB memory sticks, right now it seems more
> cost-effective to get a fast HD and/or an SSD. I used a 2TB WD Caviar
> Black for a while and it was working great for me. Apparently ccache
> benefits a lot from being on an SSD, which means that you can do with
> a small one (64GB).
>

Thanks. It helps very much.

[Al Sutton]

As a side note; I've played around with using RAM disks and not seen a huge improvement on Mac OS X.

I've got a dual quad core Xeon Mac Pro (i.e. 8 cores) with 32GB of RAM, the OS is on a 1TB Caviar Black HD (which came with the machine), and the Android source is on a 64GB Kingston V Series SSD. I've played around with copying the whole source tree to a RAM disk as well as linking out to point to a RAM disk just before compiling and I've seen my build times come down from around 28 mins to 24 mins when using make -j16.

Doing a bit of analysis showed up the build process starts off with a lot of parallel jobs at the start, but about half way through this drops down and ends up with only 1 or 2 jobs consuming only a few of the cores I have available.

My personal experience would indicate you might be better off spending some time seeing if you can get the build process to use your available cores more efficiently than spending out on better hardware :).

Al.

-- 
T: @alsutton W: www.funkyandroid.com 

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

[Dominic Binks]

On 12/29/2011 9:27 AM, Al Sutton wrote:
> As a side note; I've played around with using RAM disks and not seen a
> huge improvement on Mac OS X.
>
> I've got a dual quad core Xeon Mac Pro (i.e. 8 cores) with 32GB of RAM,
> the OS is on a 1TB Caviar Black HD (which came with the machine), and
> the Android source is on a 64GB Kingston V Series SSD. I've played
> around with copying the whole source tree to a RAM disk as well as
> linking out to point to a RAM disk just before compiling and I've seen
> my build times come down from around 28 mins to 24 mins when using make
> -j16.

Yep - we see this too - basically I reason that once you've done a sync
pretty much everything is in cache anyhow so an SSD really doesn't do
much to help. Might make the sync quicker though....

>
> Doing a bit of analysis showed up the build process starts off with a
> lot of parallel jobs at the start, but about half way through this drops
> down and ends up with only 1 or 2 jobs consuming only a few of the cores
> I have available.

Yep - seen this too both on a much lower spec machine (dual core, 4G RAM
machine which I can build ICS on quite happily - if slowly) and on a 24
core 48G RAM machine - see the same in both. It appears to be the
Dalvik test cases which I believe are only compiled for target builds
and not actually used. I was going to look and see if I can figure out
why this is, but as I will be leaving the Android project before too
long I don't think I'll get to it before I go.

JBQ, do you have any cycles to investigate why this is?

>
> My personal experience would indicate you might be better off spending
> some time seeing if you can get the build process to use your available
> cores more efficiently than spending out on better hardware :).
>
> Al.
> --

> T: @alsutton W: www.funkyandroid.com <http://www.funkyandroid.com>

>
> The views expressed in this email are those of the author and not
> necessarily those of Funky Android Limited, it's associates, or it's
> subsidiaries.
>

--
Dominic Binks: dbi...@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

[daern]

I've got a slightly related question:

Is it possible to distribute the Android build process over several
physical servers / disk arrays to further reduce the build time? Has
anyone (Google?) done this and can they give any advice as to how they
automated the process?

Thanks,

Daern

[Jean-Baptiste Queru]

On our side (Google), the results have been inconclusive at best, and
sometimes even negative (i.e. depending on the tools we try to use,
we've seen distributed builds take longer than local builds on the
master, because of the RPC cost of stat()ing all the relevant files
back on the master).

Since we (Google) feel that we are more limited by the throughput of
our build farm than by its latency, we tend to prefer solutions that
improve the build throughput, and we strongly think that have each of
our server run a single local build works better than distributing
each builds across multiple servers.

JBQ

> --
> You received this message because you are subscribed to the "Android Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

--

[daern]

Thanks for the info. I'll stick to the "keep it simple" approach
then ;-)

[Ping Yin]

On Fri, Dec 30, 2011 at 1:33 AM, Dominic Binks <dbi...@codeaurora.org> wrote:
>
> Yep - seen this too both on a much lower spec machine (dual core, 4G RAM
> machine which I can build ICS on quite happily - if slowly) and on a 24 core
> 48G RAM machine - see the same in both.

I am very interested in the build time on the 24-core machine. Can you share?

[Ping Yin]

On Fri, Dec 30, 2011 at 1:27 AM, Al Sutton <a...@funkyandroid.com> wrote:
> As a side note; I've played around with using RAM disks and not seen a huge
> improvement on Mac OS X.
>
> I've got a dual quad core Xeon Mac Pro (i.e. 8 cores) with 32GB of RAM, the
> OS is on a 1TB Caviar Black HD (which came with the machine), and the
> Android source is on a 64GB Kingston V Series SSD. I've played around with
> copying the whole source tree to a RAM disk as well as linking out to point
> to a RAM disk just before compiling and I've seen my build times come down
> from around 28 mins to 24 mins when using make -j16.
>
> Doing a bit of analysis showed up the build process starts off with a lot of
> parallel jobs at the start, but about half way through this drops down and
> ends up with only 1 or 2 jobs consuming only a few of the cores I have
> available.

Thanks for your info. I will share my build config and build time when possible.