Machine Type, Build Times
TrevM
Please be gentle with me, very much Newbie.
I'd like to get involved as Contributer, that means building
Gingerbread, Ice Cream Sandwich etc.
Information concerning how to build, how to get source etc. seems ok
(just from reading).
But no detail of build machine spec, no recomended, no minimum spec.
Also no detail at all of times (which also relates to machine spec).
Basically, my PC is a bit dated and even if it wasn't, converting to
Ubuntu is a large task in itself, so basically I need to invest hard
cash. For that I need to know what to buy, how much it is going to
cost etc.
I have spent a couple of weeks just looking, ok so it's multi-
trhreaded (-jx), so iCore 7 will give up to 6 cores, 2 threads per
core (Hyperthreading), so -j12, but what memory requirements?
On the other hand, iCore 7 only allows for single processor, with Xeon
I could have dual processor, dual 6 core, 2 threads per core, so -j24,
but again what memory, how much per thread.
On top of all that, it's basically file crunching, 3GB per second, 6GB
per second drives?
All this is high spec gear, specialised for this task, so we are
talking serious money with no much re-use.
Any suggestions / information would be gratefully received.
BR, TrevM
Jean-Baptiste Queru
as important and you'll mostly be limited by how quickly your CPU can
process data. Since memory usage is concentrated on a few individual
tasks, this is mostly independent from the number of threads. If you
use less RAM, I/O speed will matter, but the limiting factor will be
seek time, not throughput.
I personally use 2 -j tasks for each hyperthread, so on my
hyperthreaded dual-quad-core (8 cores total, 2 threads per core) I use
-j32.
JBQ
> --
> You received this message because you are subscribed to the "Android Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.
Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.
Pete
Jean-Baptiste Queru
as soon as one of the tasks needs to wait for the disk. On my
16-thread machine, -j32 is measurably faster than -j16.
Intermediate values tend to be somewhat unpredictable, as I'm guessing
that the kernel has a harder time getting the tasks to stick to CPUs.
JBQ
TrevM
using -j32.
Roughly how long does a full Galaxy Nexus build take?
Pete has single Intel Core i7 640M at 2.8 GHz? using -j4 but can
actually use -j16.
How much RAM?
Which builds do you do?
Roughly how long for each?
Purely for interest:
I have worked on projects where build was ARM based phones, built on
Linux, roughly 200 SW Engineers used to use shared build farms, high
speed file servers linked to numbers of high speed multi-processor
servers all linked together by giga-bit connections. Full build
approximately 20 minutes, re-build approximately 5 minutes. Build
times and loading constantly monitored.
But now I want to do this from home, reason for my being overly nosey,
I would like reasonable build times, but don't want to go bankrupt or
end up divorced ;-)
TrevM
Jean-Baptiste Queru
I do a full clean build of full_maguro-userdebug from a cold start in
about 30 minutes, at -j32.
JBQ
Pete
Xianghua Xiao
-j8
32 minutes for the whole panda build.
Randy
Not bad Pete
--
Dominic Binks
one thing is abundantly clear - the designers of the Android build
system did a really good job - virtually the entire build is CPU bound
not disk or RAM bound.
Thus more cores is better than more RAM. My lowly desktop machine
builds ICS in about 90 minutes (-j8). Despite having only 4G of RAM it
is CPU bound for > 90% of the build, not memory bound only touching peak
ram usage a couple of times briefly throughout the build.
The same is true for our big build servers - 12 cores (-j 24) and 72G of
RAM which build in under 20 minutes.
I'm trying to profile more slower cores vs fewer faster cores, but I
suspect that the greater number of cores will perform better (or at
least better vs price point).
Dominic
On 11/21/2011 1:34 PM, Randy wrote:
> Not bad Pete
>
> On Nov 21, 2011 3:22 PM, "Pete" <peteral...@gmail.com
> <mailto:peteral...@gmail.com>> wrote:
>
> HP DM4 (Laptop)
> i7 640M
> 8 GB RAM
> 7200 RPM HD
>
> Takes me 85 minutes using make -j8 otapackage for my Maguro build.
>
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to
> android-...@googlegroups.com
> <mailto:android-...@googlegroups.com>
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> <mailto:android-building%2Bunsu...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
>
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
--
Dominic Binks: dbi...@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
Gavin Gu
I've been doing some profiling of our builds on the machines we use and one thing is abundantly clear - the designers of the Android build system did a really good job - virtually the entire build is CPU bound not disk or RAM bound.
Thus more cores is better than more RAM. My lowly desktop machine builds ICS in about 90 minutes (-j8). Despite having only 4G of RAM it is CPU bound for > 90% of the build, not memory bound only touching peak ram usage a couple of times briefly throughout the build.
The same is true for our big build servers - 12 cores (-j 24) and 72G of RAM which build in under 20 minutes.
I'm trying to profile more slower cores vs fewer faster cores, but I suspect that the greater number of cores will perform better (or at least better vs price point).
Dominic
On 11/21/2011 1:34 PM, Randy wrote:
<mailto:peteralfonsojr@gmail.com>> wrote:
HP DM4 (Laptop)
i7 640M
8 GB RAM
7200 RPM HD
Takes me 85 minutes using make -j8 otapackage for my Maguro build.
--
You received this message because you are subscribed to the "Android
Building" mailing list.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en
--
You received this message because you are subscribed to the "Android
Building" mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
--
Dominic Binks: dbi...@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
feng shi
- DDR3 Memory 2133MHz 16GB
- 2 Plextor SSDs, 6Gbit, each 128GB, RAID0
- Fedora 16
It cost 25minutes to build ICS 4.0.1_r1 and less than 10 minutes to
build GB 2.3.7_r1
All cores' run nearly 100% and memory occupation didn't exceed 4GB.
It's also very fast if only use one SSD instead of RAID 0.
Feng Shi
Beijing, China
On 11月22日, 下午6时30分, Gavin Gu <qinwei...@gmail.com> wrote:
> Core i7 2600 with 8GB memory, -j8.
>
> total build time is ~35 minutes
>
> On Tue, Nov 22, 2011 at 6:04 AM, Dominic Binks <dbi...@codeaurora.org>wrote:
>
>
>
>
>
>
>
> > I've been doing some profiling of our builds on the machines we use and
> > one thing is abundantly clear - the designers of the Android build system
> > did a really good job - virtually the entire build is CPU bound not disk or
> > RAM bound.
>
> > Thus more cores is better than more RAM. My lowly desktop machine builds
> > ICS in about 90 minutes (-j8). Despite having only 4G of RAM it is CPU
> > bound for > 90% of the build, not memory bound only touching peak ram usage
> > a couple of times briefly throughout the build.
>
> > The same is true for our big build servers - 12 cores (-j 24) and 72G of
> > RAM which build in under 20 minutes.
>
> > I'm trying to profile more slower cores vs fewer faster cores, but I
> > suspect that the greater number of cores will perform better (or at least
> > better vs price point).
>
> > Dominic
>
> > On 11/21/2011 1:34 PM, Randy wrote:
>
> >> Not bad Pete
>
> >> On Nov 21, 2011 3:22 PM, "Pete" <peteralfons...@gmail.com
> >> <mailto:peteralfonsojr@gmail.**com <peteralfons...@gmail.com>>> wrote:
>
> >> HP DM4 (Laptop)
> >> i7 640M
> >> 8 GB RAM
> >> 7200 RPM HD
>
> >> Takes me 85 minutes using make -j8 otapackage for my Maguro build.
>
> >> --
> >> You received this message because you are subscribed to the "Android
> >> Building" mailing list.
> >> To post to this group, send email to
> >> android-building@googlegroups.**com<android-...@googlegroups.com>
> >> <mailto:android-building@**googlegroups.com<android-building@googlegroups.c om>
>
> >> To unsubscribe from this group, send email to
> >> android-building+unsubscribe@**googlegroups.com<android-building%2Bunsubscr i...@googlegroups.com>
> >> <mailto:android-building%**2Bunsu...@googlegroups.com<android-building% 252Buns...@googlegroups.com>
> >> **>
>
> >> For more options, visit this group at
> >> http://groups.google.com/**group/android-building?hl=en<http://groups.google.com/group/android-building?hl=en>
>
> >> --
> >> You received this message because you are subscribed to the "Android
> >> Building" mailing list.
> >> To post to this group, send email to android-building@googlegroups.**com<android-...@googlegroups.com>
> >> To unsubscribe from this group, send email to
> >> android-building+unsubscribe@**googlegroups.com<android-building%2Bunsubscr i...@googlegroups.com>
> >> For more options, visit this group at
> >>http://groups.google.com/**group/android-building?hl=en<http://groups.google.com/group/android-building?hl=en>
>
> > --
> > Dominic Binks: dbi...@codeaurora.org
> > Employee of Qualcomm Innovation Center, Inc.
> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
>
> > --
> > You received this message because you are subscribed to the "Android
> > Building" mailing list.
> > To post to this group, send email to android-building@googlegroups.**com<android-...@googlegroups.com>
> > To unsubscribe from this group, send email to
> > android-building+unsubscribe@**googlegroups.com<android-building%2Bunsubscr i...@googlegroups.com>
> > For more options, visit this group at
> >http://groups.google.com/**group/android-building?hl=en<http://groups.google.com/group/android-building?hl=en>
feng shi
> > >> <mailto:android-building@**googlegroups.com<android-build...@googlegroups.c om>
Pete
Al Sutton
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)
T: @alsutton
G+: http://goo.gl/ymi9b
The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.
Hi JBQ,it's time for me to upgrade my main machine. I found this thread while looking for HW related info about building AOSP. I'm currently using a not-so-brand-new AMD Phenom II 1090T six-core (8GB RAM, SATA3 SSD) running Ubuntu12/04. On this machine a clean build of ICS master / maguro (rm -rf out; time make -j6) takes 25 minutes 30 seconds consistently.real 25m30.071suser 119m58.230ssys 7m23.100sI'm a little surprised to read that a dual Xeon machine would take 20% longer than my old PC. Do you happen to have any new data?Can someone possibly provide clean build times for Ivy bridge and/or K8 cpu's? Thank you very much.
Am Montag, 21. November 2011 21:12:13 UTC+1 schrieb Jean-Baptiste Queru:
I have a dual Xeon E5620 (quad-core hyperthreaded), with 24GB of RAM.
I do a full clean build of full_maguro-userdebug from a cold start in
about 30 minutes, at -j32.JBQ
--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.
Magnus Bäck
> On 16 Jun 2012, at 23:41, othy74 wrote:
>
> > it's time for me to upgrade my main machine. I found this thread
> > while looking for HW related info about building AOSP. I'm currently
> > using a not-so-brand-new AMD Phenom II 1090T six-core (8GB RAM,
> > SATA3 SSD) running Ubuntu12/04. On this machine a clean build of ICS
> > master / maguro (rm -rf out; time make -j6) takes 25 minutes 30
> > seconds consistently.
> >
> > real 25m30.071s
> > user 119m58.230s
> > sys 7m23.100s
> >
> > I'm a little surprised to read that a dual Xeon machine would take
> > 20% longer than my old PC. Do you happen to have any new data?
> >
> > Can someone possibly provide clean build times for Ivy bridge and/or
> > K8 cpu's? Thank you very much.
for various configurations. I generally find them boring as the machines
often have very different configurations, different file systems, and
are measured by different people.
Now, given that you have six cores but only 8 GB RAM I suspect that
getting more RAM would give you the most bang for you buck.
> Do you have Ccache enabled? The reason I'm asking is because your
> build time is less than my 8 core, 32GB, 3 SSD system, and your user
> time figure looks a little low, so either you have a crazily optimised
> machine or not everything is being built.
--
Magnus Bäck
ba...@google.com
ivcarlos
I using a vm with Ubunto 12.04 with 4 cores and 6GB ram.
My real machine is a AMD FX 8120 with 12GB ram.
I use separated disks to source and build (with CCache), all virtual
disks.
Total time spent for a full build 4.0.4: 19 mins
Ivan
othy74
> Do you have Ccache enabled? The reason I'm asking is because your build time is less than my 8 core, 32GB, 3 SSD system, and your user time figure looks a little low, so either you have a crazily optimised machine or not everything is being built.
>
> Al.
complete tested maguro (not using Ccache) actually builds in 36 minutes
on my 6-core 1090T:
real 36m4.350s
user 189m22.350s
sys 9m34.520s
I'm still wondering what better hw can do. How long does it take on your
machine? And are 19 minutes at all possible?
Jean-Baptiste Queru
Bridge + SSD, with a stock 12.04.
JBQ
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en
Jean-Baptiste M. "JBQ" Queru
Jean-Baptiste Queru
JBQ
othy74
2 cores = 104 min
4 cores = 55 min
6 cores = 36 min
8 cores = 28 min
16 cores = 15 min
I want this done in 5 minutes. Hey AMD, when is my FX 64100 coming?
Chris Stratton
> Do you have Ccache enabled? The reason I'm asking is because your build time is less than my 8 core, 32GB, 3 SSD system, and your user time figure looks a little low, so either you have a crazily optimised machine or not everything is being built.
Amazon instance.
Wondering if anyone has a build time figure for that method - does it
seem like "renting" the capacity for builds would be a viable
alternative to "owning"?
One downside is that is seems one must choose between 4 virtual cores
and high memory, or 8 cores and only 7 GB. But I also don't get a
good sense what the I/O performance would be like - should one copy
into the instance storage before building, and then push the results
back to EBS before shutting down the instance?
Or would it better to do an "overnight" build on a 64-bit laptop, then
try to do partial rebuilds of only what has actually been changed?
Al Sutton
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)
T: @alsutton
G+: http://goo.gl/ymi9b
The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.
Magnus Bäck
cdesai <chiray...@gmail.com> wrote:
> Partial rebuild can't be used everytime, there are times when you need
> to do a clean build.
>
> A machine with more cores is more better than having more RAM from
> what i've known.
This isn't universally true. Moreover, there is a connection between
cores and RAM. The more cores you have, the more RAM you need to have
to keep the cache warm and avoid swapping.
> My home pc does 20m clean builds, with ccache (100G), and takes about
> 30 without ccache. - that's with a i7-2600K (stock clocks), 12GB RAM,
> and a HDD.
> The same build takes just about the same on a server with the same
> processor, but 32GB RAM.
There are many factors involved here, but it's likely that with the
amount of cores you have an increase beyong 12 GB RAM isn't likely to
help.
--
Magnus Bäck
ba...@google.com
Magnus Bäck
cdesai <chiray...@gmail.com> wrote:
[...]
> Btw, i had a question.
> On that server, we're doing android builds in a chroot, and recently
> we started doing two consecutive builds (using make -j4).
Don't you really mean consecutive builds (i.e. two or more at the same
time)? Otherwise I don't understand what you mean. Running two make
instances at the same time isn't a problem in itself, but since you're
presumably building different source trees (otherwise you'll end up in
trouble for sure) there will be greater competition for cache slots.
Now, if you have 32 GB RAM that's probably not a problem.
> I don't see much difference, but what would be more efficient in your
> opinion ?
Difference how? More efficient than what? One "make -j8" running on its
own or two "make -j4" running consecutively? Given enough RAM I don't
expect a big difference between the two. It depends on what you're
building, too. If you build an SDK, make will spend some time assembling
the Javadoc (which causes your machine to thrash seriously if you don't
have RAM to match). If the Javadoc ends up at the end of the build when
there isn't much else to build (I don't know what the dependency tree
looks like) you might not have enough tasks to keep all CPU threads
busy. If that happens, running two consecutive builds should help keep
the CPUs saturated. Again, build throughput is a function of many
factors. Attempting to saturate the CPUs might just shift the bottleneck
to something else, like the memory or I/O. Everything's connected here.
Note that it typically pays off to build with slightly more jobs than
you have CPU threads, depending on your I/O latency. In other words,
you might be able to drop a minute or two with "make -j9" instead of
"make -j8" given 8 CPU threads.
Now, this is a lot of speculation. I suggest you measure your builds in
your environment. When a build takes an hour or less there's little
point in debating what's faster.
--
Magnus Bäck
ba...@google.com
mark gross
floor ( ( min( RAM in GB/1.5, HW CPU threads) )
You and tweak things all day and get an extra 5% build time but its not worth it to me. BTW if you want to be able to use the system at all while a build is going on I like to use a -j of (Number of CPU Cores + Number of CPU sockets) i.e. 3 for an i5 or 5 for an i7 quad core. Its within 5% build times of the above formula and I can still use the system for email while building.
--mark
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en
--
create interesting things.