Experiment: smaller AOSP downloads

[Jean-Baptiste Queru]

AOSP is big, and downloading and storing all of it consumes a fair
amount of bandwidth and space.

We've been investigating ways to make it smaller. One visible part of
that work is that the master tree is "only" 6.5GB, while Jelly Bean is
about 8.5GB.

We're aiming to go further. Today, I'm deploying an experiment that
can help shrink things.

Repo recently added a feature called "manifest groups" that allows to
restrict the list of projects being downloaded.

I've set up the manifest for the AOSP master branch to include a few
groups. Here are the groups I added:

"device": files that are specific to flagship devices but aren't
necessary to build a generic platform.
"darwin": files that are only necessary when building on a darwin host
(i.e. MacOS).
"linux": files that are only necessary when building on a linux host.
"arm": files that are only necessary when building for ARM targets.
"mips": files that are only necessary when building for MIPS targets.
"x86": files that are only necessary when building for IA targets.

Here's an example, which allows to build the ARM emulator on a linux
host. In this case, the download is 3.6GB, which is a significant
improvement over a full tree:

repo init -u [URI] -g default,-darwin,-device,-x86,-mips

The syntax is hopefully self-explanatory. Note that this is only
currently supported in the master branch.

I'm looking at the possible evolutions:

-marking the projects for individual device families (i.e. crespo,
tuna, stingray, panda), so that it's possible to download the files
for one specific device without getting them all.
-marking the projects that are necessary to work on the developer tools.
-marking the projects that are necessary to work on CTS.

There's also a possibility that repo will eventually know to
automatically ignore host-specific projects.

Longer-term, we're investigating the possibility of not downloading
the entire source history for projects where it's both large and
unnecessary.

I'm open to feedback and suggestions. Please let me know whether
that's useful, whether that works, which of the future directions
would be useful... and which future features I've forgotten that'd
make life easier.

Thanks,
JBQ

--
Jean-Baptiste M. "JBQ" Queru
Technical Lead, Android Open Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Al Sutton]

Nice work :).

My only concern would be that I mirror the AOSP repository to a Mac server and then pull to my build machines which are Linux and Mac based, so would the fact I'm mirroring to a Mac server stop me getting the Linux support files?

Al.
--
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.

> --
> You received this message because you are subscribed to the "Android Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

[Jean-Baptiste Queru]

When repo eventually moves to automatically selecting host-specific
projects, it'll have options in repo init to override that behavior.

Tou can already see the syntax in "repo help init" but it hasn't been
hooked up yet. That'd be "repo init -p all".

In theory, we could make "--mirror" imply "-p all" but I'm not sure
that I like such interplay between command-line options.

JBQ

[Jean-Baptiste Queru]

-Do you have a list of external/ projects that are both large and
unused? I'll try to get such projects removed from Android entirely.

-I'm not entirely convinced about broadly defaulting to "-c". It
doesn't help for the initial sync since everything always gets
downloaded as part of the initial clone. It doesn't help for the
master branch since that one always contains the content of all other
branches. It doesn't help much for the most recent named branch since
it also contains almost everything. It doesn't help for tagged
releases since those never change and the only reason to sync those it
explicitly to update the other branches. The one case where it does
help is for incremental syncs of previous named branches (e.g.
ics-mr1) across releases of subsequent versions (e.g. Jelly Bean),
I'll look into that.

JBQ

On Thu, Aug 2, 2012 at 12:30 AM, cdesai <chiray...@gmail.com> wrote:
> Nice work sir. :)
>
> An optional group for projects under external/ would be nice, since a lot of
> them aren't compiled.
>
> One more thing would be making repo sync only the current branch by default,
> as that's what most people use. (I do know about repo sync -c, but having it
> as default would be nice IMO :))

>
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

[Conley Owens]

You typically don't want to use either flag. There is no way to list
all the groups. If you want to see them after an init, you can look
at .repo/manifest.xml for the `groups` attribute.

~cco3

On Thu, Jan 17, 2013 at 9:57 PM, Gulshan Singh <gsing...@gmail.com> wrote:
> When I type repo help init I see this:
> -g GROUP, --groups=GROUP
> restrict manifest projects to ones with a specified
> group
> -p PLATFORM, --platform=PLATFORM
> restrict manifest projects to ones with a specified
> platform group [auto|all|none|linux|darwin|...]
>
> If I'm building on linux should I be doing -g linux or -p linux, or both?
> Also is there a way to list all of the groups?