Experiment: smaller AOSP downloads

Showing 1-11 of 11 messages
Experiment: smaller AOSP downloads Jean-Baptiste Queru 8/1/12 10:36 AM
AOSP is big, and downloading and storing all of it consumes a fair
amount of bandwidth and space.

We've been investigating ways to make it smaller. One visible part of
that work is that the master tree is "only" 6.5GB, while Jelly Bean is
about 8.5GB.

We're aiming to go further. Today, I'm deploying an experiment that
can help shrink things.

Repo recently added a feature called "manifest groups" that allows to
restrict the list of projects being downloaded.

I've set up the manifest for the AOSP master branch to include a few
groups. Here are the groups I added:

"device": files that are specific to flagship devices but aren't
necessary to build a generic platform.
"darwin": files that are only necessary when building on a darwin host
(i.e. MacOS).
"linux": files that are only necessary when building on a linux host.
"arm": files that are only necessary when building for ARM targets.
"mips": files that are only necessary when building for MIPS targets.
"x86": files that are only necessary when building for IA targets.

Here's an example, which allows to build the ARM emulator on a linux
host. In this case, the download is 3.6GB, which is a significant
improvement over a full tree:

repo init -u [URI] -g default,-darwin,-device,-x86,-mips

The syntax is hopefully self-explanatory. Note that this is only
currently supported in the master branch.

I'm looking at the possible evolutions:

-marking the projects for individual device families (i.e. crespo,
tuna, stingray, panda), so that it's possible to download the files
for one specific device without getting them all.
-marking the projects that are necessary to work on the developer tools.
-marking the projects that are necessary to work on CTS.

There's also a possibility that repo will eventually know to
automatically ignore host-specific projects.

Longer-term, we're investigating the possibility of not downloading
the entire source history for projects where it's both large and
unnecessary.

I'm open to feedback and suggestions. Please let me know whether
that's useful, whether that works, which of the future directions
would be useful... and which future features I've forgotten that'd
make life easier.

Thanks,
JBQ

--
Jean-Baptiste M. "JBQ" Queru
Technical Lead, Android Open Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.
Re: [android-building] Experiment: smaller AOSP downloads Al Sutton 8/1/12 10:49 AM
Nice work :).

My only concern would be that I mirror the AOSP repository to a Mac server and then pull to my build machines which are Linux and Mac based, so would the fact I'm mirroring to a Mac server stop me getting the Linux support files?

Al.
--
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries.
> --
> You received this message because you are subscribed to the "Android Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com
> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

Re: [android-building] Experiment: smaller AOSP downloads Jean-Baptiste Queru 8/1/12 10:54 AM
When repo eventually moves to automatically selecting host-specific
projects, it'll have options in repo init to override that behavior.

Tou can already see the syntax in "repo help init" but it hasn't been
hooked up yet. That'd be "repo init -p all".

In theory, we could make "--mirror" imply "-p all" but I'm not sure
that I like such interplay between command-line options.

JBQ
Re: Experiment: smaller AOSP downloads cdesai 8/2/12 12:30 AM
Nice work sir. :)

An optional group for projects under external/ would be nice, since a lot of them aren't compiled.

One more thing would be making repo sync only the current branch by default, as that's what most people use. (I do know about repo sync -c, but having it as default would be nice IMO :))
Re: [android-building] Re: Experiment: smaller AOSP downloads Jean-Baptiste Queru 8/2/12 11:01 AM
-Do you have a list of external/ projects that are both large and
unused? I'll try to get such projects removed from Android entirely.

-I'm not entirely convinced about broadly defaulting to "-c". It
doesn't help for the initial sync since everything always gets
downloaded as part of the initial clone. It doesn't help for the
master branch since that one always contains the content of all other
branches. It doesn't help much for the most recent named branch since
it also contains almost everything. It doesn't help for tagged
releases since those never change and the only reason to sync those it
explicitly to update the other branches. The one case where it does
help is for incremental syncs of previous named branches (e.g.
ics-mr1) across releases of subsequent versions (e.g. Jelly Bean),
I'll look into that.

JBQ
Re: [android-building] Re: Experiment: smaller AOSP downloads bhoj 10/4/12 8:28 AM
Hi,

Thanks for this feature. I was trying to capture the revisions as current HEAD on the master branch using repo which seems to be broken currently with latest repo tool. Is this a known issue ?

This is what I tried:

repo manifest -r -o current-revision.xml
Traceback (most recent call last):
  File "/home/vishalbhoj/android/nexus/.repo/repo/main.py", line 384, in <module>
    _Main(sys.argv[1:])
  File "/home/vishalbhoj/android/nexus/.repo/repo/main.py", line 364, in _Main
    result = repo._Run(argv) or 0
  File "/home/vishalbhoj/android/nexus/.repo/repo/main.py", line 136, in _Run
    result = cmd.Execute(copts, cargs)
  File "/home/vishalbhoj/android/nexus/.repo/repo/subcmds/manifest.py", line 73, in Execute
    self._Output(opt)
  File "/home/vishalbhoj/android/nexus/.repo/repo/subcmds/manifest.py", line 63, in _Output
    peg_rev = opt.peg_rev)
  File "/home/vishalbhoj/android/nexus/.repo/repo/manifest_xml.py", line 204, in Save
    p.work_git.rev_parse(HEAD + '^0'))
  File "/home/vishalbhoj/android/nexus/.repo/repo/project.py", line 1958, in runner
    capture_stderr = True)
  File "/home/vishalbhoj/android/nexus/.repo/repo/git_command.py", line 221, in __init__
    raise GitError('%s: %s' % (command[1], e))
error.GitError: rev-parse: [Errno 2] No such file or directory: '/home/vishalbhoj/android/nexus/prebuilts/eclipse-build-deps'
Re: Experiment: smaller AOSP downloads Ajay Katwe 10/10/12 6:38 AM
ya +1 on this one...most of them still hack manifest to cut down the download part we cant sync the upstream changes once we edit the manifest and useful for people having scarce bandwidth
Re: Experiment: smaller AOSP downloads Gulshan Singh 1/17/13 9:57 PM
When I type repo help init I see this:
    -g GROUP, --groups=GROUP
                        restrict manifest projects to ones with a specified
                        group
    -p PLATFORM, --platform=PLATFORM
                        restrict manifest projects to ones with a specified
                        platform group [auto|all|none|linux|darwin|...]

If I'm building on linux should I be doing -g linux or -p linux, or both? Also is there a way to list all of the groups?
Re: [android-building] Re: Experiment: smaller AOSP downloads Conley Owens 1/18/13 8:58 AM
You typically don't want to use either flag.  There is no way to list
all the groups.  If you want to see them after an init, you can look
at .repo/manifest.xml for the `groups` attribute.

~cco3
Re: [android-building] Re: Experiment: smaller AOSP downloads Gulshan Singh 1/19/13 8:48 PM
Thanks. I tried the command JBQ gave (repo init -u [URI] -g default,-darwin,-device,-x86,-mips), but after doing this repo sync doesn't do anything, it just prints a blank line. I've tried it with different group combinations, and the only time repo sync actually starts syncing projects is if I use the group all (-g all), which I believe is the same as not specifying the flag at all.

Any reason why this could happen?
Re: [android-building] Re: Experiment: smaller AOSP downloads Gulshan Singh 1/19/13 8:54 PM
I should clarify the context of "repo sync doesn't do anything". I actually did repo sync and all of the projects were deleted (I believed repo called them obsolete). After that subsequent calls to repo sync just printed the blank line. Doing a regular repo init with no -g flag and then doing a repo sync brought back all the projects. Hope that clears it up.


On Friday, January 18, 2013 11:58:22 AM UTC-5, Conley Owens wrote: