"repo sync" sizes (official documentation inaccuracies + changing from platform/manifest to mirror/manifest)

[Ben]

First time post (hoping this is the correct place). I'm relatively new to working with android source (long time working with apps) and have spent a lot of time gathering a sound understanding of the "repo" tool and what exactly is going on - rather than just following the commands and hoping. 

I am trying to establish a local mirror, so I have run the commands (I used "platform/manifest" instead of "mirror/manifest" mistakenly):

repo init -u https://android.googlesource.com/platform/manfiest --mirror 

repo sync

Completing that took 37GB of downloads. This is in start contrast to the official documentation which says "close to 13GB of network bandwidth". Based on my digging, it seems that this figure has been inaccurate ever since "device/*" projects started being added (and is now grossly inaccurate thanks to a 13GB hammerhead-kernel package alone). I realise bandwidth is probably irrelevant to most parties working on android source, but to those reading a "how to download the source" guide (i.e. just starting) I'd imagine there would be quite a few where it is important.

1. Is there any way to see (or derive) an accurate size of a "repo sync" command? Or the size of all projects in the repo default "manifest.xml"? Even the size of individual projects would be sufficient (i.e. from https://android.googlesource.com/). I know you can get the sizes in a roundabout way through github APIs but that doesn't apply to all of the AOSP git repositories.

Secondly, I'm looking at changing the "platform/manifest" mirror to "mirror/manifest" (having difficulties syncing individual branch with the mirror because packages are missing). After that first experience, I'm obviously trying to verify download usage beforehand. Based on parsing the manifests, I know that the change will mean (provided I correctly reuse what has already been downloaded) downloading 242 extra projects. 25 of these are "device/*" projects, and given 1 of those was 13GB alone, this has me concerned.

2. Does anyone know offhand (or how I can find) what I can expect this difference to be in terms of file size? Or does anyone at least have the size of a full "mirror/manifest" sync?

3. Given both manifests seem to put projects in the same place, will simply rerunning "repo init" with the "mirror/manifest" reuse the already downloaded projects (I don't want to "--reference" to the "platform/manifest" because I intend on only having one copy - i.e. not a "mirror/manifest" and "platform/manifest")? If not, how can I reuse the 37GB I've already downloaded?

Thanks in advance!

[David Pursehouse]

On 12/13/2014 10:25 AM, Ben wrote:
> First time post (hoping this is the correct place). I'm relatively new
> to working with android source (long time working with apps) and have
> spent a lot of time gathering a sound understanding of the "repo" tool
> and what exactly is going on - rather than just following the commands
> and hoping.
>
> I am trying to establish a local mirror, so I have run the commands (I
> used "platform/manifest" instead of "mirror/manifest" mistakenly):

> /repo init -u https://android.googlesource.com/platform/manfiest --mirror /
> /repo sync/
> /
> /

> Completing that took 37GB of downloads. This is in start contrast to the
> official documentation

> <https://source.android.com/source/downloading.html> which says "close

> to 13GB of network bandwidth". Based on my digging, it seems that this
> figure has been inaccurate ever since "device/*" projects started being
> added (and is now grossly inaccurate thanks to a 13GB hammerhead-kernel
> package alone). I realise bandwidth is probably irrelevant to most
> parties working on android source, but to those reading a "how to
> download the source" guide (i.e. just starting) I'd imagine there would
> be quite a few where it is important.
>

The bit about network bandwidth has been removed [1] and the "building"
page [2] now states that around 50GB is needed for a checkout. (This
refers to disk space; actual network bandwidth required to download it
might differ).


[1] https://android-review.googlesource.com/#/c/119570/2
[2] https://source.android.com/source/building.html


> *1. *Is there any way to see (or derive) an accurate size of a "repo

> sync" command? Or the size of all projects in the repo default
> "manifest.xml"? Even the size of individual projects would be sufficient
> (i.e. from https://android.googlesource.com/). I know you can get the
> sizes in a roundabout way through github APIs but that doesn't apply to
> all of the AOSP git repositories.
>
>
> Secondly, I'm looking at changing the "platform/manifest" mirror to
> "mirror/manifest" (having difficulties syncing individual branch with
> the mirror because packages are missing). After that first experience,
> I'm obviously trying to verify download usage beforehand. Based on
> parsing the manifests, I know that the change will mean (provided I
> correctly reuse what has already been downloaded) downloading 242 extra
> projects. 25 of these are "device/*" projects, and given 1 of those was
> 13GB alone, this has me concerned.
>

> *2.* Does anyone know offhand (or how I can find) what I can expect this

> difference to be in terms of file size? Or does anyone at least have the
> size of a full "mirror/manifest" sync?
>

The project statistics REST API [3] will give you some information about
the repository size. However I'm not sure if it's possible to run 'repo
forall' on an un-synched workspace...


[3]
https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#get-repository-statistics

> *3. *Given both manifests seem to put projects in the same place, will

> simply rerunning "repo init" with the "mirror/manifest" reuse the
> already downloaded projects (I don't want to "--reference" to the
> "platform/manifest" because I intend on only having one copy - i.e. not
> a "mirror/manifest" and "platform/manifest")? If not, how can I reuse
> the 37GB I've already downloaded?
>
> Thanks in advance!
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Shawn Pearce]

On Wed, Dec 17, 2014 at 12:53 AM, David Pursehouse <david.pu...@sonymobile.com> wrote:

*2.* Does anyone know offhand (or how I can find) what I can expect this
difference to be in terms of file size? Or does anyone at least have the
size of a full "mirror/manifest" sync?

The project statistics REST API [3] will give you some information about the repository size.  However I'm not sure if it's possible to run 'repo forall' on an un-synched workspace...


[3] https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#get-repository-statistics

This probably does not work on android-review.googlesource.com:

  $ curl 'https://android-review.googlesource.com/projects/platform%2fmanifest/statistics.git'

  Capability runGC is required to access this resource

And its not just that we block the public Internet from being able to trigger GCs. Our internal GC implementation is different on these servers, Gerrit can't trigger it even if it was going to allow the public to do so, and the internal implementation doesn't report stats.

So that isn't going to help.

[Les Potter]

... (2 years later) I cannot find anywhere on the internet where someone says what the current size of the AOSP master mirror is.  How much disk space will it require for a "repo sync"?  I've got a machine with a slower connection to the internet (3-5MiB/s) and a clean/scratch repo sync going on right now.  By 'du -s' it has taken up 159+GiB and is still downloading. 

I will post the final size when it's through, but wouldn't it be nice if the CI for AOSP would write the size of the repo somewhere that the public can get to (through maybe a link in the official docs)?  Whatever number I end up with after "repo sync" finishes will be wrong not too long afterwards.

[Les Potter]

repo sync of aosp mirror finished.

$ du -s -BM /usr/local/aosp/mirror

165285M /usr/local/aosp/mirror

On Wednesday, December 17, 2014 at 11:11:28 AM UTC-5, Shawn Pearce wrote:

[Mickey Killianey]

For reference, just made a mirror today:

$ du -hs /Volumes/android/mirror/

267G /Volumes/android/mirror/

[Samantha Priyadarshana]

For reference;

Just cloned a mirror today & it is 349 GB.