Re: Android source mirror update

1,428 views
Skip to first unread message

Magnus Bäck

unread,
Nov 14, 2012, 10:35:22 AM11/14/12
to repo-d...@googlegroups.com
On Wednesday, November 14, 2012 at 09:37 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> I'm searching for this for sometime now... If this is not the right
> group, could you point me the right one?

You've come to the right place.

> I was able to clone the Android source as a mirror on my server,
> and clients can download code from the server fine now.
> We are trying to make a customized version of Android based on
> gingerbread release, so I mirrored with:
>
> repo init -u https://android.googlesource.com/platform/manifest
> --mirror -b android-2.3.6_r1
> repo sync

I doubt the -b option makes any sense here. If you drop it your problem
will go away, and since you're mirroring the whole source tree anyway
(including all branches and tags) it doesn't serve any purpose.

That said, the behavior you're seeing is probably a bug in Repo.

> and clients can now download from mirror as:
>
> repo init -u ssh://<my_server_ip>/<path_to_mirror>/platform/manifest.git
> -b android-2.3.6_r1
>
> Since our team is very small, we won't have a code review tool as
> gerrit,

Having a small team doesn't preclude code reviews. Many of the positive
effects of having code reviews are unrelated to the team size.

> neither our server is a build server (builds are going to be made in
> client machines)... So what we need is a way to sync our local changes
> with our local mirror, so any new client could make a git clone and
> see the latest updates.
>
> Now, my first problem is: I can't clone a git from my mirror with git
> clone...
>
> ~$ git clone
> ssh://<my_server_ip>/<path_to_mirror>/platform/packages/apps/Settings
> Cloning into 'Settings'...
> remote: Counting objects: 61117, done.
> remote: Compressing objects: 100% (18379/18379), done.
> remote: Total 61117 (delta 32718), reused 60987 (delta 32599)
> Receiving objects: 100% (61117/61117), 77.95 MiB | 24.63 MiB/s, done.
> Resolving deltas: 100% (32718/32718), done.
> error: Trying to write non-commit object
> 4c568da55f74d1aee61a2952477b0443aeea6696 to branch HEAD
> fatal: Cannot update the ref 'HEAD'.

This SHA-1 is the android-2.3.6_r1 tag, and Git wants HEAD to point
to branches. As indicated earlier, just drop the -b option when you
initialize the mirror.

> My second problem is that I don't know the right way to push a change
> to my local mirror, do I have to use --mirror flag?

Pushing changes with the setup you've described isn't any different from
pushing any type of changes with Git.

> Also, how do I make a branch based on gingerbread on my mirror, so
> clients can repo init, pull/push as:
>
> repo init -u ssh://<my_server_ip>/<path_to_mirror>/platform/manifest.git -b
> MY_BRANCH

You create Git branches by pushing a commit (or a ref to a commit) to
the desired name of the branch.

git push <push-url> <some-commit-reference>:refs/heads/<branch-name>

Again, this isn't specific to Repo. In your case it can look like this:

git push ssh://<my_server_ip>/<path_to_mirror>/platform/manifest.git \
android-2.3.6_r1:refs/heads/my-gingerbread-development-branch

(I don't remember if Git translates the tag name to a commit in
this case. You should probably use android-2.3.6_r1^{} instead of
android-2.3.6_r1. It might not be necessary but won't hurt.)

This needs to be done for every git. The "repo forall" tool can help you
with that. This assumes that there actually is an android-2.3.6_r1 tag
in each git. I'm guessing that's the case, but it's safer to use what's
actually listed in the manifest. The "repo forall" command defines a
number of environment variables that you can use in the commands that
are executed in each git. In this case you'd want to use $REPO_LREV.

repo forall -c 'git push \
ssh://<my_server_ip>/<path_to_mirror>/platform/manifest.git \
$REPO_LREV:refs/heads/my-gingerbread-development-branch'

I have a vague recollection that the "repo forall" command includes the
manifest git when it iterates over the gits, but othewise you need to
push the branch in the manifest git too. You also need to make a commit
in the manifest branch to adjust the default revision in default.xml.
Otherwise people will continue to sync the original Gingerbread code
even though they initialize their workspace with your branch since it's
the manifest that defines which version is synced in each git.

I have written about this topic a few times before. You might want to
search the list archives.

--
Magnus Bäck
ba...@google.com
Message has been deleted
Message has been deleted
Message has been deleted

Magnus Bäck

unread,
Nov 19, 2012, 2:37:20 PM11/19/12
to repo-d...@googlegroups.com
On Monday, November 19, 2012 at 12:45 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> I tried syncing dropping the -b option (which I used after reading
> http://www.omappedia.com/wiki/Android_Miscellaneous) and the tree
> synced alright. But...

That wiki page has barely been touched in two years and doesn't cover
the relative fetch URLs that were introduced about a year (and which
makes it significantly easier to set up a local mirror). I suggest
you ignore that page.

> From a client I issued repo init -u
> ssh://<server>/platform/manifest.git -b android-2.3.6_r1
>
> The init was successful, but after issuing repo sync I got a bunch of
> errors due to non existent gits, which did not happen when I mirrored
> with the -b option.

I doubt these new errors were caused by you dropping the -b option.

> The errors:
> $ repo sync
> fatal: '/pub/gittree/android/mirror/platform/dalvik' does not appear
> to be a git repository

What's the path part of the manifest URL, just /platform/manifest
or /pub/gittree/android/mirror/platform/manifest? Your command above
indicates the former, but then it's hard to understand why Repo asks
for the latter. Where is your platform/dalvik git stored, i.e. what
URL should Repo use with your current configuration?

[...]

> The .repo/manifests/default.xml head looks like:
> <remote name="aosp"
> fetch=".." /> <--- What ".." means?
> <default revision="refs/tags/android-2.3.6_r1"
> remote="aosp"
> sync-j="4" />
>
> I didn't change the fetch attribute yet because when syncing I see
> repo accessing my server, so didn't find it necessary, is it still?

The '..' means that the fetch URL is relative to the manifest URL. If
you have the gits arranged on the server in the same manner as in the
official servers (i.e. manifest in <some-prefix>/platform/manifest and
the other gits in <some-prefix>/platform/whatever you should be okay.

--
Magnus Bäck
ba...@google.com

Magnus Bäck

unread,
Nov 19, 2012, 2:46:19 PM11/19/12
to repo-d...@googlegroups.com
On Monday, November 19, 2012 at 14:00 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> :( I'm making something very stupid here... You see, my server doesn't
> have a connection to the outside world. So in my brilliant head I
> thought that it wouldn't be that big deal if I just scp the whole tree
> to my server, but... heheh ... links get lost with this operation...
> :/

Using a tool like 'tar' is a good idea if you want to copy a directory
tree and preserve as much as possible (and generally be able to tweak
how things like permissions and links are treated).

> Any advice on how to get the source into the server without accessing
> the internet?

While it should be fine to copy the bare gits straight to the server
(except you won't get a refs/meta/config ref in your gits, but I suppose
Gerrit creates it automatically as needed nowadays), I'd just sync
locally and use Git to push to the server (where you've pre-created the
gits). I'd expect it to be slower than a straight copying operation but
you'll also have to spend less time worrying about links, permissions
and such.

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 19, 2012, 2:47:19 PM11/19/12
to repo-d...@googlegroups.com
Hi Magnus, I've deleted my posts because the cause of the problems I related were introduced by me (because of an erroneous assumption). The repo path is what it is said in the output (I truncated the server path for convenience).

I did something very stupid and completely overlooked it... I did:
repo sync

and then i scped the directory containing all the source tree to my server (which doesn't have an internet connection).

When I tried initializing the repo in a connected machine, it switched branches ok. So I'll try taring the source and then scping the tar to my server and see the results...

Sorry for the confusion, and many thanks.

Magnus Bäck

unread,
Nov 19, 2012, 2:57:28 PM11/19/12
to repo-d...@googlegroups.com
On Monday, November 19, 2012 at 14:47 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> Hi Magnus, I've deleted my posts because the cause of the problems I
> related were introduced by me (because of an erroneous assumption).
> The repo path is what it is said in the output (I truncated the server
> path for convenience).
>
> I did something very stupid and completely overlooked it... I did:
>
> repo init -u https://android.googlesource.com/platform/manifest --mirror
> repo sync
>
> and then i scped the directory containing all the source tree to my
> server (which doesn't have an internet connection).
>
> When I tried initializing the repo in a connected machine, it switched
> branches ok. So I'll try taring the source and then scping the tar to
> my server and see the results...

To avoid creating a rather large tar file and copying it with scp, you
can have tar write its output to a pipe that you connect to another tar
process across an SSH connection:

tar zcf - somepath | ssh hostname "cd somepath && tar zxf -"

(I'd still push the gits via "git push" though.)

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 19, 2012, 3:04:15 PM11/19/12
to repo-d...@googlegroups.com
Hum... I don't get it, how am I supposed to push the gits into my server. The gits are in my machine and the server doesn't have any git to push to. I'm really lost on what you've said...

Magnus Bäck

unread,
Nov 19, 2012, 3:13:27 PM11/19/12
to repo-d...@googlegroups.com
On Monday, November 19, 2012 at 15:04 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> Hum... I don't get it, how am I supposed to push the gits into my
> server. The gits are in my machine and the server doesn't have any
> git to push to. I'm really lost on what you've said...

You create empty gits on the server (e.g. via Gerrit's create-project
SSH command), then push the contents of your locally cloned gits to your
empty gits on the server.

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 19, 2012, 3:17:15 PM11/19/12
to repo-d...@googlegroups.com
:/ We aren't using Gerrit.

Thanks for the support mate!


2012/11/19 Magnus Bäck <ba...@google.com>



--
Atenciosamente,
                          Hugo Frederico Soares
                          (81) 9647 4509

Magnus Bäck

unread,
Nov 19, 2012, 3:33:32 PM11/19/12
to repo-d...@googlegroups.com
On Monday, November 19, 2012 at 15:17 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> :/ We aren't using Gerrit.

That doesn't preclude this method; just fall back on the usual
"git init --bare". The create-project command was just an example.

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 19, 2012, 4:15:25 PM11/19/12
to repo-d...@googlegroups.com
Hmm... I see! I can check the manifest for the project paths and with that initialize each git!
I'll try that! Thanks mate!


2012/11/19 Magnus Bäck <ba...@google.com>
On Monday, November 19, 2012 at 15:17 EST,

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 21, 2012, 2:23:09 PM11/21/12
to repo-d...@googlegroups.com
I'm still having problems while trying to clone a git from the mirror, both when mirror is on same machine and in a remote machine.

Locally:
repo init -u https://android.googlesource.com/platform/manifest -b android-2.3.6_r1
repo sync

In another directory I do:
git clone ~/android/platform/packages/app/Stk
Cloning into 'Stk'...
done.
error: Trying to write non-commit object 752c0b344eb0225c963374b1deb8f6afcfe30b78 to branch HEAD
fatal: Cannot update the ref 'HEAD'.

Any ideas?

Hugo Soares

unread,
Nov 21, 2012, 3:06:58 PM11/21/12
to repo-d...@googlegroups.com
hmmm Found the reason for that...
Did
git symbolic-ref HEAD
on my mirror and it was 
refs/tags/android-2.3.6_r1
Since the tag isn't under refs/heads/* it was complaining. I've pointed HEAD to a valid head and then I checked out the correct tag and it worked.

Thanks!


2012/11/21 Hugo Soares <hugofrede...@gmail.com>

--

Magnus Bäck

unread,
Nov 21, 2012, 3:14:29 PM11/21/12
to repo-d...@googlegroups.com
On Wednesday, November 21, 2012 at 14:23 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> I'm still having problems while trying to clone a git from the mirror,
> both when mirror is on same machine and in a remote machine.
>
> Locally:
> repo init -u https://android.googlesource.com/platform/manifest --mirror
> repo sync
> repo init -u https://android.googlesource.com/platform/manifest
> -b android-2.3.6_r1
> repo sync

So you're running the second Repo command in the local Repo mirror
workspace?

> In another directory I do:
> git clone ~/android/platform/packages/app/Stk
> Cloning into 'Stk'...
> done.
> error: Trying to write non-commit object
> 752c0b344eb0225c963374b1deb8f6afcfe30b78 to branch HEAD
> fatal: Cannot update the ref 'HEAD'.

This is the same error that you got before. The answer then was to
drop the -b option to "repo init" when setting up the mirror, and
this still is the correct answer. You only need a single "repo init"
(without -b) and a single "repo sync". Then you're done.

Using -b when setting up a mirror doesn't make sense because a mirror
by definition contains all branches. I don't even know what the -b
option would mean in this case.

--
Magnus Bäck
ba...@google.com

Hugo Soares

unread,
Nov 21, 2012, 4:06:43 PM11/21/12
to repo-d...@googlegroups.com
From repo init --help
$ repo init --help
    -b REVISION, --manifest-branch=REVISION
                        manifest branch or revision

So even if it does nothing, it shouldn't hurt, right? And as I said before, I was able to clone after setting git HEAD to a valid refs/head in which case I set to gingerbread-mr4-release

Thanks a lot!


2012/11/21 Magnus Bäck <ba...@google.com>

Magnus Bäck

unread,
Nov 21, 2012, 4:25:38 PM11/21/12
to repo-d...@googlegroups.com
On Wednesday, November 21, 2012 at 16:06 EST,
Hugo Soares <hugofrede...@gmail.com> wrote:

> From repo init --help
> $ repo init --help
> -b REVISION, --manifest-branch=REVISION
> manifest branch or revision
>
> So even if it does nothing, it shouldn't hurt, right?

It shouldn't hurt, but it apparently does.

> And as I said before, I was able to clone after setting git HEAD to
> a valid refs/head in which case I set to gingerbread-mr4-release

Yes, and if you just drop the -b option the HEAD symlink will be correct
from the get-go and you won't have to do a thing.

--
Magnus Bäck
ba...@google.com

Jean-Baptiste Queru

unread,
Nov 26, 2012, 1:43:50 PM11/26/12
to repo-d...@googlegroups.com
FWIW (and this is specific to Android and I maintain it manually),
it's possible to mirror the entire set of all android repositories
ever with:

repo init -u https://android.googlesource.com/mirror/android --mirror

From that mirror, you can then repo init as far back as you want
without having to hit the network.

JBQ
Jean-Baptiste M. "JBQ" Queru
Technical Lead, Android Open Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

Jean-Baptiste Queru

unread,
Nov 27, 2012, 10:52:47 AM11/27/12
to cdesai, repo-d...@googlegroups.com
Yes. Typo. Gah. Sorry.

JBQ

On Tue, Nov 27, 2012 at 6:47 AM, cdesai <chiray...@gmail.com> wrote:
> Don't you mean "https://android.googlesource.com/mirror/manifest"
> mirror/manifest instead of mirror/android
Reply all
Reply to author
Forward
0 new messages