Re: Using repo to synchronize public and private servers.

[Jean-Baptiste Queru]

Great questions, thanks for asking :)

First, a few notes:
-when compared to the number of modules in Android, very little of
Android is actually (L)GPL code.
-the GPL doesn't require you to contribute your changes back to the
Android Open-Source Project (though that's appreciated), nor does it
require the maintainers of the Android Open-Source Project to accept
your contributions.

Also, you'll want to note that Google uses a single server internally
(actually a set of servers with hot and warm spares, but semantically
they behave like a single server).

Anyway, I guess that my work in that domain covers primarily 3 areas,
and that 2 of those areas will be relevant for you.

The first area is the keep manifests in sync.
The second area is to import code from the Android Open-Source Project
into Google's private server.
The third area is to export code from Google's private server to the
Android Open-Source Project.

(1) keeping the manifests in sync.

For each logical branch, there are actually 4 branches that come into play:
-the "live" branch on the open-source server.
-the "live" branch on the internal server.
-a mirror branch (on the internal server) that tracks the state of the
open-source server.
-a staging branch (on the internal server) that it used to prepare new
code drops.

There's a manifest for each of those, and I need to make sure that
those manifests stay in sync.

With a few simplifications, the way I do it is:
-set up a client that matches each manifest (one-time setup)
-run a script that syncs all the relevant clients, as a series of (cd
open/refcupcake; repo sync) commands.
-run various scripts that diff the various manifests, as series of
commands like diff open/refcupcake/.repo/manifests/default.xml
open/refdonut/.repo/manifests/default.xml

(2) importing from the Android Open-Source Project:

I have a branch on the internal server that matches each active
open-source branch, and I have a client for each of those branches.

After syncing the client, here's what I do (showing the master branch
here) (with a few simplifications as the real scripts have quite some
logging):
#fetch the state of the external server
repo forall -p -c 'git fetch
git://android.git.kernel.org/$REPO_PROJECT.git refs/heads/master'
#print the new changes, for sanity checking
repo forall -p -c 'git rev-list goog/readonly-korg-master..FETCH_HEAD'
#push to the internal server
repo forall -p -c 'git push ssh://server:29418/$REPO_PROJECT.git
FETCH_HEAD:refs/heads/readonly-korg-master'

Once that's done, I merge it into the "live" internal tree (in a
client that's synced to that tree):
#list the changes being added
repo forall -p -c 'git log goog/master..goog/readonly-korg-master'
#do the merge
repo forall -p -c 'git merge --no-commit goog/readonly-korg-master ;
git commit -m "merge from open-source master"'
#manually resolve conflicts here as necessary
#push to the internal server
repo forall -p -c 'git push ssh://android-git:29418/$REPO_PROJECT.git
HEAD:refs/heads/master'

Notice that the merge inherently has a race condition: if one of the
projects changes while I do the merge, the push will fail for not
being a fast-forward.

(3) exporting to the Android Open-Source Project

This is quite specific to the way Google works internally (and only
applies to the specific relationship between Google and the Android
Open-Source Project). I'll just outline it, but it might be relevant
if you decide to maintain on your own a public-facing version of your
internal tree, the way Google does it with the Android Open-Source
Project.

-the first step is to update an internal "staging" branch to match the
known state that we're trying to push. Our build server records the
output of "repo manifest -r -o -" for each build.
The script looks something like this (notice the very advanced xml parser!):
repo forall -p -c 'git push ssh://android-git:29418/$REPO_PROJECT.git
$(grep \"$REPO_PROJECT\" manifest.xml | cut -d \" -f
6):refs/heads/stage-korg-master'

-once that's done, I simply push it to the external server:
repo forall -p -c 'git push
ssh://review.source.android.com:29418/$REPO_PROJECT.git
goog/stage-korg-master:refs/heads/master'

Of course, that third stage won't apply to you. In your case the
changes will have to be pushed for review by the Android maintainers.
A note about that: reviewing a long string of contributions that gets
submitted in a single pass is impractical. In such situations, the
maintainers will most likely focus a lot more on other people's
contributions (which come in as a regular stream, or at least as a
trickle) to try to handle them in near-real-time, and will let your
contributions slip to the bottom of the review pile. You'll have a
much better success rate if you send the contributions one by one (or
at least in small numbers) and keep them as independent as possible -
that last part might influence the way you work internally, and I
guess that you need to anticipate that you'll need to dedicate some
resources specifically to that effort (just the way my job includes
managing the flow of incoming contributions).

I hope that helps,

JBQ

On Wed, Sep 16, 2009 at 7:09 AM, Anthony Russello <arus...@gmail.com> wrote:
> Hi Jean-Baptiste,
>
> I have been doing some research on using git for both internal (private) and
> external (public) work. I recently read the thread here:
>
>
> http://groups.google.com/group/repo-discuss/browse_thread/thread/8db926b8ca055915/78ad4886ff2890cf?#78ad4886ff2890cf
>
> In which you briefly described how Google handles it's own private and
> public git repositories.
>
> The company I work for is interested in doing the same. Of course, as per
> the GPL, our changes will need to go back into the public Android tree, but
> we would like to stabilize our work offline before submitting it to avoid
> any breakage in the public tree.
>
> What I was wondering is, could I please trouble you for more information on
> exactly how to manage this. It seems you have done a great deal of work in
> this field, and I would like to be able to leverage some of your experience.
>
> I was hoping to have something similar to the following:
>
> Internal Server 1: Internal git server for our development
> Internal Server 2: Bridge server to handle merging of public and private
> branches
>
> From what I understood from the thread above, this is roughly how you are
> handling the code now. We would very much like to do the same.
>
> What I'm missing, I guess, is experience in git sufficient enough to do
> this. I'm unsure how to begin. I was hoping you might be able to give me
> some pointers to get started, so that I can at least initially branch the
> public android repository from Master into our own git server, while still
> retaining the ability to merge changes from our own git server into the
> public repository once we stabilize our code.
>
> Any pointers or advice would be greatly appreciated.
>
> Sincerely,
> Anthony Russello
>
>
>
>
>
> >
>



--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.

Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.

[Anthony Russello]

Hi,

Thank you very much for the help.

I have one more question, how do I create the first copy of the
repository set?

The information below seems that it would work if you have an initial
copy of the repository on your local server.

I've tried

repo forall 'git push git://<ip>/android/$REPO_PROJECT.git'

but that didn't work to create the repositories, even when I already
create empty git repositories on the local server.

Any help would be appreciated,
Anthony

> >http://groups.google.com/group/repo-discuss/browse_thread/thread/8db9...

[Jean-Baptiste Queru]

I'll let other people comment more specifically on that aspect (as I
didn't do that setup myself for either of the servers that I work on,
and creating new projects for the Android Open-Source Project isn't
quite your "regular" process because we also need to handle the
specific replication setup there).

JBQ

[Brad Larson]

I'd suggest looking into repo init --mirror

Brad

[Anthony Russello]

Hi,

For anyone looking to do the same, here is what I actually did:

1. mkdir android
2. repo init -u git://android.git.kernel.org/platform/manifest.git --
mirror
3. cd android
4. for i in `find ./ -name "*.git"`; do git clone --bare -l ${i} /
var/git/android/${i}; done

The result is that I have an independent set of repositories
internally.

So, when submitting patches back to mainline, or updating from
mainline, I can just use git to push/pull between the two repositories
as Jean-Baptiste suggested.

Strictly using repo init --mirror meant that if I tried to sync, it
would overwrite my local repository changes. Instead, we will be
scripting merging from mainline into our private repo, and using git
cherry-pick to submit patches back to mainline.

I repeated this process for the android-x86 as well with great
success.

Thanks,
Anthony

[Anthony Russello]

Sorry, the steps below were in the wrong order:

1. mkdir android
2. cd android
3. repo init -u git://android.git.kernel.org/platform/manifest.git
--
mirror

4. for i in `find ./ -name "*.git"`; do git clone --bare -l ${i} /
var/git/android/${i}; done

My apologies for the spam.

Thanks,
Anthony

> ...
>
> read more »