Set up a Git server using whichever Git hosting solution you like.
I think Gerrit is an excellent choice (regardless of whether you're
interested in using the code review feature), but you could just go
with any regular machine that users can access via SSH, or another
hosting solution like Gitolite.
Periodically, sync the code from AOSP and push it to your server. You
should never ever allow your users to directly modify the AOSP branches.
Instead, use different namespaces for upstream and your own development.
For example, push all AOSP branches to aosp/* and use the root namespace
for your own branches (whose origin of course is one of the aosp/*
branches). This should sync a workspace looking at the master branch
of AOSP and push everything to your server (where all the gits must
exist):
repo init --mirror -u git://android.git.kernel.org/platform/manifest
repo sync
repo forall -c 'git push ssh://yourserver.example.com/$REPO_PROJECT \
"refs/heads/*:refs/heads/aosp/*"'
Until you're comfortable with the forall and push commands I would
strongly recommend that you try each command on a single git first
so you won't wreck havoc.
Again, create local branches based off of the AOSP branches for your
developers to work on. The first git to branch would be the manifest
git, and you'll want to adjust the fetch and review URLs listed in the
<remote> tag at the top of the file so that they point to your server.
When you want to update your local branches from AOSP, merge the
manifest itself and then each git. Something like this -- run from a
workspace looking at your code -- should work:
repo forall -c 'git merge origin/aosp/master'
repo forall -c 'git push ssh://yourserver.example.com/$REPO_PROJECT \
HEAD:refs/heads/master'
(This examples assumes that all gits have a master branch that you
should work from, which as JBQ said in the original thread in
android-building isn't true.)
No, this isn't completely trivial. Expect to spend time on figuring
this out and getting it working. This question is not limited to the
pure IT issues surrounding the setup of a server for hosting Git --
you need to figure out how your company should manage your source code,
which AOSP update strategy to use, and so on. If you want to keep your
code up to date with AOSP and reduce the maintenance cost for new
releases you need to think about how you make changes to the platform
(you might not need to change the platform at all). Some types of
changes are more expensive than others, but if you contribute changes
back to AOSP your maintenance cost will decrease over time.
--
Magnus B�ck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson
That's an excellent summary Magnus.
Michael, I would just like to add that there are several of us on this list that have done this and are pretty willing to help out with any questions you might have (especially if you're using Gerrit). Depending on the size of your developer group it could be days, weeks, or months until you have the infrastructure in place, necessary training completed, and are up to full speed with your development process.
Another thing to consider (if you haven't already) that's tangentially related to Gerrit is some type of Continuous Integration solution. Gerrit makes this easier with a pre-setup "Verified" category that you can use on each change to indicate you've tested the change and it works. It's a reasonably common practice to give an automated system permission to set that category as a method to ensure code being merged to the branch works.
Nasser
>
> --
> Magnus Bäck Opinions are my own and do not necessarily
> SW Configuration Manager represent the ones of my employer, etc.
> Sony Ericsson
>
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
--
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
You might find this useful:
ssh -p 29418 review.source.android.com gerrit ls-projects
(You'll only need the projects under device/ and platform/ for all
recent versions of Android).
JBQ
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
--
Jean-Baptiste M. "JBQ" Queru
Software Engineer, Android Open-Source Project, Google.
Questions sent directly to me that have no reason for being private
will likely get ignored or forwarded to a public forum with no further
warning.
> OK, my last question seems silly to me now. The reason it says the git
> repo doesn't exist, is because the repo doesn't actually exist on the
> "master" branch. So, do I need to do a separate clone for every
> branch?
>
> repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
> repo sync
>
> (create a new dir)
> repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
> -b froyo
> repo sync
>
> (create a new dir)
> repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
> -b froyo-plus-aosp
> repo sync
> ...
>
> Having simply a clone of the "master" branch doesn't seem sufficient,
> especially if the projects have been deleted or moved around...
Correct; "repo sync" downloads all branches in all gits, but only gits
currently listed in the manifest you're looking at.
That said, you don't need to initialize completely different workspaces
for each branch. You should also be able to do just this:
repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
repo sync
repo forall -c 'git push ...'
repo init -b froyo
repo sync
repo forall -c 'git push ...'
repo init -b froyo-plus-aosp
repo sync
repo forall -c 'git push ...'
--
Magnus B�ck Opinions are my own and do not necessarily
> I have run into a small snag, hopefully someone can shed some light on
> it; I'm probably missing something. Right now, I'm just experimenting;
> this approach may not be the one I use. As a first step, I have cloned
> the public git repo doing something like this:
>
> repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
>
> From another machine, I run a command like this:
> modified-repo init -u ssh://g...@192.168.2.36/home/git/android/platform/manifest.git
> -b froyo-plus-aosp
>
> The "modified-repo" script has the REPO_URL changed to:
> REPO_URL='ssh://g...@192.168.2.36/home/git/android/tools/repo.git'
You'd be better off just using the --repo-url option of "repo init".
Even better, use insteadOf in your .gitconfig (and have all your users
do this):
[url "ssh://g...@192.168.2.36/home/git/android/"]
insteadOf = git://android.git.kernel.org/
Then all git clone/fetch/pull/etc operations will perform a pattern
replacement and replace the original URL with your local one.
[...]
--
Magnus B�ck Opinions are my own and do not necessarily
> On Friday, April 01, 2011 at 15:42 CEST,
> Michael <mike...@gmail.com> wrote:
>
>> I have run into a small snag, hopefully someone can shed some light on
>> it; I'm probably missing something. Right now, I'm just experimenting;
>> this approach may not be the one I use. As a first step, I have cloned
>> the public git repo doing something like this:
>>
>> repo init --mirror -u git://android.git.kernel.org/platform/manifest.git
>>
>> From another machine, I run a command like this:
>> modified-repo init -u ssh://g...@192.168.2.36/home/git/android/platform/manifest.git
>> -b froyo-plus-aosp
>>
>> The "modified-repo" script has the REPO_URL changed to:
>> REPO_URL='ssh://g...@192.168.2.36/home/git/android/tools/repo.git'
>
> You'd be better off just using the --repo-url option of "repo init".
>
> Even better, use insteadOf in your .gitconfig (and have all your users
> do this):
>
> [url "ssh://g...@192.168.2.36/home/git/android/"]
> insteadOf = git://android.git.kernel.org/
>
> Then all git clone/fetch/pull/etc operations will perform a pattern
> replacement and replace the original URL with your local one.
>
> [...]
I'd stay away from the insteadOf if you're working at all directly with a.g.k.o sources.
Also, you shouldn't need to modify repo at all. We use the stock version of repo and just init to our internal manifest project. If you want to mirror repo for availability reasons (a.g.k.o sometimes has high load and will refuse connections).
For cloning we do what JBQ suggests and set up some cron jobs to sync the output of gerrit ls-projects (though we should teach ls-projects about perm-only projects so we don't mirror those).
Nasser
>
> --
> Magnus Bäck Opinions are my own and do not necessarily
> SW Configuration Manager represent the ones of my employer, etc.
> Sony Ericsson
>
> On Apr 1, 2011, at 8:52 AM, Magnus B�ck wrote:
>
> > You'd be better off just using the --repo-url option of "repo init".
> >
> > Even better, use insteadOf in your .gitconfig (and have all your
> > users do this):
> >
> > [url "ssh://g...@192.168.2.36/home/git/android/"]
> > insteadOf = git://android.git.kernel.org/
> >
> > Then all git clone/fetch/pull/etc operations will perform a pattern
> > replacement and replace the original URL with your local one.
>
> I'd stay away from the insteadOf if you're working at all directly
> with a.g.k.o sources.
Definitely -- there's no easy way to disable insteadOf without editing
$HOME/.gitconfig, so people working directly with a.g.k.o wouldn't want
to do this.
> Also, you shouldn't need to modify repo at all. We use the stock
> version of repo and just init to our internal manifest project. If you
> want to mirror repo for availability reasons (a.g.k.o sometimes has
> high load and will refuse connections).
Right. Many companies have firewalls that block port 9418, and then I
guess it's either --repo-url, insteadOf, or distribute a custom repo
binary.
> For cloning we do what JBQ suggests and set up some cron jobs to sync
> the output of gerrit ls-projects (though we should teach ls-projects
> about perm-only projects so we don't mirror those).
Yeah, that does sound better than syncing manifests and using
"repo forall" for iteration.
--
Magnus B�ck Opinions are my own and do not necessarily
> This is a great thread! Thanks for the help. I've followed most of it,
> but I'm having a little bit of trouble with the separate aosp
> namespace. I'm relatively new to Git, so I apologize for missing what
> is probably an obvious step.
>
> When I try to clone a repo after pushing with the refspec "refs/heads/
> *:refs/heads/aosp/*" I receive the warning "remote HEAD refers to
> nonexistent ref, unable to checkout." I assume this is because the
> HEAD ref points to master which has been moved to aosp/master.
Correct.
> What's the best way to deal with this?
You shouldn't make any changes to any of the upstream branches, so it's
a good thing that Git doesn't check out one of the aosp/* branches for
you. You probably want to create a master branch yourself, based on
aosp/master, and use that as the branch you work on.
If you create such a branch on the server Git will create a local
branch (master) that tracks the upstream branch (origin/master) when
you clone, and you won't get that warning anymore. Of course, upstream
in this case means the master branch on your server, which in turn has
AOSP's master branch as its upstream.
> Right now I'm keeping aosp in the root namespace and using a
> separate area to keep the development branches in, then limiting
> the permissions with Gitolite. That seems to work just fine, but
> I'd like to understand the other way.
You could do things your way too, of course -- technically it doesn't
matter. However, putting the AOSP branches in the root of the namespace
means that you assume you'll never have more than one upstream source
(unless you want to be inconsistent).
If it's a common use case for your developers to clone gits by hand
rather than use Repo it also makes more sense to give them a master
branch they can actually work with rather than one based on a branch
to which they must never ever push.
--
Magnus Bäck Opinions are my own and do not necessarily
> I am sorry to bother you mate. But I can not get it to work. I have
> been trying the whole day without success.
> Is there maybe you could show me a step by step guide on how I can
> create a mirror to our server.
> And then pushing the changes to our gerrit server so that we can have
> a versioning check system on our local mirror?
I thought my first message in this thread pretty much gave a step by
step instruction...? It even included command examples. The more
specific question you ask the more specific answer you'll get. Right
now I don't know where to start.
Make sure you have a fairly good understanding of Git before you start
doing this, otherwise everything will just be a series of weird commands
and you'll be doomed when something goes wrong.
> Also what about updating the local mirror to latest google sources?
> How would that go exactly?
This would be done in the same way as the initial push, except that you
won't necessarily have to download everything each time.
> Yes I changed it to:
> <remote name="korg"
> fetch="git://android.git.kernel.org/"
> review="review.source.android.com" />
>
> <remote name="local"
> fetch="ssh://gerrithost"
> review="mygerritServer.domain.com" />
>
> <default revision="gingerbread"
> remote="korg" />
>
> I also changed the rest of the file to follow my project_path as:
> <project path="bionic" name="MyPath/android_bionic" remote="local" />
Do you really need to rename the projects like that? It will be very
confusing. Otherwise this should work, but I'd switch the default
revision to use your new remote. Otherwise you'll have to remember to
update the manifest every time you merge new changes from upstream
(although you'll have to do that anyway since you're renaming the
projects).
> Is there something I am missing? btw, How do I push: .repo/manifests/
> back to my repo?
From a git perspective there's nothing unusual about the manifest git.
git push ssh://gerrithost/platform/manifest \
local-commit-ish:refs/heads/desired-branchname
(And by commit-ish I mean any reference that can be resolved to a
commit, e.g. a symbolic reference like HEAD, a SHA-1, a branch or
a tag.)
> Now when I do:
> repo forall -c 'git push ssh://gerrithost/$REPO_PROJECT
> HEAD:refs/heads/master'
>
> I get:
> To ssh://gerrithost/android
> ! [rejected] HEAD -> master (non-fast forward)
> error: failed to push some refs to 'ssh://gerrithost/android'
Which git is "android"? Anyway, why is this a non-fast forward? What's
HEAD pointing to, a local commit or one from AOSP? What's the master
branch on the server, your development branch or a reference branch
from AOSP?
> Everything up-to-date
> Everything up-to-date
> Everything up-to-date
> Everything up-to-date
> Everything up-to-date
Okay, looks good.
On Wednesday, April 06, 2011 at 08:56 CEST,
> Yes I changed it to:Do you really need to rename the projects like that? It will be very
> <remote name="korg"
> fetch="git://android.git.kernel.org/"
> review="review.source.android.com" />
>
> <remote name="local"
> fetch="ssh://gerrithost"
> review="mygerritServer.domain.com" />
>
> <default revision="gingerbread"
> remote="korg" />
>
> I also changed the rest of the file to follow my project_path as:
> <project path="bionic" name="MyPath/android_bionic" remote="local" />
confusing. Otherwise this should work, but I'd switch the default
revision to use your new remote. Otherwise you'll have to remember to
update the manifest every time you merge new changes from upstream
(although you'll have to do that anyway since you're renaming the
projects).
> Is there something I am missing? btw, How do I push: .repo/manifests/From a git perspective there's nothing unusual about the manifest git.
> back to my repo?
git push ssh://gerrithost/platform/manifest \
local-commit-ish:refs/heads/desired-branchname
(And by commit-ish I mean any reference that can be resolved to a
commit, e.g. a symbolic reference like HEAD, a SHA-1, a branch or
a tag.)
Which git is "android"? Anyway, why is this a non-fast forward? What's
> Now when I do:
> repo forall -c 'git push ssh://gerrithost/$REPO_PROJECT
> HEAD:refs/heads/master'
>
> I get:
> To ssh://gerrithost/android
> ! [rejected] HEAD -> master (non-fast forward)
> error: failed to push some refs to 'ssh://gerrithost/android'
HEAD pointing to, a local commit or one from AOSP? What's the master
branch on the server, your development branch or a reference branch
from AOSP?
> Everything up-to-dateOkay, looks good.
> Everything up-to-date
> Everything up-to-date
> Everything up-to-date
> Everything up-to-date
--
Magnus Bäck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson
More info at http://groups.google.com/group/repo-discuss?hl=en
> On Wed, Apr 6, 2011 at 9:48 AM, Magnus Bäck
> <magnu...@sonyericsson.com> wrote:
>
> > Do you really need to rename the projects like that? It will be
> > very confusing. Otherwise this should work, but I'd switch the
> > default revision to use your new remote. Otherwise you'll have
> > to remember to update the manifest every time you merge new
> > changes from upstream (although you'll have to do that anyway
> > since you're renaming the projects).
>
> The idea is to have a mixture of git repo's being updated. Most
> of the git's will be AOSP. And a few should only be work from me.
Ah, okay. If all AOSP gits use the korg remote this is fine. However,
if your goal is to have your users sync everything from your local
server and not contact android.git.kernel.org at all it's not what
you want to do.
> I am still a little bit lost on how to create a full copy of
> AOSP mainstream and have it correctly executing: repo init -u
> ssh://gerrithost/android.git -b gingerbread. So that it syncs
> from my local server and not from the AOSP remote.
Again, if you don't want local clients asking android.git.kernel.org
for anything you must not list the korg remote for any project in
default.xml. Secondly, make sure the revision element of each git
points to a valid branch. You can start off pointing them to aosp/master
(or in some cases aosp/tools_r8 or similar) if you've successfully
pushed refs/heads/* to refs/heads/aosp/*.
When that works I suggest you create your own development branch based
on the AOSP branch for each git and use that branch in the manifest.
To do this you can e.g. use the $REPO_RREV variable in a forall command.
That variable is set to the revision attribute listed in the manifest,
i.e. it'll be "aosp/master" for almost all gits. A little string
manipulation and you have the string you want.
repo forall -c 'git push ssh://.../$REPO_PROJECT \
origin/$REPO_RREV:refs/heads/`echo $REPO_RREV | sed "s/^aosp.//"`'
This should produce a command like this (again, verify by prepending
"echo" to the command before running anything):
git push ssh://.../$REPO_PROJECT origin/aosp/master:refs/heads/master
You should play around some with the forall command and its variables.
For example, try this:
repo forall -c 'echo $REPO_PROJECT uses $REPO_RREV'
> At this moment the feedback has been very nice but making it more
> difficult for me to understand it correctly. My knowledge of git
> is also not that much at the moment so its starting to get a bit
> confusing here :)
The stuff you want to do isn't what you'd usually start with, and it's
not obvious at first. Do take some time to dig into Git by reading a
book and experimenting.
On Wednesday, April 06, 2011 at 19:08 CEST,
> > Do you really need to rename the projects like that? It will beAh, okay. If all AOSP gits use the korg remote this is fine. However,
> > very confusing. Otherwise this should work, but I'd switch the
> > default revision to use your new remote. Otherwise you'll have
> > to remember to update the manifest every time you merge new
> > changes from upstream (although you'll have to do that anyway
> > since you're renaming the projects).
>
> The idea is to have a mixture of git repo's being updated. Most
> of the git's will be AOSP. And a few should only be work from me.
if your goal is to have your users sync everything from your local
server and not contact android.git.kernel.org at all it's not what
you want to do.
--
Magnus Bäck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson
> On Wed, Apr 6, 2011 at 10:41 PM, Magnus Bäck
> <magnu...@sonyericsson.com>wrote:
>
> > Ah, okay. If all AOSP gits use the korg remote this is fine.
> > However, if your goal is to have your users sync everything from
> > your local server and not contact android.git.kernel.org at all
> > it's not what you want to do.
>
> Yes indeed sorry for messing things up. I meant that the end goal was
> to have a mixed repo server.
> Most of the gits should checkout the korg git repo's and and for our
> edited sources to check out our changes from our gerrit server only.
> So this will mean I will have to edit the .repo/manifests/default.xml
> and adding a new remote fetch url that points to our gerrit git. With
> the remote project path's I guess?
I doubt we're using the same terminology, so it's probably better to
show an example and let the code to the talking.
I would be careful when mixing code straight from android.git.kernel.org
(remote="korg") with code from your own branches where you've made
changes. What if developers sync and get new code from AOSP that's
incompatible with your code? This way of working gets you the latest
AOSP changes immediately, which can be both good and bad. If you have
all people use your own server (remote="localserver") you can have a QA
process for making sure that AOSP changes doesn't break your stuff (i.e.
you won't update the local branches with new AOSP stuff until you've
verified that it works okay). I'm not saying this is necessary, but
depending on the amount of changes your make (and their magnitude) it
could make sense.
[...]