How to create a private mirror of the Code Aurora Forum repository?

[Paul Danset]

Would anyone care to share an up-to-date set of instructions on how to create a private mirror of the Code Aurora Forum (CAF) Android repository?

Creating a private mirror of AOSP has been covered here before; we've loosely been following these instructions among many others:

https://groups.google.com/forum/#!topic/repo-discuss/qXBILfjVV04

https://groups.google.com/forum/#!topic/repo-discuss/xuOk10tn7cM

What we've done so far.

First we create a bare local mirror using the --mirror flag.  ($TAG below refers to the proper tag for our case) :

$ cd /home/git      # this is the path where our Gerrit instance keeps its git repositories

$ repo init --mirror -b release -m $TAG \

  -u git://codeaurora.org/platform/manifest.git \

  --repo-url=git://codeaurora.org/tools/repo.git

$ repo sync -j8

Once the "repo sync" completes, restarting Gerrit shows all the new ~350 CAF git projects:

$ gerrit.sh restart

$ ssh -p 29418 myserver gerrit ls-projects

We then clone our private repo, edit the manifest.xml to point to our own server, and sync it:

$ mkdir ~/CAF

$ cd ~/CAF

$ repo init -b release -m $TAG \ 

  -u git://codeaurora.org/platform/manifest.git \

  --repo-url=git://codeaurora.org/tools/repo.git

$ vi .repo/manifest.xml

$ cd .repo/manifests

$ git diff

...

   <manifest>

  -  <remote fetch="git://codeaurora.org/" name="caf" review="codeaurora.org"/>

  +  <remote fetch="ssh://myserver:29418/" name="caf" review="http://myserver:8080/"/>

...

$ git commit -a -m "Point to local server."

$ cd ~/CAF

$ repo sync -j8

$ repo status

nothing to commit (working directory clean)

So far so good.  We've created a local client/copy off of our internal mirror, although it is not associated with any branch.  However, we have not been able to upload any changes made to our local copy up to our internal mirror.  We've tried dozens of variations without success, using "repo upload" or "repo forall -c git push ..." or simply even "git push ..."

Example:

$ repo start test001 --all

$ repo checkout test001

$ repo status

# on branch test001

nothing to commit (working directory clean)

$ cd kernel

$ echo "TEST LINE\n" >> README

$ git commit -a -m "added test line"

$ git remote -v

caf ssh://myserver:29418/kernel/msm.git (fetch)

caf ssh://myserver:29418/kernel/msm.git (push)

$ git push caf test001

...   

To ssh://myserver:29418/kernel/msm.git

 ! [remote rejected] test001 -> test001 (prohibited by Gerrit)

error: failed to push some refs to 'ssh://myserver:29418/kernel/msm.git'

$ git push caf HEAD:refs/heads/test001

...

To ssh://myserver:29418/kernel/msm.git

 ! [remote rejected] HEAD -> test001 (prohibited by Gerrit)

error: failed to push some refs to 'ssh://mysever:29418/kernel/msm.git'

$ git push caf test001 HEAD:refs/for/test001

...

To ssh://myserver:29418/kernel/msm.git

 ! [remote rejected] HEAD -> refs/for/test001 (branch test001 not found)

error: failed to push some refs to 'ssh://myserver:29418/kernel/msm.git'

In order to address the "prohibited by Gerrit" error, I've given myself various permissions (e.g. "Push", "Push Merge Commit", "Push Annotated Tag", etc) on various references (e.g. "refs/heads/*", "refs/for/*", "refs/*") without success.  Unfortunately neither the Gerrit logs nor the git trace (via "export GIT_TRACE=2") have revealed what permissions are needed.

And since "git push ..." doesn't work, it's no surprise "repo upload" doesn't work either:

$ repo --trace upload

: unpickle /home/pdanse/CAF/.repo/manifests.git/config

: unpickle /home/pdanse/.gitconfig

: load refs /home/pdanse/CAF/.repo/projects/abi/cpp.git

: unpickle /home/pdanse/CAF/.repo/projects/abi/cpp.git/config

: export GIT_DIR=/home/pdanse/CAF/.repo/projects/abi/cpp.git

: git rev-list --abbrev=8 --abbrev-commit --pretty=oneline --reverse --date-order ^eb789ea833d8d800662b67914d9c1785a58c2caa refs/heads/test001 -- 1>| 2>|

: load refs /home/pdanse/CAF/.repo/projects/bionic.git

: unpickle /home/pdanse/CAF/.repo/projects/bionic.git/config

...

...

: export GIT_DIR=/home/pdanse/CAF/.repo/projects/kernel.git

: git log --pretty=format:%cd -n 1 refs/heads/test001 -- 1>| 2>|

Upload project kernel/:

  branch test001 ( 1 commit, Tue Feb 25 04:01:26 2014 +0000):

         546734e9 added test line

to http://myserver:8080/ (y/n)? y

: cd /home/pdanse/CAF/kernel

: git update-index -q --unmerged --ignore-missing --refresh 1>| 2>|

: git diff-index -z --cached HEAD 1>| 2>|

: git diff-files -z 1>| 2>|

: git ls-files -z --others --exclude-standard 1>| 2>|

: git var GIT_COMMITTER_IDENT 1>| 2>|

: git push --receive-pack=gerrit receive-pack ssh://pdanse@myserver:29418/kernel/msm refs/heads/test001:refs/for/3d4a060263030250b8a768e2ac6ad783cefbc6b1

Counting objects: 5, done.

Delta compression using up to 12 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (3/3), 338 bytes, done.

Total 3 (delta 2), reused 0 (delta 0)

remote: Resolving deltas: 100% (2/2)

remote: Processing changes: refs: 1, done    

To ssh://pdanse@myserver:29418/kernel/msm

 ! [remote rejected] test001 -> refs/for/3d4a060263030250b8a768e2ac6ad783cefbc6b1 (branch 3d4a060263030250b8a768e2ac6ad783cefbc6b1 not found)

error: failed to push some refs to 'ssh://pdanse@myserver:29418/kernel/msm'

----------------------------------------------------------------------

[FAILED] kernel/         test001         (Upload failed)

What am I doing wrong?  Any advice / pointers would be greatly appreciated.

Paul

[David Pursehouse]

Your manifest has projects whose revision is set to a sha1 rather than a branch.

When you do 'repo upload' it pushes to 'refs/for/branch' where 'branch' is the revision of the project in the manifest.

[Paul Danset]

Hi David,

On Tuesday, February 25, 2014 6:29:33 AM UTC-8, David Pursehouse wrote:

Your manifest has projects whose revision is set to a sha1 rather than a branch.

When you do 'repo upload' it pushes to 'refs/for/branch' where 'branch' is the revision of the project in the manifest.

Thank you for your response.  Yes I think this is one of the differences between AOSP and CAF.  The CAF mirror I set up via "repo init --mirror ..." is not associated with any branch out of the box.  How do I fix this?

Thanks

Paul

 

[Magnus Bäck]

On Tuesday, February 25, 2014 at 08:39 EST,

Paul Danset <paul....@gmail.com> wrote:

> Would anyone care to share an up-to-date set of instructions on how to
> create a private mirror of the Code Aurora Forum (CAF) Android
> repository?
>
> Creating a private mirror of AOSP has been covered here before; we've
> loosely been following these instructions among many others:
>
> https://groups.google.com/forum/#!topic/repo-discuss/qXBILfjVV04
> https://groups.google.com/forum/#!topic/repo-discuss/xuOk10tn7cM

Mirroring CAF isn't much different.

> What we've done so far.
>
> First we create a bare local mirror using the --mirror flag. ($TAG
> below refers to the proper tag for our case) :
>

> $ *cd /home/git* # this is the path where our Gerrit instance keeps
> its git repositories
> $ *repo init --mirror -b release -m $TAG \*
> * -u git://codeaurora.org/platform/manifest.git \*
> * --repo-url=git://codeaurora.org/tools/repo.git*
> $ *repo sync -j8*

>
>
> Once the "repo sync" completes, restarting Gerrit shows all the new ~350
> CAF git projects:

You don't have to restart Gerrit. Flushing the project cache is enough.

https://gerrit-documentation.storage.googleapis.com/Documentation/2.8/cmd-flush-caches.html

> $ *gerrit.sh restart*
> $ *ssh -p 29418 myserver gerrit ls-projects*

>
> We then clone our private repo, edit the manifest.xml to point to our own
> server, and sync it:

With a relative remote fetch URL in the manifest ('..'), like in AOSP,
you don't have to adjust the URL in the manifest. It's surprising that
CAF haven't adopted that.

[...]

> Example:
>
> $ *repo start test001 --all*
> $ *repo checkout test001*

'repo start' checks out the newly created topic branch for you.

> $ *repo status*

> # on branch test001
> nothing to commit (working directory clean)
>

> $ *cd kernel*
> $ *echo "TEST LINE\n" >> README*
> $ *git commit -a -m "added test line"*
>
> $ *git remote -v*

> caf ssh://myserver:29418/kernel/msm.git (fetch)
> caf ssh://myserver:29418/kernel/msm.git (push)
>

> $ *git push caf test001*

> ...
> To ssh://myserver:29418/kernel/msm.git
> ! [remote rejected] test001 -> test001 (prohibited by Gerrit)
> error: failed to push some refs to 'ssh://myserver:29418/kernel/msm.git'

This tries to push the test001 branch to a test001 branch on
the server. Presumably you don't have such a branch and you
don't have Gerrit permissions to create it.

> $ *git push caf HEAD:refs/heads/test001*

> ...
> To ssh://myserver:29418/kernel/msm.git
> ! [remote rejected] HEAD -> test001 (prohibited by Gerrit)
> error: failed to push some refs to 'ssh://mysever:29418/kernel/msm.git'

Same here.

> $ *git push caf test001 HEAD:refs/for/test001*

> ...
> To ssh://myserver:29418/kernel/msm.git
> ! [remote rejected] HEAD -> refs/for/test001 (branch test001 not found)
> error: failed to push some refs to 'ssh://myserver:29418/kernel/msm.git'

Indeed, there's no test001 branch to push to.

> In order to address the "prohibited by Gerrit" error, I've given
> myself various permissions (e.g. "Push", "Push Merge Commit", "Push
> Annotated Tag", etc) on various references (e.g. "refs/heads/*",
> "refs/for/*", "refs/*") without success. Unfortunately neither the
> Gerrit logs nor the git trace (via "export GIT_TRACE=2") have revealed
> what permissions are needed.

Did you grant the Create Reference permission?

> And since "git push ..." doesn't work, it's no surprise "repo upload"
> doesn't work either:
>

> $ *repo --trace upload*

[...]

> To ssh://pdanse@myserver:29418/kernel/msm
> ! [remote rejected] test001 ->
> refs/for/3d4a060263030250b8a768e2ac6ad783cefbc6b1 (branch
> 3d4a060263030250b8a768e2ac6ad783cefbc6b1 not found)
> error: failed to push some refs to 'ssh://pdanse@myserver:29418/kernel/msm'
>
> ----------------------------------------------------------------------
> [FAILED] kernel/ test001 (Upload failed)

While unsurprising, this is a different and unrelated problem. The
manifest's revision attribute for this project specifies a SHA-1 instead
of a branch. That attribute it used to decide the push destination for
uploads, so if you want 'repo upload' to push to the test001 branch
that's what you have to list in your manifest.

If you haven't done so already, it's time to think about your branching
strategy. You'll want to segregate the branch coming from CAF (and
other upstreams) and the branches where you make your own changes.

A good way of doing this is using branch namespacing, e.g. naming your
own branches yourcompanyname/* or pushing the upstream branches to
nameofupstream/*. I prefer the latter since the common case is shorter
and because it easily supports multiple upstreams.

--
Magnus Bäck
ba...@google.com

[Magnus Bäck]

On Tuesday, February 25, 2014 at 09:45 EST,

This isn't really related to CAF vs. AOSP. I'm sure there are CAF
manifests that list branch names and AOSP manifests with SHA-1s.

You have to create branches on your server that you use to work on
and update your manifest to point there. See my previous message in
this thread.

--
Magnus Bäck
ba...@google.com

[Paul Danset]

Hi Magnus,

On Tuesday, February 25, 2014 7:00:57 AM UTC-8, Magnus Bäck wrote:

On Tuesday, February 25, 2014 at 08:39 EST,
     Paul Danset <paul....@gmail.com> wrote:

[...] 

> In order to address the "prohibited by Gerrit" error, I've given
> myself various permissions (e.g. "Push", "Push Merge Commit", "Push
> Annotated Tag", etc) on various references (e.g. "refs/heads/*",
> "refs/for/*", "refs/*") without success.  Unfortunately neither the
> Gerrit logs nor the git trace (via "export GIT_TRACE=2") have revealed
> what permissions are needed.

Did you grant the Create Reference permission?

Ah thank you!  After adding the "Create Reference" permission, I was able to run the command below to push the first simple change up to our private repo mirror.  One problem down.

$ repo forall -c git push caf test001

[...] 

 

> And since "git push ..." doesn't work, it's no surprise "repo upload"
> doesn't work either:
>
> $ *repo --trace upload*

[...]

> To ssh://pdanse@myserver:29418/kernel/msm
>  ! [remote rejected] test001 ->
> refs/for/3d4a060263030250b8a768e2ac6ad783cefbc6b1 (branch
> 3d4a060263030250b8a768e2ac6ad783cefbc6b1 not found)
> error: failed to push some refs to 'ssh://pdanse@myserver:29418/kernel/msm'
>
> ----------------------------------------------------------------------
> [FAILED] kernel/         test001         (Upload failed)

While unsurprising, this is a different and unrelated problem. The
manifest's revision attribute for this project specifies a SHA-1 instead
of a branch. That attribute it used to decide the push destination for
uploads, so if you want 'repo upload' to push to the test001 branch
that's what you have to list in your manifest.

[...]

How specifically should I add a new repo branch?  For example, by hand editing .repo/manifest.xml or adding a local_manifest.xml?

I tried hand editing the original CAF .repo/manifest.xml file by removing all the SHA-1 hash revisions, specifying the "test001" branch explicitly, and then repo syncing, without success:

Before:

  <manifest>

    <remote fetch="git://codeaurora.org/" name="caf" review="codeaurora.org"/>

    <default remote="caf" revision="jb_2.5_auto"/>

    ...

    <project groups="default" name="platform/bionic" path="bionic" revision="65994cde4686dee2e59d1d7cb2cab494d4a55a40"/>

    ...

After:

  <manifest>

    <remote fetch="ssh://mysever:29418/" name="caf" review="http://myserver:8080/"/>

    <default remote="caf" revision="test001" dest-branch="test001"/>

    ...

    <project groups="default" name="platform/bionic" path="bionic" />

    ...

  $ repo sync bionic

  $ repo status

  nothing to commit (working directory clean)

  $ repo checkout test001

  error: no project has branch test001

  $ cd bionic

  $ git branch

  * (no branch)

Thank you very much for your help.  Best regards,

Paul

[Magnus Bäck]

On Tuesday, February 25, 2014 at 14:51 EST,

Paul Danset <paul....@gmail.com> wrote:

[...]

> How specifically should I add a new repo branch? For example, by hand
> editing .repo/manifest.xml or adding a local_manifest.xml?

Edit .repo/manifests/default.xml (which .repo/manifest.xml is a symlink
to), commit, and push to your upstream.

> I tried hand editing the original CAF .repo/manifest.xml file by
> removing all the SHA-1 hash revisions, specifying the "test001"
> branch explicitly, and then repo syncing, without success:

[...]

> $ repo sync bionic
> $ repo status
> nothing to commit (working directory clean)
>
> $ repo checkout test001
> error: no project has branch test001

This is expected unless you already had a topic branch with that name.

> $ cd bionic
> $ git branch
> * (no branch)

This is expected. With Repo, unless you explicitly create a topic branch
with 'repo start' (or check out a local branch by other means) you'll be
on a detached head. However, if you do run 'repo start' at this point,
the resulting topic branch will be based on the test001 branch from the
upstream.

--
Magnus Bäck
ba...@google.com