Can I using "repo init --mirror" and "repo sync" under review_site/git to maintain a slave mirror?

[Chunlin Zhang]

I am trying to maintain a gerrit slave mirror using "repo init --mirror" under review_site/git(because I have give up to use gerrit replication due to big git repository issue,see my previous post)

Now I got some problems in this kind of trying:

1. after repo init and repo sync,I found the "refs/meta/config" was not synced because in .git "remote.origin.fetch = +refs/heads/*:refs/heads/*",after I do "git config remote.origin.fetch +refs/*:refs/*",I can git fetch "refs/meta/config",but after repo sync,it turn back to +refs/heads/*:refs/heads/* ,can I make it always "+refs/*:refs/*"? I have no idea to do this with repo(or I use a customized repo)
2. now In slave gerrit project list,I can see ".repo/manifests" ".repo/repo" this kind of git repo,can I config some directory(e.g. .repo) to do not show in the gerrit project list?

Thanks! 

[Chunlin Zhang]

To my question 1 ,I have modify repo.git_config.Remote.ResetFetch() to

  def ResetFetch(self, mirror=False):

    """Set the fetch refspec to its default value.

    """

    if mirror:

      src = 'refs/*'

      dst = 'refs/*'

    else:

      src = 'refs/heads/*'

      dst = 'refs/remotes/%s/*' % self.name

    self.fetch = [RefSpec(True, src, dst)]

And test it ok

To question 2, I decided to ignore it.

Now I have test my solution of slave gerrit mirr using "repo init --mirror" almost OK.In this solution,slave gerrit use master gerrit's database directly but with a readonly database user connection,so that slave can run in slave mode and use access control setting of master gerrit.

The last problem I solved is that when I do some access control setting modification in master gerrit,the slave gerrit do not apply the new setting,what I need to do is:

- do repo sync in review_site/git to sync the new refs/meta/config

- run "gerrit flush-caches --all" to slave gerrit

Then will apply new access control setting to slave gerrit.

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Chunlin Zhang]

I am thinking of using CI tool(I choose buildbot) to manage the sync of mirror,so the sync action can do automatically by listening the gerrit event,but the change of  access control setting (refs/meta/config) can not be triggered by event,so need to sync manually

One of the benefits is that the result of the command line of "repo sync" and "gerrit flush-caches" can be viewed in CI web page

[Magnus Bäck]

On Monday, December 24, 2012 at 10:13 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> I am trying to maintain a gerrit slave mirror using "repo init
> --mirror" under review_site/git(because I have give up to use gerrit
> replication due to big git repository issue,see my previous post)

Have you tried using replication after setting up an initial mirror,
i.e. using some other method to populate the slave and then reattempt
replication? I wouldn't be surprised if an initial replication wouldn't
work for a gigantic repository but subsequent small replications of
deltas would work just fine.

> Now I got some problems in this kind of trying:
>

> 1. after repo init and repo sync,I found the "refs/meta/config" was

> not synced because in .git "remote.origin.fetch =
> +refs/heads/*:refs/heads/*",after I do "git config
> remote.origin.fetch +refs/*:refs/*",I can git fetch
> "refs/meta/config",but after repo sync,it turn back to
> +refs/heads/*:refs/heads/* ,can I make it always "+refs/*:refs/*"?
> I have no idea to do this with repo(or I use a customized repo)

> 2. now In slave gerrit project list,I can see ".repo/manifests"

> ".repo/repo" this kind of git repo,can I config some directory(e.g.
> .repo) to do not show in the gerrit project list?

Just don't use Repo for this. A Repo mirror workspace isn't suitable
for the task, and you've discovered why. If Gerrit replication *really*
doesn't work, use plain C Git and a cron job to push from the master
(or fetch from the slave if you prefer that).

--
Magnus Bäck
ba...@google.com

[Chunlin Zhang]

On Wed, Jan 23, 2013 at 10:42 PM, Magnus Bäck <ba...@google.com> wrote:

On Monday, December 24, 2012 at 10:13 EST,

     Chunlin Zhang <zhangc...@gmail.com> wrote:

> I am trying to maintain a gerrit slave mirror using "repo init
> --mirror" under review_site/git(because I have give up to use gerrit
> replication due to big git repository issue,see my previous post)

Have you tried using replication after setting up an initial mirror,
i.e. using some other method to populate the slave and then reattempt
replication? I wouldn't be surprised if an initial replication wouldn't
work for a gigantic repository but subsequent small replications of
deltas would work just fine.

I don't know why but it fail without meaningful log.

The first initial replication lasted for a long time because we don't have so good and stable network bandwidth between master and slave gerrit.

> Now I got some problems in this kind of trying:
>

>    1. after repo init and repo sync,I found the "refs/meta/config" was

>    not synced because in .git "remote.origin.fetch =
>    +refs/heads/*:refs/heads/*",after I do "git config
>    remote.origin.fetch +refs/*:refs/*",I can git fetch
>    "refs/meta/config",but after repo sync,it turn back to
>    +refs/heads/*:refs/heads/* ,can I make it always "+refs/*:refs/*"?
>    I have no idea to do this with repo(or I use a customized repo)

>    2. now In slave gerrit project list,I can see ".repo/manifests"

>    ".repo/repo" this kind of git repo,can I config some directory(e.g.
>    .repo) to do not show in the gerrit project list?

Just don't use Repo for this. A Repo mirror workspace isn't suitable
for the task, and you've discovered why. If Gerrit replication *really*
doesn't work, use plain C Git and a cron job to push from the master
(or fetch from the slave if you prefer that).

I think it is more flexible to use repo.If I use git,I will maintain all the git repository by hand.

But  I want to find out what happen ,what is the root cause,or I will get this problem even if I don't use repo.

[Magnus Bäck]

On Wednesday, January 23, 2013 at 21:59 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Wed, Jan 23, 2013 at 10:42 PM, Magnus Bäck <ba...@google.com> wrote:
>
> > Have you tried using replication after setting up an initial mirror,
> > i.e. using some other method to populate the slave and then
> > reattempt replication? I wouldn't be surprised if an initial
> > replication wouldn't work for a gigantic repository but subsequent
> > small replications of deltas would work just fine.
>
> I don't know why but it fail without meaningful log.

Even the small delta replication after an rsync copy is in place?

If even that doesn't work, you could use replication for all other gits
but skip the big one that's giving you problems.

[...]

> > Just don't use Repo for this. A Repo mirror workspace isn't suitable
> > for the task, and you've discovered why. If Gerrit replication
> > *really* doesn't work, use plain C Git and a cron job to push from
> > the master (or fetch from the slave if you prefer that).
>
> I think it is more flexible to use repo.If I use git,I will maintain
> all the git repository by hand.

No. You will use the ls-projects SSH command (or the equivalent REST
RPC) to find which projects there are. It's probably a ten-line shell
script that you can schedule on the slave via cron. A few additional
lines if you want to detect whether there were any updates to the set
of available projects or if any access rights might have been updated
so you won't have to flush the caches to often.

[...]

--
Magnus Bäck
ba...@google.com

[Chunlin Zhang]

On Thu, Jan 24, 2013 at 10:58 PM, Magnus Bäck <ba...@google.com> wrote:

On Wednesday, January 23, 2013 at 21:59 EST,

     Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Wed, Jan 23, 2013 at 10:42 PM, Magnus Bäck <ba...@google.com> wrote:
>

> > Have you tried using replication after setting up an initial mirror,
> > i.e. using some other method to populate the slave and then
> > reattempt replication? I wouldn't be surprised if an initial
> > replication wouldn't work for a gigantic repository but subsequent
> > small replications of deltas would work just fine.
>
> I don't know why but it fail without meaningful log.

Even the small delta replication after an rsync copy is in place?

I don't understand your method,why do gerrit replication  use rsync?

If even that doesn't work, you could use replication for all other gits
but skip the big one that's giving you problems.

In out gerrit ,there is several big git project, after git gc,they can be 3+G,but before git gc they can be 80+G,so I will not use rsync.

[...]

> > Just don't use Repo for this. A Repo mirror workspace isn't suitable
> > for the task, and you've discovered why. If Gerrit replication
> > *really* doesn't work, use plain C Git and a cron job to push from
> > the master (or fetch from the slave if you prefer that).
>
> I think it is more flexible to use repo.If I use git,I will maintain
> all the git repository by hand.

No. You will use the ls-projects SSH command (or the equivalent REST
RPC) to find which projects there are. It's probably a ten-line shell
script that you can schedule on the slave via cron. A few additional
lines if you want to detect whether there were any updates to the set
of available projects or if any access rights might have been updated
so you won't have to flush the caches to often.

I have to maintain 3 slave gerrit in 3 place,and I want each slave gerrit no have full mirror of master,but only part of all git project each place need to acccess.

If I use "repo init --mirror" to maintain slave gerrit,what I want to do is to maintain 3 branch of mirror manifest.

But I will try your method to see if this problem will happen.

[...]

--
Magnus Bäck
ba...@google.com

--

[Magnus Bäck]

On Thursday, January 24, 2013 at 10:59 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Thu, Jan 24, 2013 at 10:58 PM, Magnus Bäck <ba...@google.com> wrote:
>
> > Even the small delta replication after an rsync copy is in place?
>
> I don't understand your method,why do gerrit replication use rsync?

Gerrit replication doesn't use rsync. What I'm asking is: Does
replication work with your big repository if you start off by copying it
via (for example) rsync to the slave, and thereafter enable replication?
Presumably, the subsequent replication would be fairly small in size and
hopefully won't trigger whatever problem you're having when attempting
to replicate the full repository. Put differently, does the problem
appear to be related to the amount of data being pushed or is it
connected to the repository itself?

> > If even that doesn't work, you could use replication for all other
> > gits but skip the big one that's giving you problems.
>
> In out gerrit ,there is several big git project, after git gc,they can
> be 3+G,but before git gc they can be 80+G,so I will not use rsync.

What does "git gc" and rsync have to do with other?

> > No. You will use the ls-projects SSH command (or the equivalent REST
> > RPC) to find which projects there are. It's probably a ten-line shell
> > script that you can schedule on the slave via cron. A few additional
> > lines if you want to detect whether there were any updates to the set
> > of available projects or if any access rights might have been updated
> > so you won't have to flush the caches to often.
>
> I have to maintain 3 slave gerrit in 3 place,and I want each slave
> gerrit no have full mirror of master,but only part of all git project
> each place need to acccess.
> If I use "repo init --mirror" to maintain slave gerrit,what I want to
> do is to maintain 3 branch of mirror manifest.
> But I will try your method to see if this problem will happen.

Having to maintain three manifest branches is a lot more work. If you
want to have different sets of gits replicated to each location you can
have different users doing the fetching in each location and limit which
gits (or even branches within a git) they have read access to. Plus you
won't have to much around in the config file of each git to make sure
all refs are fetched by Repo.

I don't think abandoning Repo will make a difference in your case, but
using it for this purpose is a bad idea regardless.

--
Magnus Bäck
ba...@google.com

[Chunlin Zhang]

On Fri, Jan 25, 2013 at 12:09 AM, Magnus Bäck <ba...@google.com> wrote:

On Thursday, January 24, 2013 at 10:59 EST,

     Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Thu, Jan 24, 2013 at 10:58 PM, Magnus Bäck <ba...@google.com> wrote:
>

> > Even the small delta replication after an rsync copy is in place?
>
> I don't understand your method,why do gerrit replication  use rsync?

Gerrit replication doesn't use rsync. What I'm asking is: Does
replication work with your big repository if you start off by copying it
via (for example) rsync to the slave, and thereafter enable replication?
Presumably, the subsequent replication would be fairly small in size and
hopefully won't trigger whatever problem you're having when attempting
to replicate the full repository. Put differently, does the problem
appear to be related to the amount of data being pushed or is it
connected to the repository itself?

I think if gerrit replication fail in the initial push to slave,it can be fail in the subsequent action,I can not make sure now(the test environment have be destroied),but I think there was already happen that initial replication OK but subsequent replication  fail.

What I observe was that after gerrit replication start,but the action end quickly(according monitoring the process and the network connection) without meaningful log.

> > If even that doesn't work, you could use replication for all other
> > gits but skip the big one that's giving you problems.
>
> In out gerrit ,there is several big git project, after git gc,they can
> be 3+G,but before git gc they can be 80+G,so I will not use rsync.

What does "git gc" and rsync have to do with other?

Because if I face the 80G+ or ever 120G+ git repository directory,rsync will last for long long time. 

> > No. You will use the ls-projects SSH command (or the equivalent REST
> > RPC) to find which projects there are. It's probably a ten-line shell
> > script that you can schedule on the slave via cron. A few additional
> > lines if you want to detect whether there were any updates to the set
> > of available projects or if any access rights might have been updated
> > so you won't have to flush the caches to often.
>
> I have to maintain 3 slave gerrit in 3 place,and I want each slave
> gerrit no have full mirror of master,but only part of all git project
> each place need to acccess.
> If I use "repo init --mirror" to maintain slave gerrit,what I want to
> do is to maintain 3 branch of mirror manifest.
> But I will try your method to see if this problem will happen.

Having to maintain three manifest branches is a lot more work. If you
want to have different sets of gits replicated to each location you can
have different users doing the fetching in each location and limit which
gits (or even branches within a git) they have read access to. Plus you
won't have to much around in the config file of each git to make sure
all refs are fetched by Repo.

Maintain three branch of manifest is easy,I can have those people(who want to change this) do the change themselves,they only get and push new manifest and then it is OK.

Your method is also easy,but I have do it in master gerrit,so I should do it myself.But maybe I will choose your method,I even don't think this way.

I don't think abandoning Repo will make a difference in your case, but
using it for this purpose is a bad idea regardless.

Why? Does the repo style git repository mirror can not be compatible with regular git?

[Magnus Bäck]

On Thursday, January 24, 2013 at 21:38 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Fri, Jan 25, 2013 at 12:09 AM, Magnus Bäck <ba...@google.com> wrote:

[...]

> > What does "git gc" and rsync have to do with other?
>
> Because if I face the 80G+ or ever 120G+ git repository directory,rsync
> will last for long long time.

So run rsync after "git gc" then. Unless you only want to copy some of
the refs in each git there shouldn't have to be a significant difference
in payload depending on whatever transport protocol you choose. Now, if
it was subsequent small replications that failed it doesn't matter.

> > Having to maintain three manifest branches is a lot more work. If you
> > want to have different sets of gits replicated to each location you can
> > have different users doing the fetching in each location and limit which
> > gits (or even branches within a git) they have read access to. Plus you
> > won't have to much around in the config file of each git to make sure
> > all refs are fetched by Repo.
>
> Maintain three branch of manifest is easy,I can have those people(who want
> to change this) do the change themselves,they only get and push new
> manifest and then it is OK.
> Your method is also easy,but I have do it in master gerrit,so I should do
> it myself.But maybe I will choose your method,I even don't think this way.

No, you don't have to do it on the master. The script to obtain the
list of projects and fetch from all can run on each slave. This should
be reasonably close to working (but should have more error handling):

for p in $(ssh ... gerrit ls-projects) ; do
if [ ! -d "$p.git" ] ; then
mkdir -p $p.git && ( cd $p.git && git init --bare && \
git remote add --mirror=fetch origin ssh://.../$p )
fi
( cd $p.git && git fetch origin && git remote prune origin)
done

> > I don't think abandoning Repo will make a difference in your case, but
> > using it for this purpose is a bad idea regardless.
>
> Why? Does the repo style git repository mirror can not be compatible with
> regular git?

Reiterating what I said before:

- You'll have to maintain the manifest files.
- You'll either have to maintain the config files in the gits or
maintain a modified copy of Repo to get Repo to fetch all refs.
- The project list on the client will have bogus .repo/manifests and
.repo/repo entries.

The drawback with dropping Repo:

- You'll have to write a ten-line shell script. And I just wrote it
for you.

--
Magnus Bäck
ba...@google.com

[Chunlin Zhang]

On Fri, Jan 25, 2013 at 11:17 AM, Magnus Bäck <ba...@google.com> wrote:

On Thursday, January 24, 2013 at 21:38 EST,

     Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Fri, Jan 25, 2013 at 12:09 AM, Magnus Bäck <ba...@google.com> wrote:

[...]

> > What does "git gc" and rsync have to do with other?
>
> Because if I face the 80G+ or ever 120G+ git repository directory,rsync
> will last for long long time.

So run rsync after "git gc" then. Unless you only want to copy some of
the refs in each git there shouldn't have to be a significant difference
in payload depending on whatever transport protocol you choose. Now, if
it was subsequent small replications that failed it doesn't matter.

"git gc" will  spend much of time,I have to do it in night or sometime do not have so much online user

And the sync action of slave gerrit should be much offen,I am using CI tool(buildbot) to listen the master gerrit's even,if there have patch merged,slave gerrit will try to sync,and if I use repo , I can directly use the "repo sync -jX" feature to do multitasking sync.

But your previous suggestion seem to use rsync for the initial sync,and use gerrit replication or other method else?

> > Having to maintain three manifest branches is a lot more work. If you
> > want to have different sets of gits replicated to each location you can
> > have different users doing the fetching in each location and limit which
> > gits (or even branches within a git) they have read access to. Plus you
> > won't have to much around in the config file of each git to make sure
> > all refs are fetched by Repo.
>
> Maintain three branch of manifest is easy,I can have those people(who want
> to change this) do the change themselves,they only get and push new
> manifest and then it is OK.
> Your method is also easy,but I have do it in master gerrit,so I should do
> it myself.But maybe I will choose your method,I even don't think this way.

No, you don't have to do it on the master. The script to obtain the
list of projects and fetch from all can run on each slave. This should
be reasonably close to working (but should have more error handling):

for p in $(ssh ... gerrit ls-projects) ; do
  if [ ! -d "$p.git" ] ; then
    mkdir -p $p.git && ( cd $p.git && git init --bare && \
        git remote add --mirror=fetch origin ssh://.../$p )
  fi
  ( cd $p.git && git fetch origin && git remote prune origin)
done

OK,many thanks to your script,I will follow your  method,and try if it have the issue of losing projects&groups

> > I don't think abandoning Repo will make a difference in your case, but
> > using it for this purpose is a bad idea regardless.
>
> Why? Does the repo style git repository mirror can not be compatible with
> regular git?

Reiterating what I said before:

   - You'll have to maintain the manifest files.
   - You'll either have to maintain the config files in the gits or
     maintain a modified copy of Repo to get Repo to fetch all refs.
   - The project list on the client will have bogus .repo/manifests and
     .repo/repo entries.

The drawback with dropping Repo:

   - You'll have to write a ten-line shell script. And I just wrote it
     for you.

Thank you once again!

I will try this before debug gerrit. 

[Chunlin Zhang]

On Fri, Jan 25, 2013 at 2:08 PM, Chunlin Zhang <zhangc...@gmail.com> wrote:

No, you don't have to do it on the master. The script to obtain the
list of projects and fetch from all can run on each slave. This should
be reasonably close to working (but should have more error handling):

for p in $(ssh ... gerrit ls-projects) ; do
  if [ ! -d "$p.git" ] ; then
    mkdir -p $p.git && ( cd $p.git && git init --bare && \
        git remote add --mirror=fetch origin ssh://.../$p )
  fi
  ( cd $p.git && git fetch origin && git remote prune origin)
done

OK,many thanks to your script,I will follow your  method,and try if it have the issue of losing projects&groups

I have try your method,and found the git sync very OK,this is a clean way,but the gerrit still lost projects&groups.

So I begin to try debuging gerrit now ...

 

[Björn Pedersen]

thanks to your script,I will follow your  method,and try if it have the issue of losing projects&groups

I have try your method,and found the git sync very OK,this is a clean way,but the gerrit still lost projects&groups.

So I begin to try debuging gerrit now ...

 

Hi.

probably you have to explicitly fetch at least the All-Projects. a normal ls-projects will only list "real" projects.
You can use the --type argument to ls-projects to change that. Just run ssh ... gerrit ls-projects -h

[Chunlin Zhang]

(forgot to reply all) 

I use --type ALL but still having this issue,only 2 manifest can be seen.But I don't know if I use --tree,what meaning is the (x) in the output?

t$ ssh testmirr gerrit ls-projects --type ALL --tree

`-- (x)

    |-- (x)

    |-- modem/manifest

    |-- platform/manifest

    `-- (x) 

[Chunlin Zhang]

On Fri, Jan 25, 2013 at 7:33 PM, Chunlin Zhang <zhangc...@gmail.com> wrote:

On Fri, Jan 25, 2013 at 4:17 PM, Björn Pedersen <ice...@googlemail.com> wrote:

thanks to your script,I will follow your  method,and try if it have the issue of losing projects&groups

I have try your method,and found the git sync very OK,this is a clean way,but the gerrit still lost projects&groups.

So I begin to try debuging gerrit now ...

 

Hi.

probably you have to explicitly fetch at least the All-Projects. a normal ls-projects will only list "real" projects.
You can use the --type argument to ls-projects to change that. Just run ssh ... gerrit ls-projects -h

(forgot to reply all) 

I use --type ALL but still having this issue,only 2 manifest can be seen.But I don't know if I use --tree,what meaning is the (x) in the output?

t$ ssh testmirr gerrit ls-projects --type ALL --tree

`-- (x)

    |-- (x)

    |-- modem/manifest

    |-- platform/manifest

    `-- (x) 

And I try the origin slave gerrit server having this issue,there is more git projects there,and see:

$ ssh slavemirr gerrit ls-projects --tree --type ALL

`-- (x)

    |-- (x)

    |-- (x)

    |-- (x)

    |-- (x)

    |-- platform/manifest

    |-- (x)

    |-- (x)

    |-- (x)

    |-- (x)

    |-- (x)

    |-- (x)

    `-- (x)

 Look strange.

[Magnus Bäck]

On Friday, January 25, 2013 at 01:08 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> On Fri, Jan 25, 2013 at 11:17 AM, Magnus Bäck <ba...@google.com> wrote:
>
> > So run rsync after "git gc" then. Unless you only want to copy some
> > of the refs in each git there shouldn't have to be a significant
> > difference in payload depending on whatever transport protocol you
> > choose. Now, if it was subsequent small replications that failed it
> > doesn't matter.
>
> "git gc" will spend much of time,I have to do it in night or sometime
> do not have so much online user
> And the sync action of slave gerrit should be much offen,I am using CI
> tool(buildbot) to listen the master gerrit's even,if there have patch
> merged,slave gerrit will try to sync,and if I use repo , I can
> directly use the "repo sync -jX" feature to do multitasking sync.
> But your previous suggestion seem to use rsync for the initial
> sync,and use gerrit replication or other method else?

Yes. I have not suggested using rsync as the protocol for continuous
replication, only for priming the repository.

[Magnus Bäck]

On Friday, January 25, 2013 at 06:33 EST,

Chunlin Zhang <zhangc...@gmail.com> wrote:

> I use --type ALL but still having this issue,only 2 manifest can be
> seen.But I don't know if I use --tree,what meaning is the (x) in the
> output?
>
> t$ ssh testmirr gerrit ls-projects --type ALL --tree
> `-- (x)
> |-- (x)
> |-- modem/manifest
> |-- platform/manifest
> `-- (x)

The (x) means that the project isn't visible to you, i.e. you don't have
read access to it. This isn't surprising given that the root cause of
your problems is related to the loss of group membership. Ignore the
project list until your access rights have been restored.

--
Magnus Bäck
ba...@google.com

[Anushree Ganjam]

Hi,
I am trying replication of a big repository (which has .repo and 619 .git)
Now when i do replication from master to slave, only the .git is replicated. I cannot see .repo being replicated.

Doesn't replication support .repo?
Please explain. How do i maintain .repo in case of replication.


Thanks
Anushree

[David Pursehouse]

On 11/05/2015 05:54 PM, Anushree Ganjam wrote:
> Hi,
> I am trying replication of a big repository (which has .repo and 619 .git)
> Now when i do replication from master to slave, only the .git is
> replicated. I cannot see .repo being replicated.
>
> Doesn't replication support .repo?
> Please explain. How do i maintain .repo in case of replication.
>

The .repo folder is not replicated because it's not part of the git.
It's created in the workspace when you initialise a workspace with 'repo
init'.

>
> Thanks
> Anushree
>
> On Friday, January 25, 2013 at 8:47:50 AM UTC+5:30, Magnus Bäck wrote:
>
> On Thursday, January 24, 2013 at 21:38 EST,

> ba...@google.com <javascript:>

>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>

> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Anushree Ganjam]

ok, how about the branches?
How to enable the replication of branches.

Also I can see the message in replication_log
4ad5d5dc] Push to ger...@14.142.24.164:8080:/

home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git references: [RemoteRefUpdate[remoteName=refs/heads/android-5.1.1_r24_new, NOT_ATTEMPTED, (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7, srcRef=refs/heads/android-5.1.1_r24_new, forceUpdate, message=null], RemoteRefUpdate[remoteName=refs/changes/90/1090/1, NOT_ATTEMPTED, (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7, srcRef=refs/changes/90/1090/1, forceUpdate, message=null]]
[2015-11-02 06:52:28,729] [4ad5d5dc] Replication to ger...@14.142.24.164:8080:/home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git completed in 3105 ms


do you have any idea on What is this NOT_ATTEMPTED?

Even though replication is success, the code is not synced in the slave server.

Can anyone please explain this

[Ian Kumlien]

You're actually running a git server on port 8080?

Which ports are you're machines listening on with what services?

How do you verify the replication?

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to repo-discuss...@googlegroups.com.

[Anushree Ganjam]

Hi,

You're actually running a git server on port 8080?

Yes.

Which ports are you're machines listening on with what services?

Only gerrit is listening to port 8080

How do you verify the replication?

The replication plugin created folder in the slave server and all the git projects are present.

But no branches are present.

Thanks
Anushree

[Ian Kumlien]

On 5 November 2015 at 10:26, Anushree Ganjam <anushre...@gmail.com> wrote:
> Hi,
>
> You're actually running a git server on port 8080?
> Yes.

Not really, see below

> Which ports are you're machines listening on with what services?
> Only gerrit is listening to port 8080

You tend to use a *git* server not gerrit, but git to handle the replication.

> How do you verify the replication?
> The replication plugin created folder in the slave server and all the git
> projects are present.
> But no branches are present.

And you verified this how?

[David Pursehouse]

On 11/05/2015 06:26 PM, Anushree Ganjam wrote:
> Hi,
>
> You're actually running a git server on port 8080?
> Yes.
>
>
> Which ports are you're machines listening on with what services?
> Only gerrit is listening to port 8080
>
>
> How do you verify the replication?
> The replication plugin created folder in the slave server and all the
> git projects are present.
> But no branches are present.
>

As Ian said, how are you verifying this?

What happens if you try to git clone from the replica?

If the branches you're expecting are not there, double check the
replication config to make sure it's replicating the correct refspecs.

[Ian Kumlien]

On 5 November 2015 at 10:32, Anushree Ganjam <anushre...@gmail.com> wrote:
>> You're actually running a git server on port 8080?

>> Yes.
>
> Not really, see below
>
>> Which ports are you're machines listening on with what services?
>> Only gerrit is listening to port 8080
>
> You tend to use a *git* server not gerrit, but git to handle the
> replication.
>

> I agree your point. Even i have a gerrit set up and this replication is used
> by gerrit.

gerrit replication is not designed to replicate to gerrit - it expects
a git:// and a common shell over ssh or just using ssh.

>> How do you verify the replication?
>> The replication plugin created folder in the slave server and all the git
>> projects are present.
>> But no branches are present.
>
> And you verified this how?
>

> I saw this project through gerrit -> Projects -> List -> Mygitproject ->
> Branches .. Some Default branch is present.
> On the master server there are 3 branches.

This doesn't matter while the replication is effectively broken...

[Anushree Ganjam]

As Ian said, how are you verifying this?

What happens if you try to git clone from the replica?

If the branches you're expecting are not there, double check the replication config to make sure it's replicating the correct refspecs.

Hi,

Yes, David. Thanks or the reply.

That exactly was my doubt from beginning.

What should i mention in replication.config to replicate all branches.

Currently my replication.config looks like
[remote "testreplica164"]
  url = ger...@14.142.24.164:/home/gerrit2/gerrit_testsite/git/${name}.git
  projects = tmp_replica/*
  replicationDelay = 5
  push = +refs/heads/*:refs/heads/*
  push = +refs/tags/*:refs/tags/*
  push = +refs/*:refs/*
  threads = 4
  mirror = true
  adminUrl = ger...@14.142.24.164:/home/gerrit2/gerrit_testsite/git/${name}.git
  createMissingRepositories = true

Should i add anything more? Please help me solve this

[David Pursehouse]

On 11/05/2015 06:41 PM, Anushree Ganjam wrote:
> As Ian said, how are you verifying this?
>
> What happens if you try to git clone from the replica?
>
> If the branches you're expecting are not there, double check the
> replication config to make sure it's replicating the correct refspecs.
>
> Hi,
> Yes, David. Thanks or the reply.
> That exactly was my doubt from beginning.
> What should i mention in replication.config to replicate all branches.
>
> Currently my replication.config looks like
> [remote "testreplica164"]
> url = ger...@14.142.24.164:/home/gerrit2/gerrit_testsite/git/${name}.git

Try changing this to something like:

url = git://14.142.24.164/${name}.git

> projects = tmp_replica/*
> replicationDelay = 5
> push = +refs/heads/*:refs/heads/*
> push = +refs/tags/*:refs/tags/*
> push = +refs/*:refs/*
> threads = 4
> mirror = true
> adminUrl =
> ger...@14.142.24.164:/home/gerrit2/gerrit_testsite/git/${name}.git
> createMissingRepositories = true
>
> Should i add anything more? Please help me solve this
>
> On Thu, Nov 5, 2015 at 3:06 PM, David Pursehouse
> <david.pu...@sonymobile.com

> <mailto:david.pu...@sonymobile.com>> wrote:
>
> On 11/05/2015 06:26 PM, Anushree Ganjam wrote:
>
> Hi,
>
> You're actually running a git server on port 8080?
> Yes.
>
>
> Which ports are you're machines listening on with what services?
> Only gerrit is listening to port 8080
>
>
> How do you verify the replication?
> The replication plugin created folder in the slave server and
> all the
> git projects are present.
> But no branches are present.
>
>
> As Ian said, how are you verifying this?
>
> What happens if you try to git clone from the replica?
>
> If the branches you're expecting are not there, double check the
> replication config to make sure it's replicating the correct refspecs.
>
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com

> <mailto:repo-discuss...@googlegroups.com>.

[Anushree Ganjam]

What happens if you try to git clone from the replica?

Hi,

Yes. I get a warning message when I try to clone from this.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

Thanks

Anushree

[David Pursehouse]

On 11/05/2015 06:45 PM, Anushree Ganjam wrote:
> What happens if you try to git clone from the replica?
>
> Hi,
> Yes. I get a warning message when I try to clone from this.
> warning: remote HEAD refers to nonexistent ref, unable to checkout.
>

If you're only replicating the android-5.1.1_r24_new branch and the HEAD
is pointing to master, then this is expected.

'git branch -r' will show which branches actually exist on the remote.

[Anushree Ganjam]

Try changing this to something like:

url = git://14.142.24.164/${name}.git

Hi,
I changed the url and got below exception.

[2015-11-05 09:51:29,544] [6dcf82fa] Cannot replicate to git://14.142.24.164/home/gerrit2/gerrit_testsite/git/tmp_replica/platform/manifest.git
org.eclipse.jgit.errors.TransportException: git://14.142.24.164/home/gerrit2/gerrit_testsite/git/tmp_replica/platform/manifest.git: Connection refused
        at org.eclipse.jgit.transport.TransportGitAnon.openConnection(TransportGitAnon.java:150)
        at org.eclipse.jgit.transport.TransportGitAnon$TcpFetchConnection.<init>(TransportGitAnon.java:179)

The same transport exception i faced before when not mentioning 8080 (port)

Thanks

Anushree

[Anushree Ganjam]

If you're only replicating the android-5.1.1_r24_new branch and the HEAD is pointing to master, then this is expected.

'git branch -r' will show which branches actually exist on the remote.

Hi,

I am not mentioning in the replication.config as android-5.1.1_r24_new .

I have sent you the replication.config also in the previous mail.

Please help me how to replicate all the branches.

Thanks

Anushree

[Ian Kumlien]

you have to install and start git on the correct port - you also need ssh running with a preconfigured ssh key

Basically, gerrit assumes that it can sign in over ssh without having to know any password.

[Anushree Ganjam]

Hi,

Yes. Git is installed and running on right port.

I have already mentioned replication_log in the previous mail.

It shows clearly replication is success. "Created remote repository"

Even ssh of slave is present in known hosts of Master server.

I just want to know how to replicate the branches. Code is not syncing because branches are not replicated.

Thanks

Anushree

[Ian Kumlien]

How much have you replicated so far? Is it a permission issue? does git branch -v give you the correct response when you run it in the actual git on the slave?

According to your earlier emails, git is not up and running... So, how do you run it? xinetd? whats the configuration?

Your configuration stated above only uses ssh so you should be able to see if something goes wrong by examining the logfiles.

[Anushree Ganjam]

Hi,

How much have you replicated so far? Is it a permission issue? does git branch -v give you the correct response when you run it in the actual git on the slave?

I can see the git projects but not the branches.

According to your earlier emails, git is not up and running... So, how do you run it? xinetd? whats the configuration?

I am not sure, how to explain you this one. But when i see ps -e | grep git -- I can see some processes running for git.

Your configuration stated above only uses ssh so you should be able to see if something goes wrong by examining the logfiles.

Yes, I have examined the log files.

home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git references: [RemoteRefUpdate[remoteName=refs/heads/android-5.1.1_r24_new, NOT_ATTEMPTED, (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7, srcRef=refs/heads/android-5.1.1_r24_new, forceUpdate, message=null], RemoteRefUpdate[remoteName=refs/changes/90/1090/1, NOT_ATTEMPTED, (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7, srcRef=refs/changes/90/1090/1, forceUpdate, message=null]]
[2015-11-02 06:52:28,729] [4ad5d5dc] Replication to ger...@14.142.24.164:8080:/home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git completed in 3105 ms


do you have any idea on What is this NOT_ATTEMPTED?

Even though replication is success, the code is not synced in the slave server.

How do i replicate the branches.

Thanks

Anushree

[David Pursehouse]

On 11/05/2015 07:29 PM, Anushree Ganjam wrote:
> Hi,

> How much have you replicated so far? Is it a permission issue? does git
> branch -v give you the correct response when you run it in the actual
> git on the slave?
>
> I can see the git projects but not the branches.
>
> According to your earlier emails, git is not up and running... So, how
> do you run it? xinetd? whats the configuration?
>
> I am not sure, how to explain you this one. But when i see ps -e | grep
> git -- I can see some processes running for git.
>
> Your configuration stated above only uses ssh so you should be able to
> see if something goes wrong by examining the logfiles.
>
> Yes, I have examined the log files.
> home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git
> references:
> [RemoteRefUpdate[remoteName=refs/heads/android-5.1.1_r24_new,
> NOT_ATTEMPTED, (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7,
> srcRef=refs/heads/android-5.1.1_r24_new, forceUpdate, message=null],

> RemoteRefUpdate[remoteName=refs/changes/90/1090/1,*NOT_ATTEMPTED,*
> (null)...d16aaec37d44ab1b200b4c799feacc6a7f873af7,
> srcRef=refs/changes/90/1090/1, forceUpdate, message=null]]
> [2015-11-02 06:52:28,729] [4ad5d5dc] *Replication to
> ger...@14.142.24.164:8080:/home/gerrit2/gerrit_testsite/git/tmp_replica/platform/packages/apps/Browser.git
> completed in 3105 ms*
>
>
> do you have any idea on What is this *NOT_ATTEMPTED?
>
> *Even though replication is success, the code is not synced in the slave

> server.
>
> How do i replicate the branches.
>

Again, how are you confirming that the branches are not there?

What is the result of 'git branch -r'? What about 'git ls-remote'?

[Anushree Ganjam]

Hi David,
I have just cloned Browser repository.
Cloning into 'Browser'...
Checking connectivity... done.

warning: remote HEAD refers to nonexistent ref, unable to checkout.

Entered the Browser folder, which has only .git and no other folders
'git branch -r'

Result is nothing. No output shown

user@user-HP-ProBook-450-G2:~/new/test/Browser$ git ls-remote
From ssh://m...@14.142.24.164:29418/tmp_replica/platform/packages/apps/Browser
0000000000000000000000000000000000000000    capabilities^{}

Thanks

[Anushree Ganjam]

Hi ,
Can i do first repo init and repo sync on slave server.
And then proceed with replication ?
So Now i will have all .git projects being linked. Even the branches will be ready.
Is this the right way to do replication?

Regards
Anushree

[Chunlin Zhang]

You should use --mirror if you want to do replication in this way.

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/X4mKjLhJx-8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

[Anushree Ganjam]

Hi All,
Thanks for all your replies.
Replication is working well.

For a complete repo to be replicated.
1. Infact as we know .repo folder cannot be replicated.
2. So We need to setup mirror first on slave server
3. Then we need to setup replication on the master server to replicate the changes to slave server.


Kudos !!

- Anushree