Best way to setup mirror for Gerrit and all of its Git repos

[Sam Xiao]

Hi all,

I'm new to here and actually already opened a similar post in
Stackoverflow:
http://stackoverflow.com/questions/8828914/best-way-to-setup-mirror-for-gerrit-and-all-of-its-git-repos

But I couldn't find a good solution for my needs, so my team suggested
that I can ask in here.

Let's me re-post again here:
I had searched on the web for quite some times, but still couldn't
find a good guide on setting a Gerrit mirror.

I have a Gerrit server (named A), and I want to setup a mirror to a
new server (named B). I want pull from B instead of using the
processing power from A because A serves as the main Gerrit server and
I don't want to put more workload on it when I can mirror it using SSH
from B.

I'm using H2 database and I want to mirror the whole Gerrit + all of
its Git repos, not just simply Git repos because I want to keep the
permissions/history from the Gerrit database too.

I had managed to narrow down 2 possible ways to do it:

1) Using 'gerrit replicate' http://gerrit.googlecode.com/svn/documentation/2.1.6/cmd-replicate.html
Not sure what it meant by "[--url < PATTERN >]". Where should I put
run this command? Is it on server A or B?

2) Use 'rsync' which is OK, but then I cannot sync review_site/etc/
gerrit.config because it contains the IP of server A and don't want to
have that on server B.

A follow-up testing on Rsync. I used Rsync both Git repo and Gerrit's
Review_Site directories. Then I ran this command on Server B 'java -
jar gerrit-2.1.6.war init -d /home/gerrit2/review_site/' to re-
initialize it. Then start it with 'review_site/bin/gerrit.sh start'.
But all the groups seem to be missing.


My goal is simple: running a mirror of both Gerrit and the
repositories. And having Gerrit in mirror meaning we will like to
keep all the user accounts/groups/permissions/etcs all the same.

If anyone has a better ways/solution recommended for this, please let
me know!

Thanks in advanced.
Sam.

[Matthias Sohn]

2012/1/13 Sam Xiao <s.x...@sta.samsung.com>

If you want a read-only mirror follow [1].

If you want a standby server which can takeover the master server's role in

case this breaks due to e.g. hardware failure you may try the following: 

We use the following setup to feed a standby mirror server which can be

switched in a few minutes to replace the master Gerrit server in case it e.g. 

breaks due to hardware failure.

For that we use Gerrit's builtin replication to mirror all repositories to the 

standby server into the Gerrit installation we have installed also on this server 

so that the Gerrit application on the standby server finds these repositories.

In order to also mirror the database we did setup postgres log replication

mirroring new database logs every few minutes to the standby server's

postgres database over ssh which is running in recovery mode.

In order to fail-over to the standby server, restart the standby server's

database in normal operation mode and start the Gerrit application on

the backup server and switch the DNS entry pointing to the Gerrit server.

[1] http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.2/config-replication.html

--
Matthias

[Sam Xiao]

I want a duplicate server (a mirror, but not just read-only)

Is there a way to mirror the database if I'm using the built-in H2?


On Jan 16, 1:58 pm, Matthias Sohn <matthias.s...@googlemail.com>
wrote:

> 2012/1/13 Sam Xiao <s.x...@sta.samsung.com>
>
>
>
>
>
>
>
>
>
> > Hi all,
>
> > I'm new to here and actually already opened a similar post in
> > Stackoverflow:
>

> >http://stackoverflow.com/questions/8828914/best-way-to-setup-mirror-f...

> [1]http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.2/co...
>
> --
> Matthias

[Shawn Pearce]

On Mon, Jan 16, 2012 at 14:14, Sam Xiao <s.x...@sta.samsung.com> wrote:
> I want a duplicate server (a mirror, but not just read-only)
>
> Is there a way to mirror the database if I'm using the built-in H2?

Not easily, no. The built-in H2 only works as a local process.

If you want to use H2 replicated, you need to setup an H2 database
server process yourself using the H2 documentation and distribution,
then point the Gerrit server at the H2 server IP and port. By the time
you go through this work you may be better off switching the database
to PostgreSQL or MySQL, where the failover/warm-standby system
configuration is well documented and support is readily available from
any number of companies.

This is why Matthias suggested using the database replication system
to make the backup server stay roughly current with the master.

Also, unfortunately Gerrit is still a single-master system. There can
be only one server running the Gerrit proces in master-mode that
accepts writes.

[Sam Xiao]

Thanks Shawn.

I'm glad to get this response from the Gerrit person. ;-)

If we need to switch to MySQL or PostgreSQL, is there a way or script
to migrate the H2 db content to it?

[Shawn Pearce]

On Tue, Jan 17, 2012 at 14:11, Sam Xiao <s.x...@sta.samsung.com> wrote:
> If we need to switch to MySQL or PostgreSQL, is there a way or script
> to migrate the H2 db content to it?

Nope. :-(

[Matthias Sohn]

2012/1/17 Sam Xiao <s.x...@sta.samsung.com>

Thanks Shawn.

I'm glad to get this response from the Gerrit person. ;-)

If we need to switch to MySQL or PostgreSQL, is there a way or script
to migrate the H2 db content to it?

maybe SQL workbench [1] or DB copy plugin of SQuirrel [2] can help to 

migrate the data

[1] http://www.sql-workbench.net/manual/data-pumper.html

[2] http://dbcopyplugin.sourceforge.net/

--
Matthias

[Sam Xiao]

If I use this way to backup the H2 db as a SQL-syntax text script
file: http://www.h2database.com/html/tutorial.html#upgrade_backup_restore
Then I import it into MySQL, and let Gerrit connect to the newly
created MySQL db, would this work?

Thanks!

[Sam Xiao]

Hi Shawn,

I managed to extract the SQL in script format ending .sql by using
"gsql \> SCRIPT to 'backup.zip' compression zip;" after connecting to
H2's DB using "sudo java -jar gerrit.war gsql -d review_site"

Now having the .sql backup script file, is it possible to run this and
migrate to MySQL?
Or do I need to make further modification?
Or what's the database name/user accounts/etc I need to modify/use for
loading this onto MySQL?

Thanks in advanced
Sam.

On Jan 17, 3:17 pm, Shawn Pearce <s...@google.com> wrote:

[Tariq Mehmood]

Hi, I am also working on the same issue and at the same stage as you are.

any updates are very much appreciated.......................

[Phil Hord]

Tariq,

I experimented with this some to migrate from Postgres to MySQL.  What worked for me was this procedure:

1. Init the target database (MySQL) using the normal Gerrit init methods. 
  a. This ensures the schema is "correct" for the target database.
  b. Make sure your target gerrit is running the same schema version as the source.

2. Export the database to a SQL script.  Exclude or strip the schema creation part of the script.

3. Convert the SQL script to be compatible with the target (MySQL) database.
  a. I used pg2mysql_cli.php

4. Remove some competing information from the SQL script.  I discovered this list by experimenting and waiting for MySQL to complain.  You should do the same.
I wound up with this to mogrify my /tmp/foo.mysql script, but this is probably not compatible with your current Gerrit version and DB combination, so you should investigate here on your own:

  sed -ibak -e '1i delete from account_group_names ; delete from account_groups;
     delete from approval_categories; delete from approval_category_values;
     delete from schema_version; ' -e '/system_config/d'    /tmp/foo.mysql

5. Import the data using the SQL script.

6. Start Gerrit on the target and test, test, test.

Phil

[Magnus Bäck]

On Friday, March 29, 2013 at 11:40 EDT,
Wei Wang <weiw...@gmail.com> wrote:

> Just to confirm, using the strategy that Matthias Sohn wrote, I am
> only able to send updates from Gerrit Server A to Gerrit Server B,
> and not the other way around?
>
> That means that all of the users must only continue to git push to
> Gerrit Server A, and should not ever push to Gerrit Server B?

Yes. Server B will be configured to reject pushes. The described setup
isn't a multi-master configuration where both servers can accept write
operations. It only allows server B to be used for failovers and to
share the load with server A.

--
Magnus Bäck
ba...@google.com

[Wei Wang]

OK so I have decided to do this strategy.

But the problem is that I have no idea how to go about it, and I can't really find any documentation.

How do I install gerrit in "slave mode"? All I can find is this page:

http://gerrit-documentation.googlecode.com/svn/Documentation/2.5.2/config-gerrit.html#_a_id_container_a_section_container

It does not even properly explain what exactly "slave mode" entails. Anywhere I can go to read about it?

[Magnus Bäck]

On Wednesday, April 17, 2013 at 19:42 EDT,

Wei Wang <weiw...@gmail.com> wrote:

> OK so I have decided to do this strategy.
>
> But the problem is that I have no idea how to go about it, and I can't
> really find any documentation.
>
> How do I install gerrit in "slave mode"? All I can find is this page:
> http://gerrit-documentation.googlecode.com/svn/Documentation/2.5.2/config-gerrit.html#_a_id_container_a_section_container

Yes, I believe that's the only setting that you need to change to make a
server into a slave.

> It does not even properly explain what exactly "slave mode" entails.
> Anywhere I can go to read about it?

I don't think there's any end-to-end documentation on setting up a slave
server and enabling replication to it, but the documentation of the
Gerrit daemon itself explains the --slave option in (I think) sufficient
detail.

http://gerrit-documentation.googlecode.com/svn/Documentation/2.5.2/pgm-daemon.html

Replication was moved to a plugin in Gerrit 2.5, and if you install
the replication plugin it'll publish its own documentation under
/plugins/replication/Documentation/. See also
https://code.google.com/p/gerrit/issues/detail?id=1641.

--
Magnus Bäck
ba...@google.com

[mannemgo...@gmail.com]

Hi Shawn,

While trying to replicate using replication plugin, the mirrors are not getting automatically synced.

And also can we replicate only particular repo's on where the changes are done(like:-"ref-updated hook"), if yes please guide me.

My replication config looks like below:

[remote "************"]

  url = gerrit@****.ad.example.com:/path/${name}.git

  push = +refs/heads/*:refs/heads/*

  push = +refs/tags/*:refs/tags/*

#  push = +refs/for*:refs/for*

  timeout = 45

  replicationDelay = 5

  authGroup = Administrators

  autoReload = true

  replicateOnStartup = true

  defaultForceUpdate = true

  threads = 4

Thanks in advance!

Gopi

[Sven Selberg]

Hi,

You might want to give some more details about what "the mirrors are not gettting automatically synced" means. Like have you tried to force a replication through the command line tools? What do you see in the logs (error_log, replication_log).

> And also can we replicate only particular repo's on where the changes are done(like:-"ref-updated hook"), if yes please guide me.

The replication plugin handles that for you out-of-the-box.

/Sven

[gk]

Hello Sven,

Thanks for your reply.

I see no logs reported in the error_log and replication_log. But when i create a project in master it is not reflected in the mirror UI until or unless i do a restart of mirror.

    > Like have you tried to force a replication through the command line tools?

Yes, we do use command line tools for replication by running a cronjob(replication start --all) 

   >The replication plugin handles that for you out-of-the-box

How to get it done? Any script i need to use or i followed the documentation provided by gerrit but was not lucky(ref-updated --oldrev <old rev> --newrev <new rev> --refname <ref name> --project <project name> --submitter <submitter>). please guide me using the hook.

Thanks,

GK

[scgov...@gmail.com]

Hi Shawn. it is very useful discussion for me. in my case we are trying to replicate the SSH key into the slaves thinking may be the user can add their SSh keys to the master and then they can perform the cloning from the slave without adding the ssh keys again into the slave. simply we want to add the keys into master and then can be able to clone from any of the mirror closer and available for the user.

Is there any such possibility. In our setup we are using different databases for the master and slaves. master using the postgreSQL and slaves have H2, mysql.

If we configure the slaves with the same database as the master, can we achieve the ssh keys replication.

What is the best practice for the database configuration in case of gerrit master and mirror setup.

[David Pursehouse]

On Thu, Jul 20, 2017 at 2:35 PM <scgov...@gmail.com> wrote:

Hi Shawn. it is very useful discussion for me. in my case we are trying to replicate the SSH key into the slaves thinking may be the user can add their SSh keys to the master and then they can perform the cloning from the slave without adding the ssh keys again into the slave. simply we want to add the keys into master and then can be able to clone from any of the mirror closer and available for the user.

Is there any such possibility. In our setup we are using different databases for the master and slaves. master using the postgreSQL and slaves have H2, mysql.

If we configure the slaves with the same database as the master, can we achieve the ssh keys replication.

What version of Gerrit are you using?

Since version 2.13 the users' ssh keys are stored in git (in the All-Users repository) rather than in the database.

See [1] for details.

[1] https://gerrit-review.googlesource.com/#/c/76474/

 

What is the best practice for the database configuration in case of gerrit master and mirror setup.

On Tuesday, January 17, 2012 at 6:17:54 PM UTC-5, Shawn Pearce wrote:

On Tue, Jan 17, 2012 at 14:11, Sam Xiao <s.x...@sta.samsung.com> wrote:
> If we need to switch to MySQL or PostgreSQL, is there a way or script
> to migrate the H2 db content to it?

Nope. :-(

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.