Gerrit Replication Plugin and HA

38 views
Skip to first unread message

Tom Printy

unread,
Apr 16, 2020, 10:46:43 AM4/16/20
to Repo and Gerrit Discussion
Hello,

I am in the process of implementing a single site HA instance. Currently i have several Gerrit slaves located throughout the world. I am currently using the replication plug-in to keep these slaves up to date.

My HA instance will have 3 Nodes at the master site (A,B,C) only one will be configured as the writter (accepting a push). Which node(s) should I run the replication plugin on?  My thought it one of the other nodes that is NOT serving as the writer (B or C).

What is the correct one?

Thanks
-Tom

Luca Milanesio

unread,
Apr 16, 2020, 11:04:00 AM4/16/20
to Tom Printy, Luca Milanesio, Repo and Gerrit Discussion
All of them, as I guess if A goes down then B or C will take over and therefore they’ll need to replicate changes / ref-updates.

Luca.

Marco Miller

unread,
Apr 16, 2020, 11:05:26 AM4/16/20
to Repo and Gerrit Discussion
-My internal team at Ericsson is currently working on the HA plugin ([1] below) to
1. sync project events from writing/active master to its passive companion, then
2. trigger the corresponding replication events towards replication plugin (on passive),
3. so that the replication plugin does that replication from passive master, offloading the active one.

Which HA plugin are you planning to use, [1] or multi-site [2], and from which Gerrit version first?

/Marco

Luca Milanesio

unread,
Apr 16, 2020, 11:08:40 AM4/16/20
to Marco Miller, Luca Milanesio, Repo and Gerrit Discussion

On 16 Apr 2020, at 16:05, Marco Miller <marco....@gmail.com> wrote:

-My internal team at Ericsson is currently working on the HA plugin ([1] below) to
1. sync project events from writing/active master to its passive companion, then
2. trigger the corresponding replication events towards replication plugin (on passive),
3. so that the replication plugin does that replication from passive master, offloading the active one.

Which HA plugin are you planning to use, [1] or multi-site [2], and from which Gerrit version first?


^^^ This is an HA solution


^^^ That is not an HA solution

They are both multi-master plugins but they have different goals: HA is for high-availability (hence the name) whilst multi-site is for reducing the latency across geo-locations.
We typically use both of them: HA (inside the same location) and multi-site (across geo-locations).

Luca.

/Marco

On Thursday, April 16, 2020 at 10:46:43 AM UTC-4, Tom Printy wrote:
Hello,

I am in the process of implementing a single site HA instance. Currently i have several Gerrit slaves located throughout the world. I am currently using the replication plug-in to keep these slaves up to date. 

My HA instance will have 3 Nodes at the master site (A,B,C) only one will be configured as the writter (accepting a push). Which node(s) should I run the replication plugin on?  My thought it one of the other nodes that is NOT serving as the writer (B or C).

What is the correct one?

Thanks
-Tom


-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/aaefc409-a8e5-4bc1-9167-ac9c237ceb3c%40googlegroups.com.

Marco Miller

unread,
Apr 16, 2020, 11:31:38 AM4/16/20
to Repo and Gerrit Discussion
-And Tom mentioned that he'd stick to a single site, so HA plugin [1] it would be.
You could try replicating from all master nodes, but I don't see why replicating from a non-writing one, unless to offload the writer.
Note that I'm not aware of a production HA configuration with more than 2 master nodes.

Multi-site [2] still has HA properties, as it extends high availability to multiple sites, in case of single-site disaster.
/Marco

Luca Milanesio

unread,
Apr 16, 2020, 11:36:47 AM4/16/20
to Marco Miller, Luca Milanesio, Repo and Gerrit Discussion

On 16 Apr 2020, at 16:31, Marco Miller <marco....@gmail.com> wrote:

-And Tom mentioned that he'd stick to a single site, so HA plugin [1] it would be.

Yep, that’s the best solution.

You could try replicating from all master nodes, but I don't see why replicating from a non-writing one, unless to offload the writer.

Not from all the master at the same time, but from “the active master” which could be one of the three.
E.g. You could serve different projects in write mode from different masters, to have a better distribution of the load.

HA is not “active-passive” but is “all active and one writer at a time”, but the policy is defined at HAProxy level and you can define whatever you want for improving the responsiveness and scalability of the system.

Luca;.

Note that I'm not aware of a production HA configuration with more than 2 master nodes.

Multi-site [2] still has HA properties, as it extends high availability to multiple sites, in case of single-site disaster.
/Marco

On Thursday, April 16, 2020 at 11:08:40 AM UTC-4, lucamilanesio wrote:
On 16 Apr 2020, at 16:05, Marco Miller <marco....@gmail.com> wrote:

-My internal team at Ericsson is currently working on the HA plugin ([1] below) to
1. sync project events from writing/active master to its passive companion, then
2. trigger the corresponding replication events towards replication plugin (on passive),
3. so that the replication plugin does that replication from passive master, offloading the active one.

Which HA plugin are you planning to use, [1] or multi-site [2], and from which Gerrit version first?

^^^ This is an HA solution
^^^ That is not an HA solution

They are both multi-master plugins but they have different goals: HA is for high-availability (hence the name) whilst multi-site is for reducing the latency across geo-locations.
We typically use both of them: HA (inside the same location) and multi-site (across geo-locations).

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nasser Grainawi

unread,
Apr 16, 2020, 1:47:13 PM4/16/20
to Marco Miller, Repo and Gerrit Discussion
On Apr 16, 2020, at 9:31 AM, Marco Miller <marco....@gmail.com> wrote:

-And Tom mentioned that he'd stick to a single site, so HA plugin [1] it would be.
You could try replicating from all master nodes, but I don't see why replicating from a non-writing one, unless to offload the writer.

We added the replication distributor feature for exactly that reason. We have 3 active masters and ~5 more passive masters (they’ll become active after we get past some infra limitations) and we want replication work shared amongst them all.

Note that I'm not aware of a production HA configuration with more than 2 master nodes.

We’re not using the HA plugin, but we have 3 active masters (on our 2.7 fork) and a total of almost 10 masters, all sharing the same NFS mount.


Multi-site [2] still has HA properties, as it extends high availability to multiple sites, in case of single-site disaster.
/Marco

On Thursday, April 16, 2020 at 11:08:40 AM UTC-4, lucamilanesio wrote:
On 16 Apr 2020, at 16:05, Marco Miller <marco....@gmail.com> wrote:

-My internal team at Ericsson is currently working on the HA plugin ([1] below) to
1. sync project events from writing/active master to its passive companion, then
2. trigger the corresponding replication events towards replication plugin (on passive),
3. so that the replication plugin does that replication from passive master, offloading the active one.

Which HA plugin are you planning to use, [1] or multi-site [2], and from which Gerrit version first?

^^^ This is an HA solution
^^^ That is not an HA solution

They are both multi-master plugins but they have different goals: HA is for high-availability (hence the name) whilst multi-site is for reducing the latency across geo-locations.
We typically use both of them: HA (inside the same location) and multi-site (across geo-locations).

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Tom Printy

unread,
Apr 16, 2020, 2:11:19 PM4/16/20
to Nasser Grainawi, Marco Miller, Repo and Gerrit Discussion
HI All

Thanks for the feedback it is much appreciated!

We are on 2.14 ( I know I know upgrade) so we can't use https://gerrit.googlesource.com/plugins/multi-site yet,  BUT we will get there.(Zebra Technologies). Are goal is to be running another multi-node master in India later this year, but need to get the equipment first.

I have configured the HA proxies (we have 3) to only have 1 writer. This will help use ease into the HA configuration.  This is inline with the Active Passive configuration here https://gerrit.googlesource.com/plugins/high-availability#active_passive-configuration

I like the simplicity of having the replication plugin running on all instances with the same configuration. I will go with this for now.

Does anyone have solution for watching the Gerrit Queue across multiple nodes?  A little dashboard would be nice. I am typically asked to look at the queues to see how long till a .git it replicated.

Has anyone experimented recently running a Mutl-Node master on Google cloud?  I know there were some disk speed issue but wondering if this has changed recently?

Thanks Everyone
-T






To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/49E2F439-BC38-4E98-AB3F-2C36A59DA970%40codeaurora.org.

Marco Miller

unread,
Apr 17, 2020, 8:52:03 AM4/17/20
to Repo and Gerrit Discussion
On Thursday, April 16, 2020 at 2:11:19 PM UTC-4, Tom Printy wrote:
HI All

Thanks for the feedback it is much appreciated!

We are on 2.14 ( I know I know upgrade) so we can't use https://gerrit.googlesource.com/plugins/multi-site yet,  BUT we will get there.(Zebra Technologies). Are goal is to be running another multi-node master in India later this year, but need to get the equipment first.

I have configured the HA proxies (we have 3) to only have 1 writer. This will help use ease into the HA configuration.  This is inline with the Active Passive configuration here https://gerrit.googlesource.com/plugins/high-availability#active_passive-configuration

I like the simplicity of having the replication plugin running on all instances with the same configuration. I will go with this for now.

Does anyone have solution for watching the Gerrit Queue across multiple nodes?  A little dashboard would be nice. I am typically asked to look at the queues to see how long till a .git it replicated.

-This would be another topic, but you can check [1,2] below and the related (detailed) topic [3].
I didn't check myself if it readily includes some queue monitoring support, but this would be the wanted framework for it.


Has anyone experimented recently running a Mutl-Node master on Google cloud?  I know there were some disk speed issue but wondering if this has changed recently?

Another topic, which you could re-post about as well.

Thanks Everyone
-T
Reply all
Reply to author
Forward
0 new messages