Gerrit Multi Master

211 views
Skip to first unread message

Sassy Natan

unread,
Aug 30, 2021, 2:27:18 AM8/30/21
to Repo and Gerrit Discussion
Hi Group,

Thanks for this amazing tool.

I was assign to a task to upgrade Gerrit to a new version.

I have read the https://gerrit.googlesource.com/plugins/high-availability/ document and also went on some of the links how to setup the env.

Most of the info is clear to me with the exception of active RW/active RW.

To my understating the HAProxy is the one who split the work within the servers, so RO go to all available servers using round robin, while RW point to the active server based in the HTTP Check.


How is Multi RW are possible?  How they overcome the CAP issue? 
Is the RW means to have multi RW server, but each server hold a specific Repo? 

So I have server X, Y and Z with repo names: 1,2,3,4,5,6. Where Repo 1,2 are located at X, 3,4 are located at Y and 5,6 are located at Z?

Can anyone explain more about this? 

Thank You



mac...@gmail.com

unread,
Aug 30, 2021, 6:37:54 AM8/30/21
to Repo and Gerrit Discussion
Hi,
Please find my answers below:

How is Multi RW are possible?  How they overcome the CAP issue? 
We are aiming for eventual consistency. This means that at some point all nodes are going to be synchronised. So the next question will be how to avoid split-brain[1] situation(some nodes behind or multiple writes to the same repo at the same time).
For that we are using global-refdb(currently we have two implementations for zookeeper[2] or dynamo db[3]). Every time Gerrit updates the ref first is checking if the local  sha1 for that ref is equal with the sha1 in global-refdb if yes it will create a distributed lock update local ref and the global-refdb. If local ref sha1 is not equal to the global-refdb this means that the node is behind and ref update will be rejected. 
In this approach all nodes can serve RW traffic for all repositories.

Is the RW means to have multi RW server, but each server hold a specific Repo? 
This is also a valid approach if you can evenly distribute repos across nodes.  What we often see is that some repos are very active and some less in that case some of the nodes will have to handle more traffic than other. With this approach you don't need to synchronise state with global-refdb because only one node is serving write traffic for a particular repo.

You are referring to high-availability plugin but if you want to distribute traffic across multiple sites(for example data centres) you should look at multi-site plugin.

Sassy Natan

unread,
Dec 9, 2021, 10:56:03 AM12/9/21
to mac...@gmail.com, Repo and Gerrit Discussion
Thank You very much.

From which version gerrit supports multi master topology?
If I enable the global-refdb (which require Gerrit v3.2 or later) along with the HA Plugin can I set up in the HAProxy something like this:

  backend http-servers
    server server1 <server1_ip>:<server1_http_port>
    server server2 <server2_ip>:<server2_http_port> (NO BACKUP HERE!!!!!)

With this topology both servers can get POST data and not only GET requests.


In my setup the gerrit hits many clients and we want to scale it with a multi master on the same data center.
We have a working HA Setup, where we have 3 RO slaves and one RW master.

We still use version 2.16 - but I plan to move to 3.4, but for the meanwhile my question focuses on whether we can enable Multi master with version 2.16.
If not, does having the global-refdb + HA plugin be enough  to get what is needed?

Thanks for such a great and amazing  tool! 




Sassy


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/1IK_HGjbdR8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f5084230-6e24-426b-9d4d-d018497625e1n%40googlegroups.com.


--
Regards,

Sassy Natan
972-(0)54-2203702

Luca Milanesio

unread,
Dec 9, 2021, 11:48:25 AM12/9/21
to Repo and Gerrit Discussion, Luca Milanesio, Marcin Czech, Sassy Natan
Hi Sassy,
Gerrit supports HA setup since v2.13, thanks to the high-availability plugin (see [1]).

On 9 Dec 2021, at 15:55, Sassy Natan <sas...@gmail.com> wrote:

Thank You very much.

From which version gerrit supports multi master topology?
If I enable the global-refdb (which require Gerrit v3.2 or later) along with the HA Plugin can I set up in the HAProxy something like this:

  backend http-servers
    server server1 <server1_ip>:<server1_http_port>
    server server2 <server2_ip>:<server2_http_port> (NO BACKUP HERE!!!!!)

With this topology both servers can get POST data and not only GET requests.

For having all nodes active in RW, you need to have Gerrit v3.3 or later and use a global-refdb.
The above repository ^^^^^^^^^^^^^^^ is 8 years old and largely obsolete.


In my setup the gerrit hits many clients and we want to scale it with a multi master on the same data center.
We have a working HA Setup, where we have 3 RO slaves and one RW master.

We still use version 2.16 - but I plan to move to 3.4, but for the meanwhile my question focuses on whether we can enable Multi master with version 2.16.

See [1].

If not, does having the global-refdb + HA plugin be enough  to get what is needed?

You need v3.3 or later for having a global-refdb.


Thanks for such a great and amazing  tool! 

You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CACHQFzGYL5%2BWKk1Wx9%2B2uRiTCCQ%2BqA5ESgsa5U%3DcFR-xaderNw%40mail.gmail.com.

Sassy Natan

unread,
Dec 9, 2021, 2:02:19 PM12/9/21
to Luca Milanesio, Marcin Czech, Repo and Gerrit Discussion
Awesome, thanks!
Reply all
Reply to author
Forward
0 new messages