Gerrit High-Availability Setup

[Leigh Grealis]

Hi Group,

I have recently setup and pretty successfully tested the Gerrit HA plugin against a couple of Gerrit masters and with compiling the most recent versions of the HA plugin and a bit of tweaking, this looks fantastically promising and I thank everyone for the hard work that's gone into this.

Specifically I'd like to ask if anyone has attempted to setup more than two masters to use this plugin - or even if more than two is possible...?

I guess my apprehension is around this bit:

peerInfo.url

Specify the URL for the peer instance.

What I have at the moment is pretty easy

Server 1:

peerinfo.url = http://server2:8080/gerrit/

Server 2:

peerinfo.url = http://server1:8080/gerrit/

So - they point at each other.

What if I have three? Could it be as simple as round-robin (Server1->Server2, Server2->Server3, Server3->Server1)? Is there a sneaky option to have two peerinfo.url options perhaps?

We're investigating this for a pretty good reason - although we have some beefy servers, there's already around 10k users, 3k projects. Given a bit of growth, there's a point when one server dropping would switch a pretty big load to the other. I like the idea of having 3 at this kind of scale and upwards, afterall one of the goals is to stop scaling vertically and be able to horizontally scale.

Has anyone done this or have any insights? I know it's pretty easy to stick another VM into our test rig and try this out (or stop being lazy and look at the source code perhaps) but would value anyone being able to share any experience.

I'd be happy to do a bit of work on this and contribute back if no one has done this before.

Thanks,

Leigh

[Luca Milanesio]

Hi Leigh,

there is a reason behind the plugin-name: high-availability and not multi-master or high-scalability.

The plugin was designed by Ericsson to allow them to have a fully automated active failover system.

See their presentation at:

https://docs.google.com/file/d/0B5_-qRW_6quzaVVRdE9jZTB0TFU/edit?hl=en&forcehl=1

Basically the central node (master) has 2x Gerrit both running as masters using the high-availability plugin.

The HAProxy in front of them makes sure that if one fails, the second gets the traffic.

The indexing and caching sync performed by the high-availabilty plugin makes this possible.

Using high-availabilty plugin with more than 2 nodes it is possible I believe, but I am not sure how the usage can scale up even more or in an elastic way.

We need real multi-master for that, which isn't there yet.

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leigh Grealis]

Thanks Luca.

That's interesting. In the testing I've done, with HAproxy just doing simple round-robin of any incoming requests, it appears that both master nodes can effectively be actively serving responses consistently. Even stream-events seemed to be handled nicely from a Jenkins Gerrit Trigger plugin.

Am I missing something which would make this a dangerous usage pattern? My testing has been reasonably anecdotal rather than any serious testing of high-volume/load usage which I was planning to get to.

Should the plugin just be seen as a way to ensure fast fail-over rather than providing a more load-balanced type of configuration? This would still have obvious benefits but would change my direction for sure.

Kind Regards,
Leigh

[Luca Milanesio]

Hi Leigh,

see my feedback below.

On 17 Aug 2017, at 12:21, Leigh Grealis <lgre...@clearvision-cm.com> wrote:

Thanks Luca.

That's interesting. In the testing I've done, with HAproxy just doing simple round-robin of any incoming requests, it appears that both master nodes can effectively be actively serving responses consistently.

You haven't tested enough IMHO :-)

There is a scenario where two masters are managing two concurrent pushes to the same repo to the same master at the same time by two different users.

There isn't any transaction coordination between the two masters and no 2-phase-commit between the JGit repo and the DB: you would end up with one of the two users having his push failing silently.

It *could be* that the scenario I've described doesn't apply to your use-case: in that case it is perfectly fine for you to have both master available with a round-robin load balancing.

Even stream-events seemed to be handled nicely from a Jenkins Gerrit Trigger plugin.

Yes, high-availability manages the events propagation between the two masters.

Am I missing something which would make this a dangerous usage pattern? My testing has been reasonably anecdotal rather than any serious testing of high-volume/load usage which I was planning to get to.

See the scenario Ive described above.

Should the plugin just be seen as a way to ensure fast fail-over rather than providing a more load-balanced type of configuration?

Yes, exactly.

As you've realised, as the number of nodes increases, the traffic between them will increase significantly and the configurations (and restart) needed will make the configuration quite unmanageable.

This would still have obvious benefits but would change my direction for sure.

I believe the design of the plugin would need to change to support N (possibly high number) of nodes.

Possibly replacing the REST API invocation with an event bus (Kafka / RabbitMQ, other?)

       

Philadelphia, PA - USA: +1 (215) 278 8706

London / Southampton - UK: +44 (0) 845 459 9530

Dublin - Ireland: +353 1960 9597

Košice - Slovakia / Central Europe: +421 55 308 29 16

Bangalore - India: +91 (0) 804 901 6431

How was my service? Please give feedback.

Save paper – think before you print! This E-mail and its attachments are strictly confidential. If you are not the intended addressee you may not copy, forward, disclose or use any part of this email or its associated attachments (if present). If you have received this message in error, please contact the sender immediately and delete all copies from your system. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for delivery, security or virus integrity nor for any errors or omissions

[Martin Fick]

On Thursday, August 17, 2017 12:52:41 PM Luca Milanesio
wrote:

> Hi Leigh,
> see my feedback below.
>
> > On 17 Aug 2017, at 12:21, Leigh Grealis
> > <lgre...@clearvision-cm.com> wrote:
> >
> > Thanks Luca.
> >
> > That's interesting. In the testing I've done, with
> > HAproxy just doing simple round-robin of any incoming
> > requests, it appears that both master nodes can
> > effectively be actively serving responses consistently.
> You haven't tested enough IMHO :-)
> There is a scenario where two masters are managing two
> concurrent pushes to the same repo to the same master at
> the same time by two different users.
>
> There isn't any transaction coordination between the two
> masters and no 2-phase-commit between the JGit repo and
> the DB: you would end up with one of the two users having
> his push failing silently. It *could be* that the
> scenario I've described doesn't apply to your use-case:
> in that case it is perfectly fine for you to have both
> master available with a round-robin load balancing.

I don't believe this is accurate Luca. I believe this would
fail in the same way it would fail with two users performing
actions simulataenously on a single server. Afterall, the
single server is threaded.

I believe that the main concern left with multi-master is
consitency during ref-repacking over NFS. Please as Hugo
about this. We have not found any issues yet on our servers
with repacking on NFS, but we do something slightly different
(we always make a link to the packfile before repacking), so
perhaps that solves this issue?

FYI, we are about to go full multi-master with our 2.7
version of Gerrit. :)

Thanks,

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Luca Milanesio]

Not really, two threads on the same server have a JGit cache with exactly the same objects and can be internally synchronised inside the same JGit JVM.
Unless JGit isn't thread-safe, on the same JVM you don't have any issue with concurrency.

> Afterall, the
> single server is threaded.

Yep.

With multiple servers, you rely only on the file-system level lock, that's it.
If you have a fully tuned NFS with the associated locking mechanism that works like a charm ... then you *should* be reasonably fine, but again has to be fully tested.

The safest approach IMHO is giving some sharding logic to the front-end HAProxy service to make sure that two pushes on the same repo end up to the same node.
Starting from Gerrit master (2.15.x) that would be actually possible :-)

>
> I believe that the main concern left with multi-master is
> consitency during ref-repacking over NFS. Please as Hugo
> about this.

We did some tests as well on that, but it is a completely different issue GC-ing and repacking during production load.
Not necessarily associated with Multi-master, it is an issue as well for a single master.

> We have not found any issues yet on our servers
> with repacking on NFS, but we do something slightly different
> (we always make a link to the packfile before repacking), so
> perhaps that solves this issue?
>
> FYI, we are about to go full multi-master with our 2.7
> version of Gerrit. :)

Wow! With 2.7 I guess you did not have Lucene, so possibly that made your implementation a bit easier?

[Martin Fick]

On Thursday, August 17, 2017 03:56:52 PM Luca Milanesio
wrote:

Jgit is relying on this same lock. There is no object level
sychronization, only ref level. Objects in git are
immutable, they either exist or don't. Refs are mutable and
jgit uses file level locking to prevent the other threads in
the same process from updating the same ref at the same
time. There is no internal synchonization. jgit is
designed to operate just fine along with other git
implementations accessing the repo at the same time, even
from different servers.

> The safest approach IMHO is giving some sharding logic to
> the front-end HAProxy service to make sure that two
> pushes on the same repo end up to the same node. Starting
> from Gerrit master (2.15.x) that would be actually
> possible :-)
> > I believe that the main concern left with multi-master
> > is
> > consitency during ref-repacking over NFS. Please as
> > Hugo
> > about this.
>
> We did some tests as well on that, but it is a completely
> different issue GC-ing and repacking during production
> load. Not necessarily associated with Multi-master, it is
> an issue as well for a single master.

Agreed, I believe that is why Hugo's (Ericsson) team is only
doing gc from the same server. But it is also stopping him
from moving to MM.

> > We have not found any issues yet on our servers
> >
> > with repacking on NFS, but we do something slightly
> > different (we always make a link to the packfile before
> > repacking), so perhaps that solves this issue?
> >
> > FYI, we are about to go full multi-master with our 2.7
> > version of Gerrit. :)
>
> Wow! With 2.7 I guess you did not have Lucene, so possibly
> that made your implementation a bit easier?

Yes, we only needed to coordinate the events, which I have
done with our own plugin using just the filesystem (for now).
I preparing to push that soon.

[Luca Milanesio]

Need to dig into the latest JGit internals, I was pretty sure that refs were cached in memory and doesn't get them reloaded all the times.
Refs are packed as well, so they are equally subject to JGit cache I believe.

I've recently fixed some issues in JGit in-memory cache corruption, which was quite scary :-(
But again, needs to be throughly tested as the file-system based JGit mutli-master scenario would be heavily used for the first time in the world ... I believe by you :-)
I am not aware of anyone else using it active-active on the same repo from different / multiple legs pushing at the same time to the same branch.

Google's implementation is a DFS-JGit, a completely different beast :-O

[Martin Fick]

On Thursday, August 17, 2017 04:27:32 PM Luca Milanesio

> > synchronization. jgit is designed to operate just fine

> > along with other git implementations accessing the repo
> > at the same time, even from different servers.
>
> Need to dig into the latest JGit internals, I was pretty
> sure that refs were cached in memory and doesn't get them
> reloaded all the times. Refs are packed as well, so they
> are equally subject to JGit cache I believe.

I believe there is no caching for packed-refs either in
jgit, deliberately to prevent out-of-date (from the time the
read begins) read issues. Aside from deletes, I believe
that transactions do not modify packed-refs, only refpacking
does. I think this a large reason that the new reftable is
being worked on by Shawn.

However, NFS does cache depending on your configuration, and
that is why ref-packing can be an issue. I believe ref-
packing does not actually edit the file, I believe it
replaces it, so caching tends to not really be an issue
either. But replacing the files can potentially cause a
failed read on NFS (stale file handle, which should be
retried, and jgit probably still needs such a modification),
or an out-of-date one (after the read, which is generally
acceptable) on local storage. But these "should" not cause
any ref-updating issues (just the mentioned ref-packing
issue).

> I've recently fixed some issues in JGit in-memory cache
> corruption, which was quite scary :-( But again, needs to
> be throughly tested as the file-system based JGit
> mutli-master scenario would be heavily used for the first
> time in the world ... I believe by you :-) I am not aware
> of anyone else using it active-active on the same repo
> from different / multiple legs pushing at the same time
> to the same branch.

Generally speaking most Gerrit users rarely push to the same
branch, they let Gerrit do that by submitting changes.
However, ref-locking is safe here, one will fail, and not
silently!

> Google's implementation is a DFS-JGit, a completely
> different beast :-O

Yes, and no. :) Sounds like it is moving that way with the
new ref-table implementation. It believe it is currently
DFS/DDB. i.e. the refs are not stored in a DFS, they are
stored in a distributed DB.

[Luca Milanesio]

True, in that case ReviewDb is acting as coordination and transactional isolation.

Would it be worth to write a dedicated chapter about the HA-configurations and the current Multi-master capabilities?
Otherwise people need to get them distilled from discussions like this on the mailing list :-(

> However, ref-locking is safe here, one will fail, and not
> silently!

As you guys are moving into multi-master, I trust your word to have done all the necessary stress tests about it :-)
Which JGit version are you currently running on?

>
>> Google's implementation is a DFS-JGit, a completely
>> different beast :-O
>
> Yes, and no. :) Sounds like it is moving that way with the
> new ref-table implementation. It believe it is currently
> DFS/DDB. i.e. the refs are not stored in a DFS, they are
> stored in a distributed DB.

The current implementation is Yes, the future is No :-)
Hopefully the Git community will accept Shawn ref-table proposal !!!

Luca.

[Martin Fick]

On Thursday, August 17, 2017 05:17:01 PM Luca Milanesio
wrote:

> Would it be worth to write a dedicated chapter about the
> HA-configurations and the current Multi-master
> capabilities? Otherwise people need to get them distilled
> from discussions like this on the mailing list :-(

This was the starting intent of the multi-master "plugin".
It is entirely a documentation project so far. It describes
some of the things needed to setup a multi-master. I
believe it is a good place to describe all things related to
multi-master. We also used to have a web-page describing MM
stuff, and it would be good to get any info missing from
there into the MM plugin docs.

The intent in the doc plugin can be to discuss all MM issues
and offer solutions, whether the solution be use a specific HA
plugin, configure a load balancer, use RR DNS, or there
currently is no implementation but the solution needs to do
blah blah blah... There are several solutions to some of
the issues, including multiple plugins.

I envision the MM being a place where you might add any glue
that you might need between these various solutions, perhaps
it becomes the place, or even just the interfaces to define
your cluster?

> As you guys are moving into multi-master, I trust your
> word to have done all the necessary stress tests about it
> :-) Which JGit version are you currently running on?

We have tested some use cases. For stress testing we have
been operating many gerrit slaves with concurrent writting
to NFS for years. This is no one-to-one to a gerrit
master, but it stresses most of the the same code paths.
I'll let you know the problems we run into. :) We have
been running active/hot for a while now, and we sometimes
run some connections on the hot node. We will be slowly
bringing in traffic to the hot node soon, and montoring for
issues,

[Rainbow]

[Luca Milanesio]

On 17 Aug 2017, at 21:31, Martin Fick <mf...@codeaurora.org> wrote:

On Thursday, August 17, 2017 05:17:01 PM Luca Milanesio
wrote:

Would it be worth to write a dedicated chapter about the
HA-configurations and the current Multi-master
capabilities? Otherwise people need to get them distilled
from discussions like this on the mailing list :-(

This was the starting intent of the multi-master "plugin".  
It is entirely a documentation project so far.  It describes
some of the things needed to setup a multi-master.  I
believe it is a good place to describe all things related to
multi-master.  We also used to have a web-page describing MM
stuff, and it would be good to get any info missing from
there into the MM plugin docs.

Are you referring to:

https://gerrit.googlesource.com/plugins/multi-master/+/master/src/main/resources/Documentation/config.md ?

I believe it is a good base for documentation, however, would be nice to have a sort of "index of alternatives for multi-master" in the main Gerrit documentation set or at least the web-site.

For instance, in your documentation there is no mention of the high-availability plugin solution to the cache coherency.

Additionally, doesn't talk about Lucene-index related problems and options.

The intent in the doc plugin can be to discuss all MM issues
and offer solutions, whether the solution be use a specific HA
plugin, configure a load balancer, use RR DNS, or there
currently is no implementation but the solution needs to do
blah blah blah...  There are several solutions to some of
the issues, including multiple plugins.  

Yep, exactly. And there is no "one size fits all" solution, it depends on the scenarios.

I envision the MM being a place where you might add any glue
that you might need between these various solutions, perhaps
it becomes the place, or even just the interfaces to define
your cluster?

As you guys are moving into multi-master, I trust your
word to have done all the necessary stress tests about it
:-) Which JGit version are you currently running on?

We have tested some use cases.  For stress testing we have
been operating many gerrit slaves with concurrent writting
to NFS for years.   This is no one-to-one to a gerrit
master, but it stresses most of the the same code paths.  
I'll let you know the problems we run into. :)   We have
been running active/hot for a while now, and we sometimes
run some connections on the hot node.  We will be slowly
bringing in traffic to the hot node soon, and montoring for
issues,

It would be great if you guys talk about your mult-master experiences at the next forthcoming Gerrit User Summit 2017.

Are you planning to attend? to present?

[Martin Fick]

On Friday, August 18, 2017 08:40:56 AM Luca Milanesio wrote:
> > On 17 Aug 2017, at 21:31, Martin Fick
> > <mf...@codeaurora.org> wrote:
> >
> > On Thursday, August 17, 2017 05:17:01 PM Luca Milanesio
> >
> > wrote:
> >> Would it be worth to write a dedicated chapter about
> >> the
> >> HA-configurations and the current Multi-master
> >> capabilities? Otherwise people need to get them
> >> distilled
> >> from discussions like this on the mailing list :-(
> >
> > This was the starting intent of the multi-master
> > "plugin". It is entirely a documentation project so
> > far. It describes some of the things needed to setup a
> > multi-master. I believe it is a good place to describe
> > all things related to multi-master. We also used to
> > have a web-page describing MM stuff, and it would be
> > good to get any info missing from there into the MM
> > plugin docs.
>
> Are you referring to:
> https://gerrit.googlesource.com/plugins/multi-master/+/mas
> ter/src/main/resources/Documentation/config.md

> <https://gerrit.googlesource.com/plugins/multi-master/+/m
> aster/src/main/resources/Documentation/config.md> ?

Yes.

> I believe it is a good base for documentation, however,
> would be nice to have a sort of "index of alternatives
> for multi-master" in the main Gerrit documentation set or
> at least the web-site.

There's this:

https://gerrit.googlesource.com/homepage/+/md-
pages/docs/MultiMaster.md

That info should probably get included in the MM plugin. I
suggest the the MM plugin doc be the main index for all the
other MM related plugins.

> For instance, in your documentation there is no mention of
> the high-availability plugin solution to the cache
> coherency.

Yes, I don't know enough about those plugins to describe
them and the problems they solve. I do think they would be
great additions to the doc.

I don't see the websessions-flat-file plugin mentioned there,
I will work on adding that to the doc.

> Additionally, doesn't talk about Lucene-index
> related problems and options.

Yes, nor any plugins that help with this. As we don't have
an index (2.7), I think it would be better for someone else
to add info about this. I might upload a change that at
least mentions it as an issue.

> It would be great if you guys talk about your mult-master
> experiences at the next forthcoming Gerrit User Summit
> 2017. Are you planning to attend? to present?

That sounds like a good idea. I will try to pitch that to
management as a reason to attend! ... I hope I can,

[Luca Milanesio]

Cool, keeping finger crossed and hope to see you soon in London !

Luca.

[Saša Živkov]

Actually, from my last talk with Hugo I remember that the main (and only?) issue

is the delay in the Lucene indexing between the two nodes.

When a change C is pushed to node A, then it is indexed on A and the indexing

event is then propagated to the node B.

However, the same client (knowing about the existence of the change C) may

immediately query for the change C and this may happen before the indexing

has been propagated/finished on B. If HAProxy would forward this change query

to the node B, the result set would be empty, which is wrong.

If you can tolerate these short periods of change query inconsistency then you likely

can use HA plugin and configure HAProxy to do round-robin load balancing.

 

about this.  We have not found any issues yet on our servers
with repacking on NFS, but we do something slightly different
(we always make a link to the packfile before repacking), so
perhaps that solves this issue? 

FYI, we are about to go full multi-master with our 2.7
version of Gerrit. :)

Thanks,

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Luca Milanesio]

Or if the push operation could hold until the indexes are replicated, the problem would be 100% resolved, isn't it?

If you can tolerate these short periods of change query inconsistency then you likely

can use HA plugin and configure HAProxy to do round-robin load balancing.

It depends on how "close" are the two hosts.

They *must* be close because otherwise the shared storage of the Git repos would be unusable ... there are situation though where the Git repos are "virtually shared" (e.g. Cassandra JGit DFS backend) and the index consistency sync could take a few seconds.

[Matthias Sohn]

If the load balancer implements sticky sessions this could help to avoid this

potential issue as long as the client uses the same session for subsequent requests.

[Hugo Arès]

Sorry for the late response, I was away for the last 3 weeks.

HA plugin is designed to be used in active-passive setup. Some changes are required for it to be used in a active-active cluster.

First of all, cache evictions, indexing events and stream events are forwarded to the other server asynchronously, they need to be done synchronously for the plugin to be used in an active-active setup to make sure the subsequent requests don't return stale info if they are handled by the other server. Sticky sessions could help but they do not solve the problem.

As Luca mentioned in a previous response, changing the transport layer would be a good idea, REST API calls works fine for active-passive but maybe it can become an issue in an active-active setup.

Another thing is secondary indexes are duplicated in the current implementation of the high-availability plugin. This is good enough for active-passive but maybe not for active-active. Our plan is to use Elasticsearch as the secondary index implementation and stop forwarding the indexing events when going to active-active setup.

For the number of nodes, the plugin could be changed to support n number of nodes but it would probably wiser to use a messaging system between the nodes instead of REST API calls.

That said, forwarding the events synchronously in the only change you need to do for sure to use the plugin with 2 active nodes.This could be a configuration option.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[lucamilanesio]

Thanks to Martin to have presented the Multi-Master setup of Gerrit Code Review at Qualcomm at the Gerrit User Summit 2017 in London.

I really liked the interactive talk and the discussion with Hugo during the presentation.

YouTube video and the full transcript is now available at:

https://goo.gl/hAZgfm

A beautiful Christmas present and "homework" for all of the Gerrit admins who want to dive into multi-master in 2018 !

Thanks again Martin and Hugo !

Luca.