Gerrit HA with events-log plugin

[Leigh Grealis]

Hi Group,

We've been experimenting with the HA setup further (in active/hot-standby configuration). We're looking at a variant of the shared database model because there is no team that provides HA for the database so each node has it's own postgres database with streaming replication running from master->standby.

I was anticipating that there would be no issue on the standby as Gerrit would be started up with the database in read-only mode. I didn't bank on the standby server needing write access to the database because of the events-log plugin which wants to write as soon as it starts and then eventually falls back to local h2 database.

There's a possible option I see - start up Gerrit standby without the plugin and at the point of failover reload the plugin with "allowRemoteAdmin = true". This could be orchestrated pretty efficiently I guess but there might be some events not replicated through postgres during the transition. I could possibly also use the h2 database for the events log and put that on the shared storage so both instances have access to it in the same way as the repo data - but that feels a bit non-production with h2.

Has anyone else used this combination and come up with anything or see any nicer solutions?

Additionally, is anyone using events-log with h2 database on a production system? If so, what sort of events volume do you have?

Thanks
Leigh

[Hugo Arès]

On Tuesday, December 5, 2017 at 4:05:31 PM UTC-5, Leigh Grealis wrote:

Hi Group,

We've been experimenting with the HA setup further (in active/hot-standby configuration). We're looking at a variant of the shared database model because there is no team that provides HA for the database so each node has it's own postgres database with streaming replication running from master->standby.

The idea behind the HA plugin is to make sure the standby server is always ready to serve traffic. If your standby server have a ready only database, then it's not ready to server traffic until you reverse the master database and the streamed one which will cause a gap and request will be dropped. Typically, the reverse proxy (e.g. HAProxy) will redirect the traffic from primary Gerrit to standby when it detects that primary is unhealthy. This does not mean that primary is dead so all new incoming requests are handled by standby while primary is just finishing ongoing requests. Your master db with readonly db setup won't work because both Gerrit maters need to write in the db during the transition.
 

I was anticipating that there would be no issue on the standby as Gerrit would be started up with the database in read-only mode. I didn't bank on the standby server needing write access to the database because of the events-log plugin which wants to write as soon as it starts and then eventually falls back to local h2 database.

There's a possible option I see - start up Gerrit standby without the plugin and at the point of failover reload the plugin with "allowRemoteAdmin = true". This could be orchestrated pretty efficiently I guess but there might be some events not replicated through postgres during the transition. I could possibly also use the h2 database for the events log and put that on the shared storage so both instances have access to it in the same way as the repo data - but that feels a bit non-production with h2.

 

Why don't you use the same database server for both Gerrit masters instead?  There are 2 things going on here, HA for gerrit and HA for the database server. I think from Gerrit masters perspective, there should be only "one" database  and you should find a ha setup for you database server.

 

Has anyone else used this combination and come up with anything or see any nicer solutions?

On our side, we have one master database server which we stream to a backup database server. Gerrit masters connect to the database through HAProxy, master database is the default and backup is the backup in the HAProxy configuration. In case the master db dies, all we have to do is to restart the backup db server as a primary database and Gerrit master cluster will be working without restart them, thanks to HAProxy.

 

Additionally, is anyone using events-log with h2 database on a production system? If so, what sort of events volume do you have?

Embedded H2 databases should never be used in production, it can one execute one request at the time which can become a major bottleneck.
 

Thanks
Leigh

[luca.mi...@gmail.com]

Apart from the Guava caches that have been on H2 for a very long time :-)

In Gerrit 2.15 I would recommend using H2 for the internal groups once everything else is on NoteDb

 

Thanks
Leigh

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.