Replication.config for Gerrit 2.16 and NoteDB enabled

[Doug Luedtke]

We have over a dozen mirrors. I think I have too many refs configured to go to the mirrors. I'm trying to reduce the Gerrit task queue that seems to have more than 100 replication tasks queued throughout the day.

The replication documentation has a part for replication of account data (NoteDb), but it doesn't mention the groups or the basic refs to include. I've added some to use a mirror for a NoteDB backup also. I now want to minimize the number of refs to other mirrors. Any input would be greatly appreciated.

We do git tunneling over ssh to save up to 50% of the transport time around the world. The tunneling script runs a autossh command for a list of mirrors.

Example section in the replication config:

[remote "gerrit-mirror02"]
  url = git://localhost:port/gerrit-mirror/repositories/${name}.git
  adminUrl = appUsername@box-name.company.com:/path/to/gerrit-mirror/repositories/${name}.git
  push = +refs/heads/*:refs/heads/*
  push = +refs/tags/*:refs/tags/*
  push = +refs/changes/*:refs/changes/*
  push = +refs/notes/*:refs/notes/*
  push = +refs/meta/*:refs/meta/*
  push = +refs/users/*:refs/users/*
  push = +refs/starred-changes/*:refs/starred-changes/*
  push = +refs/sequences/*:refs/sequences/*
  push = +refs/groups/*:refs/groups/*
  mirror = true
  threads = 5

Which refs can I safely remove and allow users to still clone?

Should I have All-Users as a separate section in the replication config?

Each of the 15 mirrors has its own section in the replication config.

Also, we only allow SSH for cloning from the mirrors.

[Luca Milanesio]

On 22 Oct 2019, at 20:06, Doug Luedtke <douglas...@gmail.com> wrote:

We have over a dozen mirrors. I think I have too many refs configured to go to the mirrors. I'm trying to reduce the Gerrit task queue that seems to have more than 100 replication tasks queued throughout the day.

How may active repos do you have? How many refs in total per repo?

What are you using the replicas for? CI builds? DR-sites?

The replication documentation has a part for replication of account data (NoteDb), but it doesn't mention the groups or the basic refs to include. I've added some to use a mirror for a NoteDB backup also. I now want to minimize the number of refs to other mirrors. Any input would be greatly appreciated.

We do git tunneling over ssh to save up to 50% of the transport time around the world. The tunneling script runs a autossh command for a list of mirrors.

The most effective protocol for replication would be Git/HTTPS, however, the Git protocol over SSH Tunnels is the 2nd best choice.

Example section in the replication config:

[remote "gerrit-mirror02"]
  url = git://localhost:port/gerrit-mirror/repositories/${name}.git
  adminUrl = appUsername@box-name.company.com:/path/to/gerrit-mirror/repositories/${name}.git
  push = +refs/heads/*:refs/heads/*
  push = +refs/tags/*:refs/tags/*
  push = +refs/changes/*:refs/changes/*
  push = +refs/notes/*:refs/notes/*
  push = +refs/meta/*:refs/meta/*
  push = +refs/users/*:refs/users/*
  push = +refs/starred-changes/*:refs/starred-changes/*
  push = +refs/sequences/*:refs/sequences/*
  push = +refs/groups/*:refs/groups/*
  mirror = true
  threads = 5

Which refs can I safely remove and allow users to still clone?

Who is service the Git protocol on the replicas? A simple Git server? A Gerrit slave?

Should I have All-Users as a separate section in the replication config?

I believe so, they are a sort of "unusual repos". If you do not have Gerrit slaves service the Git protocol on the mirrors, you may not need them at all.

Each of the 15 mirrors has its own section in the replication config.

Also, we only allow SSH for cloning from the mirrors.

-- 

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d2026493-f0f8-4cc4-a6dd-8e9355a23ece%40googlegroups.com.

[mf...@codeaurora.org]

Depending on the amount of ref-updates you are experiencing, having 15 mirrors could be more than you can handle with your current setup (we can only handle about 5 destinations). Do you find that you are WAN limited? Are all 15 mirrors at different sites with different WANs? We avoid having more than one replication entry per site and force the mirrors on each site to share the back end so as to spare the WAN and master loads.

If you already make good use of your WAN, and the master is your limit, then you may want to consider the improvements being proposed to make replication scale better for more destinations by using more masters, see here: https://gerrit-review.googlesource.com/c/homepage/+/239032/8 I suspect the proposed solution would work even if you added masters just for replication,

-Martin

--

[Doug Luedtke]

On Tuesday, October 22, 2019 at 3:03:59 PM UTC-5, lucamilanesio wrote:

On 22 Oct 2019, at 20:06, Doug Luedtke <douglas...@gmail.com> wrote:

We have over a dozen mirrors. I think I have too many refs configured to go to the mirrors. I'm trying to reduce the Gerrit task queue that seems to have more than 100 replication tasks queued throughout the day.

How may active repos do you have? How many refs in total per repo?

I'll have to get back to you on that. My sshd_log and httpd_log are not populating for some reason in production. They worked in the test environments. All worked in 2.14.18. Added the flogger lines via the init steps.

 

What are you using the replicas for? CI builds? DR-sites?

11 of the mirrors are in offices around the world. They are for users to clone locally, much quicker than going across the ocean, and also to save WAN bandwidth for the remote offices. I think three of those offices around the world do use their local mirror for their build farms. I didn't spec those mirrors for build nodes, but they are working for them. 4 of the mirrors are here for build nodes to use. 

The replication documentation has a part for replication of account data (NoteDb), but it doesn't mention the groups or the basic refs to include. I've added some to use a mirror for a NoteDB backup also. I now want to minimize the number of refs to other mirrors. Any input would be greatly appreciated.

We do git tunneling over ssh to save up to 50% of the transport time around the world. The tunneling script runs a autossh command for a list of mirrors.

The most effective protocol for replication would be Git/HTTPS, however, the Git protocol over SSH Tunnels is the 2nd best choice.

Example section in the replication config:

[remote "gerrit-mirror02"]
  url = git://localhost:port/gerrit-mirror/repositories/${name}.git
  adminUrl = appUsername@box-name.company.com:/path/to/gerrit-mirror/repositories/${name}.git
  push = +refs/heads/*:refs/heads/*
  push = +refs/tags/*:refs/tags/*
  push = +refs/changes/*:refs/changes/*
  push = +refs/notes/*:refs/notes/*
  push = +refs/meta/*:refs/meta/*
  push = +refs/users/*:refs/users/*
  push = +refs/starred-changes/*:refs/starred-changes/*
  push = +refs/sequences/*:refs/sequences/*
  push = +refs/groups/*:refs/groups/*
  mirror = true
  threads = 5

Which refs can I safely remove and allow users to still clone?

Who is service the Git protocol on the replicas? A simple Git server? A Gerrit slave?

The mirrors are Gerrit slaves. We only allow cloning via ssh. That is how things were setup 10 years ago, and I have not had a reason to change it.

Should I have All-Users as a separate section in the replication config?

I believe so, they are a sort of "unusual repos". If you do not have Gerrit slaves service the Git protocol on the mirrors, you may not need them at all.

Each of the 15 mirrors has its own section in the replication config.

Also, we only allow SSH for cloning from the mirrors.

-- 

--
To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Doug Luedtke]

On Tuesday, October 22, 2019 at 4:14:32 PM UTC-5, MartinFick wrote:

Depending on the amount of ref-updates you are experiencing, having 15 mirrors could be more than you can handle with your current setup (we can only handle about 5 destinations). Do you find that you are WAN limited? Are all 15 mirrors at different sites with different WANs? We avoid having more than one replication entry per site and force the mirrors on each site to share the back end so as to spare the WAN and master loads.

My location is not WAN limited. Offices are WAN limited. That is why we setup mirrors years ago. Everything stays inside the firewall across the world between offices.

Each site has it's own replication entry in the replication.config. 

 

If you already make good use of your WAN, and the master is your limit, then you may want to consider the improvements being proposed to make replication scale better for more destinations by using more masters, see here: https://gerrit-review.googlesource.com/c/homepage/+/239032/8 I suspect the proposed solution would work even if you added masters just for replication,

I'm wanting to go to multi-master multi-site in the future just for the zero downtime upgrades. I am just not ready for that step today.

 

-Martin

[Doug Luedtke]

Example replication.log entries.

Replication tasks are getting rescheduled. There seems to be a schedule for refs/changes/XX/XXXXXX/# and then another one for refs/changes/XX/XXXXXX/meta. Should each replication job be creating two pushes to each mirror? They didn't in 2.14 when comparing logs.

[2019-11-01 09:09:28,133] [] scheduling replication path/to/repository:refs/changes/55/852555/6 => git://localhost:9432/big-instance-mirror/repositories/path/to/repository.git
[2019-11-01 09:09:28,169] [] scheduled path/to/repository:refs/changes/55/852555/meta => [95beb892] push git://localhost:9432/big-instance-mirror/repositories/path/to/repository.git to run after 15s
[2019-11-01 09:09:40,690] [b6bad426] Rescheduling replication to git://localhost:9424/path/to/another/repository.git to avoid collision with the in-flight push [8cc53412].
[2019-11-01 09:09:43,135] [95beb892] Replication to git://localhost:9432/big-instance-mirror/repositories/path/to/repository.git started...
[2019-11-01 09:09:43,135] [95beb892] Push to git://localhost:9432/big-instance-mirror/repositories/path/to/repository.git references: [RemoteRefUpdate[remoteName=refs/changes/55/852555/6, NOT_ATTEMPTED, (null)...SHA1------------------------------------, srcRef=refs/changes/55/852555/6, forceUpdate, message=null], RemoteRefUpdate[remoteName=refs/changes/55/852555/meta, NOT_ATTEMPTED, (null)...SHA1-Another----------------------------, srcRef=refs/changes/55/852555/meta, forceUpdate, message=null]]
[2019-11-01 09:09:43,263] [95beb892] Replication to git://localhost:9432/big-instance-mirror/repositories/path/to/repository.git completed in 128ms, 15001ms delay, 0 retries

[mf...@codeaurora.org]

I think that yes, with notedb each change upload now has at least two refs to replicate instead of one (I'm not sure if that's the case for each patchset?). These events should get consolidated into a single push assuming that you have a long enough replication delay, but it is a bit more data.

However now every comment, every vote, and every addition of reviewer or ccer will also result in a relationship event. Effectively every state change since there is no DB anymore. I suspect this includes every edit to unpublished comments, every "resolved" comment toggle, etc.. If you have a CI system that marks every patchset, this can add up quickly. I would expect anywhere from 5-20x more replication events with Notedb. While these are likely small events data wise, much of the overhead is the same as with larger data events. Maybe someone using NoteDb can post some data on how many more replication events they see now?

Perhaps you are just experiencing this extra event workload?

-Martin

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ecd2140d-322e-405b-b0d2-5db5f1d94e83%40googlegroups.com.

[Luca Milanesio]

On 2 Nov 2019, at 01:19, mf...@codeaurora.org wrote:

I think that yes, with notedb each change upload now has at least two refs to replicate instead of one (I'm not sure if that's the case for each patchset?). These events should get consolidated into a single push assuming that you have a long enough replication delay, but it is a bit more data.

However, even if the NoteDb data is small to replicate, the ref-advertisement phase for a repo with lots of refs can be significant.

We see for repos with large number of refs that even a single NoteDb update takes no less than 1-2 mins.

However now every comment, every vote, and every addition of reviewer or ccer will also result in a relationship event. Effectively every state change since there is no DB anymore. I suspect this includes every edit to unpublished comments, every "resolved" comment toggle, etc.. If you have a CI system that marks every patchset, this can add up quickly. I would expect anywhere from 5-20x more replication events with Notedb. While these are likely small events data wise, much of the overhead is the same as with larger data events. Maybe someone using NoteDb can post some data on how many more replication events they see now?

Yes, we can collect some data and share some graphs of the pre/post NoteDb migration.

Will get some graphs out and share.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d626caad-4c1d-4843-8fb6-793aeb347f12%40email.android.com.

[Doug Luedtke]

Thank you, Martin and Luca.

I've since tried adding a delay of 60 seconds and I'm not noticing a change in the task queue. We are single master and many mirrors, are there any branches I can eliminate from being replicated? If a branch is not replicated, will the replication still try to replicate the unchanged portion of the repository? For testing the replicationDelay = 60, I only made that change to one of the 15 mirrors.

[remote "gerrit-mirror02"]
  url = git://localhost:port/gerrit-mirror/repositories/${name}.git
  adminUrl = appUsername@box-name.company.com:/path/to/gerrit-mirror/repositories/${name}.git
  push = +refs/heads

/*:refs/heads/*

  push = +refs/tags/*:refs/tags/*
  push = +refs/changes/*:refs/changes/*
  push = +refs/notes/*:refs/notes/*
  push = +refs/meta/*:refs/meta/*
  push = +refs/users/*:refs/users/*
  push = +refs/starred-changes/*:refs/starred-changes/*
  push = +refs/sequences/*:refs/sequences/*
  push = +refs/groups/*:refs/groups/*

  replicationDelay = 60

[Luca Milanesio]

See inline some more refs that you *don't need* on the slaves (commented with *NOT NEEDED*)

On 5 Nov 2019, at 19:00, Doug Luedtke <douglas...@gmail.com> wrote:

Thank you, Martin and Luca.

I've since tried adding a delay of 60 seconds and I'm not noticing a change in the task queue. We are single master and many mirrors, are there any branches I can eliminate from being replicated? If a branch is not replicated, will the replication still try to replicate the unchanged portion of the repository? For testing the replicationDelay = 60, I only made that change to one of the 15 mirrors.

[remote "gerrit-mirror02"]
  url = git://localhost:port/gerrit-mirror/repositories/${name}.git
  adminUrl = appUsername@box-name.company.com:/path/to/gerrit-mirror/repositories/${name}.git
  push = +refs/heads

/*:refs/heads/*

  push = +refs/tags/*:refs/tags/*
  push = +refs/changes/*:refs/changes/*
  push = +refs/notes/*:refs/notes/*
  push = +refs/meta/*:refs/meta/*
  push = +refs/users/*:refs/users/*

  push = +refs/starred-changes/*:refs/starred-changes/* (*NOT NEEDED*)
  push = +refs/sequences/*:refs/sequences/* (*NOT NEEDED*)

  push = +refs/groups/*:refs/groups/*
  replicationDelay = 60
  mirror = true
  threads = 5

P.S. you should get some statistics from your replication log and start doing some profiling of the slowest ones. P.S. Do you regularly GC the repos on the slaves also?

HTH

Luca.

[Doug Luedtke]

On Tuesday, November 5, 2019 at 1:04:57 PM UTC-6, lucamilanesio wrote:

P.S. you should get some statistics from your replication log and start doing some profiling of the slowest ones. P.S. Do you regularly GC the repos on the slaves also?

The slow one are busy or large. Busy as in multiple updates a minute and large as in 2G to 5G in size.

We do gerrit gc the mirrors twice a day. Currently the mirrors are fast for cloning. These are a pool of 4 hardware boxes that match the largest master specs. Load balanced behind HA-Proxy. This was a solution we put in place over a year ago to keep up with Jenkins demand and their growth of 40% YoY.

[Doug Luedtke]

Which branches contain the comments, votes, reviewers, and non-essential for source code cloning? What if I eliminated those for the mirrors? Is there documentation about the Gerrit created branches?

[Luca Milanesio]

On 5 Nov 2019, at 21:36, Doug Luedtke <douglas...@gmail.com> wrote:

Which branches contain the comments, votes, reviewers, and non-essential for source code cloning?

The 'changes/nn/nnnnn/meta' refs. Unfortunately, you can't exclude them easily.

I am planning to extend the refs-filter plugin (see [1]) to allow that: from the first trials, it would speedup the replication by a factor of 2x (before NoteDb) and even further (after NoteDb).

Starting from Gerrit v3.1, thanks to the Git Protocol v2 support, having so many refs won't be a slowdown anymore.

Luca.

[1] https://gerrit.googlesource.com/modules/git-refs-filter/

What if I eliminated those for the mirrors? Is there documentation about the Gerrit created branches?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/174d4c92-8dfb-489e-b0aa-8296bd8cd6c0%40googlegroups.com.

[DmitryB]

Starting from Gerrit v3.1, thanks to the Git Protocol v2 support, having so many refs won't be a slowdown anymore.

Luca.

Does v3.1 support refs/changes/* replication? https://gerrit-review.googlesource.com/c/gerrit/+/230071 

[Luca Milanesio]

The above change is unrelated: it is about accepting pushes of refs/changes/* not preventing them to be replicated.

Hope this clarifies.

Luca.

[Carbon Robin]

Luca,

You said refs/starred-changes/* is (* NOT NEEDED *), but the replication plugin documentation shows replication of NoteDB account data requires that branch from the All-Users repository. Is that only needed for Multi-Master or also for the slave mirrors?

Source: https://gerrit.googlesource.com/plugins/replication/+/refs/heads/stable-2.16/src/main/resources/Documentation/about.md

[Luca Milanesio]

On 7 Nov 2019, at 18:45, Carbon Robin <douglas...@gmail.com> wrote:

Luca,

You said refs/starred-changes/* is (* NOT NEEDED *), but the replication plugin documentation shows replication of NoteDB account data requires that branch from the All-Users repository.

I believe there is a typo in the documentation: the starred changes is not needed for slaves, as they don't have a GUI.

I'll upload a fix on that.

Is that only needed for Multi-Master or also for the slave mirrors?

The multi-master and multi-site setup is a completely different beast :-)

I'll talk about the multi-master/multi-site in the next week Gerrit User Summit in Sunnyvale, including all the consequences associated with replication.

If you are interested and you want more info and ask questions, you can register on-line to watch the summit in streaming at:

https://live.gerritforge.com

HTH

Luca.

Source: https://gerrit.googlesource.com/plugins/replication/+/refs/heads/stable-2.16/src/main/resources/Documentation/about.md

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a0a95e27-f68f-4bb7-a934-85d7f9393fb3%40googlegroups.com.

[Luca Milanesio]

On 7 Nov 2019, at 21:08, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 7 Nov 2019, at 18:45, Carbon Robin <douglas...@gmail.com> wrote:

Luca,

You said refs/starred-changes/* is (* NOT NEEDED *), but the replication plugin documentation shows replication of NoteDB account data requires that branch from the All-Users repository.

I believe there is a typo in the documentation: the starred changes is not needed for slaves, as they don't have a GUI.

I'll upload a fix on that.

Done:

https://gerrit-review.googlesource.com/c/plugins/replication/+/244281

Luca.

[Doug Luedtke]

Update:

After the GUS2019 we ended up ordering a new server with more than twice the specs. The new box was rushed to be delivered and took about a month to get racked. Then I was able to use it as a stage environment for Java tuning and load testing. Load testing was done by having Jenkins clone from it non-stop. Then we simulated a large replication load by creating a replication.config that pointed to 4 VMs with 10 destinations each and 40 threads total.

Load testing showed promising results. We pushed out 5GB repos and 1GB repos. We were able to saturate the VM IO and barely show a CPU run queue on the box.

On February 22nd we migrated to the new box. Today we are able to replicate to 16 mirrors at 5 threads each and the queue is being worked like it was previous to Gerrit 2.16 with PostgreSQL again. Very fast at working the queue. Also running "gerrit gc" now takes less than 10 minutes instead of 55 minutes. We are no longer sitting at a CPU run queue of 70 for 18 hours a day.

Where we started:

* Gerrit 2.16

* NoteDB migrated

* CPU - 24 cores (48 threads)

* 382GB RAM

* SSDs

* 96GB Java heap

* G1GC

* 16 mirrors around the world

* 2400 repos

* 1700 users

* Total repos size 150GB (Largest repo 6GB)

New box specs:

* CPU - 56 cores (112 threads)

* 768GB RAM

* SSDs, repos stored on SAN with dual fiber connection

* Java Options:

javaOptions = "-XX:+UseG1GC"

javaOptions = "-Xms256g"

javaOptions = "-Xmx256g"

javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"

javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"

javaOptions = "-Xloggc:/<GERRIT_HOME>/logs/javagc-%t.log"

javaOptions = "-XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=35"

javaOptions = "-XX:ParallelGCThreads=70 -XX:ConcGCThreads=18 -XX:InitiatingHeapOccupancyPercent=60"

javaOptions = "-XX:MaxGCPauseMillis=500"

javaOptions = "-XX:+ExplicitGCInvokesConcurrent"

javaOptions = "-XX:+PrintGCDetails"

javaOptions = "-XX:+PrintGCDateStamps"

javaOptions = "-XX:+PrintGCTimeStamps"

javaOptions = "-XX:+PrintGCCause"

We are also not experiencing any stop the world Java GC anymore. The old box just couldn't handle our replication load for Java tuning to make a difference.

[Doug Luedtke]

Other changes we made:

* Upgraded from Gerrit 2.16.11 to Gerrit 2.16.15

* Switched from Apache to Nginx

* Changed auth from HTTP_LDAP to LDAP.

* Work with Jenkins Admins to pull from mirrors instead of master. 10s of thousands of clones a day difference.