It looks like using the bitmap index for pushing from this large repository slows down upload-pack considerably.Did anybody else observe similar behavior ?So far we run git gc using git 2.26.2.We plan to try again with bitmap index generated by jgit gc.
On Tue, Sep 7, 2021 at 3:03 PM Matthias Sohn <matthi...@gmail.com> wrote:
For some months we were observing sporadic performance issues (few times per week)
with replicating a large repository to another datacenter (Germany to Netherlands) using the replication plugin.This server still runs gerrit 2.16 (jgit 5.1.15) and it will be upgraded to gerrit 3.2 (jgit 5.12.0) soon.
Most of the time the replication delay for this repository was between 5-10 minutes. Sporadically (few times a week)a replication task ran for 2-3 hours. During that time other replication tasks for the same repository kept being rescheduled.This isn't acceptable for the project team since they need prompt results from their CI jobs fetching from the Gerrit replica.Sometimes restarting the replication plugin could workaround the long running outlier task, sometimes not.We created thread dumps for such a long runner and found that it's busy inflating the bitmap index all the time(all thread dumps looked like this) :
This led us to the idea to try without a bitmap index and to our surprise this reduced the replication delay to 20-90 seconds !
The slowest tasks now take 2.5 minutes only. No more outliers taking more than 2 hours.
Here is the replication delay with git bitmap index (left of the red arrow) and without bitmap index (right of the arrow).Note that the vertical axis has a logarithmic scale.
It looks like using the bitmap index for pushing from this large repository slows down upload-pack considerably.Did anybody else observe similar behavior ?
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
On Mon, Sep 13, 2021 at 6:01 PM Matthias Sohn <matthi...@gmail.com> wrote:On Tue, Sep 7, 2021 at 3:03 PM Matthias Sohn <matthi...@gmail.com> wrote:For some months we were observing sporadic performance issues (few times per week)with replicating a large repository to another datacenter (Germany to Netherlands) using the replication plugin.This server still runs gerrit 2.16 (jgit 5.1.15) and it will be upgraded to gerrit 3.2 (jgit 5.12.0) soon.Most of the time the replication delay for this repository was between 5-10 minutes. Sporadically (few times a week)a replication task ran for 2-3 hours. During that time other replication tasks for the same repository kept being rescheduled.This isn't acceptable for the project team since they need prompt results from their CI jobs fetching from the Gerrit replica.Sometimes restarting the replication plugin could workaround the long running outlier task, sometimes not.We created thread dumps for such a long runner and found that it's busy inflating the bitmap index all the time(all thread dumps looked like this) :This led us to the idea to try without a bitmap index and to our surprise this reduced the replication delay to 20-90 seconds !The slowest tasks now take 2.5 minutes only. No more outliers taking more than 2 hours.Here is the replication delay with git bitmap index (left of the red arrow) and without bitmap index (right of the arrow).Note that the vertical axis has a logarithmic scale.It looks like using the bitmap index for pushing from this large repository slows down upload-pack considerably.Did anybody else observe similar behavior ?I asked around internally to our JGit experts. Here are comments:- push is using the JGit client. That hasn't had much love; we don't use it all inside Google (we use pull replication internally).- there is a plan to have negotiation on push, but likely wouldn't affect this problem, which seems more basic.
- one theory: it's trying to replicate refs/changes/*. Since those are in the GC_REST pack, bitmaps won't work for them,so all work trying to look for them is wasted.
- upgrading JGit might help; this is code from ~2018 ?
There are quite a number of changes affecting bitmap implementation in jgit between 5.1.15 and 5.12.0hence we will also try with gerrit 3.2.-Matthias
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3Rz-s24aA7hc_9L61z4PY-L%3DPVxaP6BOL6peEhJrGjFYg%40mail.gmail.com.