slow pushes to gerrit

[Ishaaq Chandy]

Hi all,

Of late we've been noticing that to our Gerrit instance has been annoyingly slow. This is primarily due to the fact that, more often than not, pushes seem to send up over 4MB of data from the local git client to gerrit despite the fact that the commits being pushed are nowhere near that size.

We used to reduce the chances of this occurring by doing a fetch before pushing but that no longer seems to work.

We're using Gerrit v2.7. Here's some of the (possibly) relevant tuning params we've used:

[container]

        heapLimit = 8g

[core]

        packedGitOpenFiles = 4096

        packedGitLimit = 2g

        packedGitWindowSize = 32k

Any ideas as to what is causing this? Or is this expected behaviour now?

Thanks,

Ishaaq

[Matthias Sohn]

questions for the repositories you observe slow pushes for:

- How many refs does your repository have?

- How many objects are there? Does it have many large objects? 

- Are you maxing out on any caches?

- Do you run gc regularly? Do you use git gc or gerrit gc ?

--
Matthias

[Ishaaq Chandy]

On 16 October 2013 09:00, Matthias Sohn <matthi...@gmail.com> wrote:

- How many refs does your repository have?

By refs do you mean branches or changesets?

Branches: 49

changesets: not sure how to count but we're up to change number 9035 at last count

 

- How many objects are there? Does it have many large objects? 

Again, how does one tell? From memory we had some devs push some large binaries in the past but we don't do that anymore so while they are there in the repos they are part of ancient history and only affect us if we are cloning from scratch - which doesn't happen often (less than once a week??)

 

- Are you maxing out on any caches?

Not sure, here's the output from show-caches - not sure how to interpret the data:

Gerrit Code Review        2.7                       now    22:10:18   UTC

                                                 uptime   13 days 20 hrs

  Name                          |Entries              |  AvgGet |Hit Ratio|

                                |   Mem   Disk   Space|         |Mem  Disk|

--------------------------------+---------------------+---------+---------+

  accounts                      |    44               |   2.4ms | 99%     |

  accounts_byemail              |    42               |   1.3ms | 99%     |

  accounts_byname               |    35               |   1.4ms | 99%     |

  adv_bases                     |                     |         |         |

  changes                       |                     |         |         |

  groups                        |     7               |   3.4ms | 96%     |

  groups_byinclude              |     6               |   1.2ms | 50%     |

  groups_byname                 |                     |         |         |

  groups_byuuid                 |     9               | 408.7us | 96%     |

  groups_external               |     1               | 759.6us | 50%     |

  groups_members                |     6               |   2.5ms | 99%     |

  permission_sort               |   100               |         | 99%     |

  plugin_resources              |                     |         |         |

  project_list                  |     1               |   6.0ms | 96%     |

  projects                      |    65               |  63.0ms | 99%     |

  sshkeys                       |    31               |   1.7ms | 99%     |

D diff                          |   159   2474  14.36m| 174.5ms | 96%  98%|

D diff_intraline                |  2596   6212   2.94m|   5.6ms | 51% 100%|

D git_tags                      |                0.00k|         |  0%     |

D web_sessions                  |   126   2005 943.28k|         | 99%  55%|

SSH:      2  users, oldest session started    9 days  2 hrs ago

Tasks:    3  total =    1 running +      0 ready +    2 sleeping

Mem: 806.00m total = 326.56m used +  78.18m free + 401.26m buffers

       7.11g max

          82 open files,        8 cpus available,       71 threads

 

 

- Do you run gc regularly? Do you use git gc or gerrit gc ?

About once a week using a cronjob, on weekends. Using git gc. I have run gerrit gc as well manually a few times - just haven't set it up with cron yet.

--
Matthias

[Ishaaq Chandy]

UPDATE - using gerrit's ls-user-refs command I see that the ref-count is now 13319 - looks like each changeset has a number of patch sets each of which is counted as a ref

[Matthias Sohn]

On Wed, Oct 16, 2013 at 12:14 AM, Ishaaq Chandy <ish...@gmail.com> wrote:

On 16 October 2013 09:00, Matthias Sohn <matthi...@gmail.com> wrote:

- How many refs does your repository have?

By refs do you mean branches or changesets?

Branches: 49

changesets: not sure how to count but we're up to change number 9035 at last count

usually for repositories managed by Gerrit the majority of refs are under refs/changes

(in the bare repository managed by the server) these are used to manage the patchsets

git for-each-ref refs/heads/ | wc -l

git for-each-ref refs/tags/ | wc -l

git for-each-ref refs/changes/ | wc -l

 

- How many objects are there? Does it have many large objects? 

Again, how does one tell? From memory we had some devs push some large binaries in the past but we don't do that anymore so while they are there in the repos they are part of ancient history and only affect us if we are cloning from scratch - which doesn't happen often (less than once a week??)

git count-objects -v

you may use the script [1] to find the biggest objects in your repo 

[1] https://gist.github.com/msohn/7000083

--

Matthias

[Ishaaq Chandy]

Ok, total refs = 13094

And here is the result from your script (paths removed from output):

size   packed  objectId

42629  1892    e36abca4f39bca3fcc3e98c569dc5884e21803f1

25347  1602    2f7040b243573fcd0f9fb744fab2adc17bc5005e

11169  574     8ad8d1dbf65f60b2b3bce5123f3741a36e1d12b0

5381   4931    81d11df5f2c8ed6e33a1350d98e800d943bac7e9

4645   1253    52f1650206a67d5df883a98d54492939cd260777

3549   205     78f38512a4212d048af051b50af3e9b319dca33a

2822   206     f941753011be03cf86e2b242bb521c7a4f647333

2321   2322    fb3e0433317f0e5ddb98b348dc00ac90c0ef2f74

2143   1156    8b162c39e0da2d4b07f7308a557b2128cfbf36a9

1317   696     ef1e7aab9849bc2d53f59cbee7d1e7c5e777fd8e

982    981     7a8b1d0aa8aefc5ce464e156a532294f11105e9e

874    68      d174545a76bad195ec1c73c21608c1ffc81967d0

710    175     cfa980120dd78c60610b3b8c5f6d572c00ad3e41

640    56      d2080493ecf2b42b0693bad0a05ffa8cde9a2775

606    39      1624743ee5ef55673e57f82e4ebee114902ca041

571    232     f12ad5e64e19675c0111c0a62d56d846df73f7f2

564    488     2890410026103954d54b97180eae6efdcfc291d2

448    110     5de191fddce862da09fa36344a92c4da266f187f

448    106     ee0e9a2734902047768beda36f4d368cbfb9c1a1

446    163     a4671afeeb792a285c5104a49e4896e756876e9a

425    35      56ab0f3cdb1555b23695e065016d9c0bf581935e

374    371     1fc478842f51e7519866f474a02ad605235bc6a6

372    31      aff10f043366be059cd1ad2eeecd9f2353fd631d

.. but, as I said, none of the above have actually changed much - most were deleted and the last one to be modified was in January this year.

[Bassem Rabil]

We tried the workaround described in this issue:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=394078

Simply add this parameter to your repository config

[core]
   bigFileThreshold = 20M

and then run git gc --aggressive to repack your repository with smaller pack files. After applying this the delta calculations for smaller pack files is much faster, and thus pushes takes shorter time.

I hope this helps.

Regards

Bassem

[Ishaaq Chandy]

Interesting. Are you sure this is the same problem though? Remember that those big files are way in the past and have not been touched for a long time so I don't understand why they should affect current push requests from clients.

If I were to do this config change - where should I do it? In the gerrit config or on each repo's git config?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bassem Rabil Guendy]

I am not sure if it is the same problem, for our case the symptoms we faced was slow push and freeze on delta compression. If you want to try it, you need to modify the repo config. You can revert it back by setting this value to large size 240m and git gc —aggressive to repack with larger pack files if needed.

Regards

Bassem

[Ishaaq Chandy]

Hmm, that got me reading more documentation and I stumbled upon this:

https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#_a_id_pack_a_section_pack

I wonder if I should turn on deltacompression - but the bit about more memory use worries me - we already face situations where the jvm's oldgen heap goes over 90% at times and every couple of weeks I find I have to restart Gerrit.

I also discovered this presentation by Shawn: http://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/Scaling%20Up%20JGit%20-%20EclipseCon%202013.pdf

The second last slide in the presentation mentions this config for "Big" JGit Servers (not sure our Gerrit instance qualifies):

-Xmx8g # gobs of JVM heap

# for `jgit daemon` or gerrit.config

[core]

packedGitLimit = 4g # around 50% of JVM heap

packedGitOpenFiles = 8192 # don't forget to configure ulimit

streamFileThreshold = 2047m # avoids pathological inflate

deltaBaseCacheLimit = 50m

# applies to `jgit gc`

[pack]

bigFileThreshold = 20m # don't delta compress big binaries

indexVersion = 2 # use index v2

[Ishaaq Chandy]

Tried turning on deltacompression, deltaBaseCacheLimit, enabled a cron job that runs gerrit gc once an hour. All to no avail, the problem has persisted.

Nor did setting bigFileThreshold and running gc --aggressive help.

Any other suggestions? This is really puzzling.

[Bassem Rabil Guendy]

Did you try setting bigFileThreshold for the repository in the configuration file of the affected repositories  ?

 

Regards

Bassem

 

From: Ishaaq Chandy [mailto:ish...@gmail.com]
Sent: October-20-13 7:18 AM
To: Bassem Rabil Guendy
Cc: Repo and Gerrit Discussion; Matthias Sohn
Subject: Re: slow pushes to gerrit

 

Tried turning on deltacompression, deltaBaseCacheLimit, enabled a cron job that runs gerrit gc once an hour. All to no avail, the problem has persisted.

[Ishaaq Chandy]

Yes, I did. As I said, it would have been surprising if it did work (since the files in question haven't been touched in ages) - but it didn't.

[Ishaaq Chandy]

FYI - I've finally got to the bottom of the problem and found a fix:

Turns out the problem wasn't Gerrit - it was the git client. I was using git v1.8.3.x - upgraded to v1.8.4.2 (the latest version available on the Ubuntu git-core PPA - https://launchpad.net/~git-core/+archive/ppa ) and the problem disappeared!

Reading the following two release logs explains why - there have been changes/bug-fixes to the pack/smart-protocol:
https://raw.github.com/git/git/master/Documentation/RelNotes/1.8.4.txt
https://raw.github.com/git/git/master/Documentation/RelNotes/1.8.4.2.txt

In fact the case may not be closed completely. According to a comment on https://raw.github.com/git/git/master/Documentation/RelNotes/1.8.4.3.txt - there may be an edge case for making new clones as well, unfortunately they haven't release 1.8.4.3 yet to the PPA, so I'd have to build git from source to get it, but for now, I'm happy with the improvement.