Gerrit 2.12 - access to some repositories gets slow

[Saša Živkov]

Anyone running Gerrit 2.12 and seeing degrading performance for only some repos?

We have been observing exactly that issue since we upgraded to 2.12.

For example: fetching from a tiny repository would take more than 10 seconds while

fetching from a large repository would be very fast.

Similarly, posting a comment on some change(s) from affected repository

would be extremely slow (several seconds) while for other repositories it is as fast as usual.

Once the issue occurs it is permanent, until next Gerrit server restart.

We also found the reason for that: Repository.useCnt in JGit unexpectedly drops below 1.

Once this happens all cached entities of that repository are removed on every close() method call

and they are loaded again every time repository is opened. This can happen multiple times

during processing of a single request.

This issue could be detected by creating a thread dump during a slow request and searching

for WindowCache.removeAll in it.

We don't know yet where is the bug: in Gerrit, in JGit or in a plugin.

[Jean-Jacques Lafay]

We've also had performance issues with 2.12.5, which I also tracked down to JGit repo cache being flushed.

Upon further investigation (and thanks to your hint about useCnt), this was due to a double close in MergeOp (lines 446 and 461). Interestingly, the flush occurs when the count is exactly 0, so after enough changes have been merged (mainly more than the number of indexing threads that kick in immediately after a successful merge), it always stays negative, and performances return to normal.

This specific bug has been fixed in 2.13, as a side-effect of this change, and I proposed this one for 2.12.

Also note a somewhat related issue, as well as this JGit change.