Gerrit Site Frozen after "Get changes to reindex caused by refs/notes/review update of project"

[Stephen Cprek]

Hi,

I upgraded my gerrit site version to 2.11.2 this weekend and things seemed fine, then all the sudden the site freezes and I checked the queue (output below)

It's not uncommon for things to get slow every now and then, but not sure why I was required to force restart it after 15 min of no tasks being completed.

Is this related to online reindexing to update the "status of a change" for example all the red "cannot merge" statements? Just trying to figure out how I can prevent this from

happening in the future. Maybe too much stress on the machine?

I have a gerrit + jenkins instance running within jetty with heapLimit = 14g

[database]

       ....

        connectionPool = true

        poolMinIdle = 8

        poolMaxIdle = 16

        poolLimit = 400

        poolMaxWait = 15s

[sshd]

        ....

        threads = 80

[index]

        type = LUCENE

        threads = 16

[index "changes_open"]

        commitWithin = 0

[index "changes_closed"]

        commitWithin = 0

I've been passed on the role of managing this server and commitWithin was used for minimal loss. Maybe this is the issue?

gerrit show-queue -w

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.086      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.085      Index change xxxxxx of project A

waiting .... 14:51:53.105      Get changes to reindex caused by refs/notes/review update of project A

------------------------------------------------------------------------------

  78 tasks

stayed at 78 for over 15 mins

[Dariusz Trawinski]

I noticed similar symptops with tons of gerrit reindex tasks in the queue causing very heavy load on the server. It didn't break the service but this heavy load is causing occasionally freeze from full jvm full gc (all heap size is getting consumed).

I'm not sure what is triggering such high volume of reindex jobs in a single project and what could be the tuning on jvm or gerrit side.

Any suggestions would be most welcomed.

[Björn Pedersen]

Is it  the same as here? https://groups.google.com/forum/#!topic/repo-discuss/Kpgumrnnza4

[Dariusz Trawinski]

I have doubts if this is identical but maybe I'm wrong. I the scenario I observed there was very heavy load from online reindexing in a big project. There was also no particular user side operation identified that would trigger such action. Could it be related to cache timeout? Generally all changes were eventually successfully indexed but it is not clear why this indexing was initiated while the changes were quite old.

[Saša Živkov]

On Wed, Nov 25, 2015 at 12:54 PM, Dariusz Trawinski <dtra...@gmail.com> wrote:

I have doubts if this is identical but maybe I'm wrong. I the scenario I observed there was very heavy load from online reindexing in a big project. There was also no particular user side operation identified that would trigger such action. Could it be related to cache timeout? Generally all changes were eventually successfully indexed but it is not clear why this indexing was initiated while the changes were quite old.

Online reindexing is triggered automatically after an upgrade if the (lucene) index schema in the new Gerrit

version is newer than the index schema in the previous version. Basically the index(es) based on the old schema

version needs to be converted to the new schema version.

 

On Wednesday, November 25, 2015 at 10:02:17 AM UTC+1, Björn Pedersen wrote:

Is it  the same as here? https://groups.google.com/forum/#!topic/repo-discuss/Kpgumrnnza4

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dariusz Trawinski]

In my scenario last upgrade to 2.11.2 happened few months ago which was followed by offline reindexing. At closer look at impacted changes it looks like the culprit could be in their related changes. It looks like the changes being reindexed there related and there was about 4000 of them. It looks like like updating one change is triggering reindex of all related changes. With huge number of related changes in big projects it could probably lead to server overloading and hang.

Is it a correct hypothesis?

[Bassem Rabil]

You can throttle the number of reindexing threads using index.batchThreads [1]. This can help you to avoid overloading your instance with such background task. At our instance we set this to 4-6 threads. The drawback here is that the process queue in Gerrit instance is large for longer time, but this is not affecting core tasks for the instance like for example receiving a push or reviewing a change using web UI.

[1] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#index

Regards

Bassem

[Dariusz Trawinski]

That is interesting comment. In my case index.threads is set to 4. However in the observations sporadically there are being triggered 48 running threads of index operations in show-queue which the equal to the number of CPU cores. That should indicated the limitation is specific to index.batchThreads which has the default value. Based on documentation batch threads are specific to background operations such as schema upgrades. However the last upgrade to 2.11.2 was completed several months ago so that should not be the case. What could be other background operations in gerrit?

[Bassem Rabil]

One potential background operation is the reindexing with mergeability checks which is performed on all open changes towards a branch when a change is merged to this branch, basically the check is performed to update the mergeability status with the new branch tip merged. This is performed after the initial online reindexer is done upon upgrading your instance. These are considered batch checks that are throttled using the same parameter index.batchThreads. You need to restart the instance after setting this in gerrit.config. You can ensure the number of threads allocated for the batch indexing using JavaMelody plugin installed on your instance to view the number of threads called "Index-Batch-x".