query on merged changes consumes too many CPU resources

[xf zheng]

once user queris on merged changes withou any other filters , gerrit will cost a lot of time to return  and  a lot of CPU resources.

when this happened, i checkout the debug log and found the query read almost all repos' changes although the user do not have the access right. also a item in Queue:Index-Interfactive displayed according. If I clicked multiple time of Changes->Merged menu, the Queue:Index-Interfactive items  accumulated as well  and gerrit cost all CPU resources and we must restart gerrit.

any idea will be appreciated! 

[Sven Selberg]

An educated guess is that you have updated not to long ago and used "online reindex" (i.e. you did not take Gerrit down to do an offline reindex) (I'm guessing the online reindex is involved, I have no actual proof).
Then Gerrit will reindex changes in the background or when necessary in same transaction as the query if the query hits "stale" changes.
I'm not well versed enough in the ins and outs of the index to give you a precise root cause but I have seen similar behavior when f.i. querying for every commit on a branch. Gerrit will encounter changes that it considers needs a reindex and reindex them in the same transaction as the  query. In the normal case all "active" or "recent" changes have been "touched" so that the index is up-to-date for them, but when you are making a query that includes changes that are "inactive" and "old" (like for every commit on a branch, or all merged commits) you can end up in this situation.

QC have recently done a lot of improvements to the index, perhaps some of them might be able to explain this more in depth.

 

[Sven Selberg]

On Monday, December 20, 2021 at 10:22:45 AM UTC+1 Sven Selberg wrote:

On Friday, December 17, 2021 at 3:43:03 PM UTC+1 zheng...@gmail.com wrote:

once user queris on merged changes withou any other filters , gerrit will cost a lot of time to return  and  a lot of CPU resources.

when this happened, i checkout the debug log and found the query read almost all repos' changes although the user do not have the access right. also a item in Queue:Index-Interfactive displayed according. If I clicked multiple time of Changes->Merged menu, the Queue:Index-Interfactive items  accumulated as well  and gerrit cost all CPU resources and we must restart gerrit.

any idea will be appreciated! 

An educated guess is that you have updated not to long ago and used "online reindex" (i.e. you did not take Gerrit down to do an offline reindex) (I'm guessing the online reindex is involved, I have no actual proof).
Then Gerrit will reindex changes in the background or when necessary in same transaction as the query if the query hits "stale" changes.
I'm not well versed enough in the ins and outs of the index to give you a precise root cause but I have seen similar behavior when f.i. querying for every commit on a branch. Gerrit will encounter changes that it considers needs a reindex and reindex them in the same transaction as the  query.

...and this quickly becomes quite load intensive and require a lot of CPU resources...

[xf zheng]

Sven, many thanks for your advice.

Same guess at the beginning. but it can still reproduce after I replicated prod data into testing env && re-index all data offline. 

[Sven Selberg]

On Tuesday, December 21, 2021 at 2:20:43 AM UTC+1 zheng...@gmail.com wrote:

Sven, many thanks for your advice.

Same guess at the beginning. but it can still reproduce after I replicated prod data into testing env && re-index all data offline. 

That would suggest that my assumption about this being directly related to online reindex was faulty.

[xf zheng]

after upgrade to 3.4.1, the problem is still existing. The steps to reproduce the problem as below,
1. login with a userid which has only 2 repos access right( 1.5k repos totally)
2. click Merge menu 3 times to run in new tab pages

it showed that
1. 3 items in Queue:Index-Interactive with very long time execution
2. CPU utilization is very high

how to fix this problem or it runs as design?

[Sven Selberg]

On Monday, December 27, 2021 at 9:19:14 AM UTC+1 zheng...@gmail.com wrote:

after upgrade to 3.4.1, the problem is still existing. The steps to reproduce the problem as below,
1. login with a userid which has only 2 repos access right( 1.5k repos totally)
2. click Merge menu 3 times to run in new tab pages

it showed that
1. 3 items in Queue:Index-Interactive with very long time execution
2. CPU utilization is very high

how to fix this problem or it runs as design?

On a very lean 3.4.2 instance the status:merged query returns after a couple of seconds with hundreds of thousands of changes so I'd say you have some sort of issue.

I'm running out of ideas but if you could share how large your instance is (number of changes, cpus, memory etc) and the index settings from gerrit.config perhaps someone who is more index-savvy than me might be able to help.

/Sven

[Martin Fick]

On 2021-12-27 01:19, xf zheng wrote:
> after upgrade to 3.4.1, the problem is still existing. The steps to
> reproduce the problem as below,
> 1. login with a userid which has only 2 repos access right( 1.5k repos
> totally)

I suspect that this ACL configuration is the source of your slowness. I
don't know if this has been addressed in a current version of Gerrit,
but I know that in older versions of Gerrit at least, this was a recipe
for long CPU intensive queries. In order to find changes to display,
Gerrit must iterate through almost all of your changes in order to find
enough of them that are actually visible to the user. In your case,
there
may not even be any merged changes visible to the user in those 2 repos,
and thus Gerrit is iterating through all merged changes?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[xf zheng]

Configuration as below,

OS:32C,64G
1.5K repos and 500K+ changes

Version: Gerrit 2.16.2 (Docker)

gerrit.conf

[container]

    heapLimit=48G

[index]

    threads=15

[sshd] 

    threads=8

[httpd]

    maxThreads=35

    maxQueued=400

[core]

    packedGitLimit=24G

[cache "permission_sort"]

    memoryLimit=20480

[cache "projects"]

    memoryLimit=2048

[cache "accounts"]

    memoryLimit=4096

[cache "groups_bysubgroup"]

    memoryLimit=4096

[xf zheng]

I checked the debug log and just as you said, gerrit will fetch through all merged changes from almost all repos. Thanks for your advice, will simplify the ACL configuration to see if any improvements. 

[xf zheng]

I have simplified the ACL for all projects and the problem still happens. 

Here is the solution I purposed so far,

install javamelody plugin

kill the java thread which executes more than 10 mins within the plugin

Hope it may help others who have the same problems.