HTTPS GET threads are not closed even after users close their review page

davidch...@gmail.com

unread,

Sep 16, 2021, 12:39:52 PM9/16/21

to Repo and Gerrit Discussion

Hi Folks,

We noticed long running HTTPD threads on javamelody monitoring system. When lots of such HTTP requests got accumulated, server CPU spiked and application went unresponsive. We checked with the user who triggered the query and got response that he was checking for his watched changes and closed the browser as query failed with some errors.

Any idea why gerrit is not closing these threads even after user closes the browser and ends the connection.?

Thread dump related to the query is given below.

"HTTP GET /gerrit/changes/?O=81&S=0&n=25&q=is%3Awatched%20is%3Amerged (user from XX.183.XX.YY)" #1172231 prio=5 os_prio=0 tid=0x00007e9ec8002800 nid=0x3a9a waiting on condition [0x00007e9e06b69000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00007f6ff138e590> (a com.google.common.util.concurrent.TrustedListenableFutureTask)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:502)

at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)

at com.google.gerrit.lucene.LuceneChangeIndex$ChangeDataResults.toList(LuceneChangeIndex.java:407)

at com.google.gerrit.lucene.LuceneChangeIndex$ChangeDataResults.iterator(LuceneChangeIndex.java:401)

at com.google.common.collect.Iterables$5.iterator(Iterables.java:698)

at com.google.gerrit.server.index.change.IndexedChangeQuery$1.iterator(IndexedChangeQuery.java:97)

at com.google.common.collect.Iterables$2.iterator(Iterables.java:515)

at com.google.common.collect.Iterables$5.iterator(Iterables.java:698)

at com.google.common.collect.FluentIterable$2.iterator(FluentIterable.java:277)

at com.google.gerrit.index.query.AndSource.readImpl(AndSource.java:127)

at com.google.gerrit.index.query.AndSource.read(AndSource.java:83)

at com.google.gerrit.index.query.QueryProcessor.query(QueryProcessor.java:266)

at com.google.gerrit.index.query.QueryProcessor.query(QueryProcessor.java:190)

at com.google.gerrit.server.restapi.change.QueryChanges.query(QueryChanges.java:139)

at com.google.gerrit.server.restapi.change.QueryChanges.apply(QueryChanges.java:115)

at com.google.gerrit.server.restapi.change.QueryChanges.apply(QueryChanges.java:42)

at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:458)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:485)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:121)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.GwtCacheControlFilter.doFilter(GwtCacheControlFilter.java:72)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:501)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)

at org.eclipse.jetty.server.HttpChannel$$Lambda$541/683286400.dispatch(Unknown Source)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)

at java.lang.Thread.run(Thread.java:748)

Regards,
Challs

davidch...@gmail.com

unread,

Oct 7, 2021, 3:19:51 AM10/7/21

to Repo and Gerrit Discussion

Hi Folks,

We are seeing issues again with HTTPD threads which are not getting closed even after reporting 502 error. Stack trace added below.

"HTTP GET /gerrit/changes/?O=81&S=0&n=100&q=is%3Awatched%20-status%3Aabandoned (user from XX.XX.XX.XX)" prio=5 WAITING

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:502)

com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)

com.google.gerrit.lucene.LuceneChangeIndex$ChangeDataResults.toList(LuceneChangeIndex.java:407)

com.google.gerrit.lucene.LuceneChangeIndex$ChangeDataResults.iterator(LuceneChangeIndex.java:401)

com.google.common.collect.Iterables$5.iterator(Iterables.java:698)

com.google.gerrit.server.index.change.IndexedChangeQuery$1.iterator(IndexedChangeQuery.java:97)

com.google.common.collect.Iterables$2.iterator(Iterables.java:515)

com.google.common.collect.Iterables$5.iterator(Iterables.java:698)

com.google.common.collect.FluentIterable$2.iterator(FluentIterable.java:277)

com.google.gerrit.index.query.AndSource.readImpl(AndSource.java:127)

com.google.gerrit.index.query.AndSource.read(AndSource.java:83)

com.google.gerrit.index.query.QueryProcessor.query(QueryProcessor.java:266)

com.google.gerrit.index.query.QueryProcessor.query(QueryProcessor.java:190)

com.google.gerrit.server.restapi.change.QueryChanges.query(QueryChanges.java:139)

com.google.gerrit.server.restapi.change.QueryChanges.apply(QueryChanges.java:115)

com.google.gerrit.server.restapi.change.QueryChanges.apply(QueryChanges.java:42)

com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:458)

javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:485)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:121)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.GwtCacheControlFilter.doFilter(GwtCacheControlFilter.java:72)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)

net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)

com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)

com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)

com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)

org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)

org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)

org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)

org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

org.eclipse.jetty.server.Server.handle(Server.java:501)

org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)

org.eclipse.jetty.server.HttpChannel$$Lambda$359/510049514.dispatch(Unknown Source)

org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)

org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)

org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)

org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)

org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)

org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)

java.lang.Thread.run(Thread.java:748)

Regards,

Challs

Luca Milanesio

unread,

Oct 7, 2021, 3:48:34 AM10/7/21

to Repo and Gerrit Discussion, Luca Milanesio, davidch...@gmail.com

On 7 Oct 2021, at 08:19, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Folks,

We are seeing issues again with HTTPD threads which are not getting closed even after reporting 502 error. Stack trace added below.

It looks like the Lucene did not answer and left the future pending.

Which Gerrit version are you running?

Did you get a full JVM thread dump?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/c6445ddf-dd0e-4c94-85ca-8cb11ac77dcdn%40googlegroups.com.

davidch...@gmail.com

unread,

Oct 7, 2021, 4:03:54 AM10/7/21

to Repo and Gerrit Discussion

Hi Luca,

Thanks for the reply.

Which Gerrit version are you running?

We are still in v2.16.22. Version upgrade plan is ongoing, we will be upgrading soon v3.X.

Did you get a full JVM thread dump?

Yes, I have got the full thread dump. As it is too big and contains company/repo/user/IP details I am unable to share full thread dump here. Please let me know the interested parts from thread dump Shall share it here after sanitizing.

Regards,

Challs

Luca Milanesio

unread,

Oct 7, 2021, 4:06:58 AM10/7/21

to Repo and Gerrit Discussion, Luca Milanesio, davidch...@gmail.com

On 7 Oct 2021, at 09:03, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Luca,

Thanks for the reply.

Which Gerrit version are you running?
We are still in v2.16.22. Version upgrade plan is ongoing, we will be upgrading soon v3.X.

I’ll see what I can find there.

Did you get a full JVM thread dump?
Yes, I have got the full thread dump. As it is too big and contains company/repo/user/IP details I am unable to share full thread dump here.

Yeah, I know. Have you thought about Enterprise Support instead? That would be more suitable for deep analysis of this kind.

HTH

Luca.

[1] https://www.gerritcodereview.com/support.html#enterprise-support

Please let me know the interested parts from thread dump Shall share it here after sanitizing.

Regards,
Challs

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/4fc39e32-f17f-44ba-a3d5-617bd0bb27b9n%40googlegroups.com.

Luca Milanesio

unread,

Oct 7, 2021, 4:10:01 AM10/7/21

to Repo and Gerrit Discussion, Luca Milanesio, davidch...@gmail.com

On 7 Oct 2021, at 09:06, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 7 Oct 2021, at 09:03, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Luca,

Thanks for the reply.

Which Gerrit version are you running?
We are still in v2.16.22. Version upgrade plan is ongoing, we will be upgrading soon v3.X.

I’ll see what I can find there.

From what I see in the code, the execution is delegated to another thread:

@IndexExecutor(INTERACTIVE) ListeningExecutorService executor,

You should find that one in the JVM thread dump and see why it is blocked.

Luca.

davidch...@gmail.com

unread,

Oct 7, 2021, 6:41:20 AM10/7/21

to Repo and Gerrit Discussion

Hi Luca,

Thanks for looking into the issue.

You should find that one in the JVM thread dump and see why it is blocked.

When I checked the threaddump, there are no "blocked" threads.

Additionally we were able to reproduce the issue in another instance which is running on v2.16.22. With no "watched projects" configuration, if we trigger below query in UI it is resulting in 502 error, but in backend it trying to collect/query all changes continuously and it is never getting stopped.

https://server_name/gerrit/q/is:watched%20-status:abandoned

In another gerrit instance with version v3.2.7, no such issues seen.

Regards,

Challs

Luca Milanesio

unread,

Oct 7, 2021, 11:03:49 AM10/7/21

to Repo and Gerrit Discussion, Luca Milanesio, davidch...@gmail.com

On 7 Oct 2021, at 11:41, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Luca,

Thanks for looking into the issue.

You should find that one in the JVM thread dump and see why it is blocked.
When I checked the threaddump, there are no "blocked" threads.

They do not necessarily need to be blocked, maybe they are busy in doing something else?

Without the full thread-dump is more a guess-work I am afraid.

Additionally we were able to reproduce the issue in another instance which is running on v2.16.22. With no "watched projects" configuration, if we trigger below query in UI it is resulting in 502 error, but in backend it trying to collect/query all changes continuously and it is never getting stopped.

https://server_name/gerrit/q/is:watched%20-status:abandoned

In another gerrit instance with version v3.2.7, no such issues seen.

Can you reproduce the issue systematically?

Luca.

Regards,
Challs

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a2090cb5-0d32-4f25-9d76-92afa4e7eea3n%40googlegroups.com.

davidch...@gmail.com

unread,

Oct 7, 2021, 11:51:43 AM10/7/21

to Repo and Gerrit Discussion

Hi Luca,

Without the full thread-dump is more a guess-work I am afraid.

I have attached full thread dump collected from QA instance which is created from data dump taken from production instance. Issue was reproduced in this instance. CPU load in server started to spike after executing the query in browser.

Can you reproduce the issue systematically?

I have not tried to reproduce the issue in fresh installation of v2.16.22. Will give a try tomorrow. QA instance had similar configuration setup like production . So we tried the same query there.

Regards,

Challs

ThreadDump_QA.txt

Luca Milanesio

unread,

Oct 7, 2021, 1:52:41 PM10/7/21

to davidch...@gmail.com, Luca Milanesio, Repo and Gerrit Discussion

On 7 Oct 2021, at 16:51, davidch...@gmail.com <davidch...@gmail.com> wrote:

Hi Luca,

Without the full thread-dump is more a guess-work I am afraid.
I have attached full thread dump collected from QA instance which is created from data dump taken from production instance. Issue was reproduced in this instance. CPU load in server started to spike after executing the query in browser.

I see two threads executing Lucene queries:

"Index-Interactive-42" prio=5 RUNNABLE

org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$2.get(Lucene54DocValuesProducer.java:501)

org.apache.lucene.util.LongValues.get(LongValues.java:45)

org.apache.lucene.search.FieldComparator$LongComparator.compareBottom(FieldComparator.java:418)

org.apache.lucene.search.MultiLeafFieldComparator.compareBottom(MultiLeafFieldComparator.java:50)

org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$1.collect(TopFieldCollector.java:117)

org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:56)

org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:48)

org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:668)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:472)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:591)

org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:576)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:503)

com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.doRead(LuceneChangeIndex.java:366)

com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.access$200(LuceneChangeIndex.java:267)

com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:318)

com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:315)

[…]

"Index-Interactive-44" prio=5 RUNNABLE

org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:284)

org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)

org.apache.lucene.search.TopFieldCollector.updateBottom(TopFieldCollector.java:407)

org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$1.collect(TopFieldCollector.java:130)