1 core pegged at 100% cpu

[James Nord]

Hi all,

I'm running a slightly old version of gerrit (2.8) and am seeing one core (4 core box) being pegged at 100%.

its always the same thread - and a thread dump shows its a HTTP (worker?) thread.

"HTTP-5861" prio=10 tid=0x00002aaac86be800 nid=0x2c0e runnable [0x0000000044232000]

   java.lang.Thread.State: RUNNABLE

        at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:811)

        at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)

        at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)

        at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1078)

        at org.eclipse.jetty.server.AbstractHttpConnection$Output.sendContent(AbstractHttpConnection.java:1185)

        at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:831)

        at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:513)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89)

        at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)

        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)

        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)

        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)

        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:67)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:365)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)

        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)

        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

        at java.lang.Thread.run(Thread.java:744)

Gerrit is using the defaults for the [httpd] section so I should have 2 acceptor threads and a minimum of 5 spare worker threads (up to a max of 25).

I've checked the release notes for the various newer releases and haven't seen anything related to this.

As the HTTP working (according to my understanding) is pooled I don't see why I should get all the load on one thread - it should flip.

So anyone aware of anything in this area that has been fixed, or have seen this before?

Regards,

/James

[Saša Živkov]

Are you using SSL (https)?

Jetty is known for having issues with handling SSL connections... and the only

way to unblock this thread is to restart Gerrit.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leandro Fonseca]

Hello,

I'm having this same problem, using Gerrit 2.11.3 and HTTPS.

Was there any fix released for this issue on later versions? Is there any workaround that can be made besides restarting the Gerrit process?

Thanks,

Leandro.

[Leandro Fonseca]

Also, does anyone knows what exactly can trigger this problem (some specific query, http clone, etc)?

Thanks,

Leandro

[Leandro Fonseca]

I have added a more complete description of my problem on below post:

https://groups.google.com/forum/#!topic/repo-discuss/AvxOQW8qtCE