[JIRA] (JENKINS-42513) Increase default keep-alive timeouts

4 views
Skip to first unread message

dbeck@cloudbees.com (JIRA)

unread,
Mar 6, 2017, 1:10:02 PM3/6/17
to jenkinsc...@googlegroups.com
Daniel Beck created an issue
 
Jenkins / Improvement JENKINS-42513
Increase default keep-alive timeouts
Issue Type: Improvement Improvement
Assignee: Daniel Beck
Components: core, winstone-jetty
Created: 2017/Mar/06 6:09 PM
Priority: Minor Minor
Reporter: Daniel Beck

Multiple reports (like JENKINS-42058) indicate the default timeouts are too aggressive when dealing with terrible connections.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jsoref+jenkins@gmail.com (JIRA)

unread,
Mar 15, 2019, 4:53:07 PM3/15/19
to jenkinsc...@googlegroups.com
Josh Soref commented on Improvement JENKINS-42513
 
Re: Increase default keep-alive timeouts

Note: --httpKeepAliveTimeout=10000 appears to be honored whether winstone means jetty or actually winstone...

This is what we hit:

java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:235)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:514)
        at net.bull.javamelody.internal.web.FilterServletOutputStream.write(FilterServletOutputStream.java:88)
        at net.bull.javamelody.internal.web.CounterResponseStream.write(CounterResponseStream.java:81)
        at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:604)
        at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
        at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:349)
        at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:141)
        at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
        at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
        at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
        at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
        at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
        at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:282)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:503)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

I'm going to try this for a week and see if it causes our problems to go away.

Afaict, this wasn't a problem until a recent jenkins (or build node) upgrade. But recently it's become a significant problem.
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

daniel@danielgrunwald.de (JIRA)

unread,
Oct 23, 2019, 6:28:02 AM10/23/19
to jenkinsc...@googlegroups.com

It's not just "terrible connections"; it could also be a consumer being slow for another reason.
We just ran into this issue with a shell script doing:

  curl -f $JENKINS/$JOB/artifact/binaries.tar.gz | tar xz

Sometimes unpacking was too slow (due to high load on the VM) and Jenkins terminated the download.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

dbeck@cloudbees.com (JIRA)

unread,
Nov 3, 2019, 5:41:02 PM11/3/19
to jenkinsc...@googlegroups.com
Daniel Beck assigned an issue to Unassigned
Change By: Daniel Beck
Assignee: Daniel Beck
Reply all
Reply to author
Forward
0 new messages