[JIRA] (JENKINS-61823) "--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

2 views
Skip to first unread message

daniel@danielgrunwald.de (JIRA)

unread,
Apr 6, 2020, 12:50:02 PM4/6/20
to jenkinsc...@googlegroups.com
Daniel Grunwald created an issue
 
Jenkins / Bug JENKINS-61823
"--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1
Issue Type: Bug Bug
Assignee: Unassigned
Components: winstone-jetty
Created: 2020-04-06 12:49
Priority: Minor Minor
Reporter: Daniel Grunwald

After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
It seems that Jenkins no longer honors the "--httpKeepAliveTimeout=60000" switch.
I confirmed that Jenkins is running with switch.
Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
 at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
 at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Caused: java.io.IOException
 at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
 at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
 at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
 at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
 at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
 at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
 at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
 at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
 at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
 at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
 at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
 at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)

The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

daniel@danielgrunwald.de (JIRA)

unread,
Apr 6, 2020, 12:51:02 PM4/6/20
to jenkinsc...@googlegroups.com
Daniel Grunwald updated an issue
Change By: Daniel Grunwald
After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
It seems that Jenkins no longer honors the "\-\-httpKeepAliveTimeout=60000" switch.
I confirmed that Jenkins is running with
this switch.

Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

{code}

java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Caused: java.io.IOException
at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
{code}


The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.

daniel@danielgrunwald.de (JIRA)

unread,
Apr 6, 2020, 12:58:03 PM4/6/20
to jenkinsc...@googlegroups.com

daniel@danielgrunwald.de (JIRA)

unread,
Apr 6, 2020, 1:57:02 PM4/6/20
to jenkinsc...@googlegroups.com
Daniel Grunwald commented on Bug JENKINS-61823
 
Re: "--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

I downgraded to 2.204.6, and now my downloads are complete again.

daniel@danielgrunwald.de (JIRA)

unread,
Apr 6, 2020, 1:58:02 PM4/6/20
to jenkinsc...@googlegroups.com

dbeck@cloudbees.com (JIRA)

unread,
Apr 6, 2020, 4:41:03 PM4/6/20
to jenkinsc...@googlegroups.com

olamy@apache.org (JIRA)

unread,
Apr 7, 2020, 12:41:06 AM4/7/20
to jenkinsc...@googlegroups.com

olamy@apache.org (JIRA)

unread,
Apr 7, 2020, 12:44:06 AM4/7/20
to jenkinsc...@googlegroups.com

olamy@apache.org (JIRA)

unread,
Apr 7, 2020, 12:50:06 AM4/7/20
to jenkinsc...@googlegroups.com

temporary workaround until is fixed with new version is to use  --KeepAliveTimeout

but it will be fixed with next version of winstone and this workaround will not work anymore  

olamy@apache.org (JIRA)

unread,
Apr 7, 2020, 12:59:06 AM4/7/20
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 10:58:03 AM4/7/20
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 11:01:04 AM4/7/20
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 11:01:05 AM4/7/20
to jenkinsc...@googlegroups.com
Oleg Nenashev started work on Bug JENKINS-61823
 
Change By: Oleg Nenashev
Status: Open In Progress

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 11:04:10 AM4/7/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Labels: lts-candidate non-trivial-lts-backporting regression

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 11:07:04 AM4/7/20
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-61823
 
Re: "--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

BACKPORTING NOTES:

  • Winstone master branch includes a patch for Winstone redirects by Alex Earlhttps://github.com/jenkinsci/winstone/pull/98 
  • The fix is quite trivial, but we may not want to backport it to 2.222.x due to our experience in the past months
  • I can prepare a release of Winstone 5.9.1 with a backport
Reply all
Reply to author
Forward
0 new messages