[JIRA] Created: (JENKINS-11130) java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
mcs130@gmail.com (JIRA)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: JENKINS-11130
URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
Project: Jenkins
Issue Type: Bug
Components: cli
Affects Versions: current
Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
* Tomcat 7.0.14
* JDK 1.6.0_14
* Only ONE Master configuration - no slaves are configured
* 3 Executors - (one less than the max number of CPUs)
Reporter: mark streit
We are currently seeing the following issue using the jenkins-cli.jar
with Jenkins 1.430. Tried going to 1.431, same issue.
Each time we insure we have the latest matching jenkins-cli.jar using the link on the Manage Jenkins page.
From inside a Jenkins job (I'll call it the parent) I'm trying to
execute multiple jobs using the client jar. On the last job, I
include the -s parameter so the parent job will not finish until the
last job in the list is complete. The problem is that after about 15
secs of waiting, the parent job ends (fails) with the EOF exception
below.
IMPORTANT INFO: When performing this exact same build using an older hudson-cli.jar with an older (pre-fork) version of Hudson v1.339, THIS WORKS. We want to move all of our builds over to the latest Jenkins version, and this is blocking us at the moment.
We have searched all day and tried numerous things including
reducing the number of jobs and also trying to run them sequentially.
Stack trace seen in Console at failure:
[workspace] $ /bin/bash -xe /opt/apache-tomcat-7.0.14_jenkins/temp/
hudson4690948305077288103.sh
+ WAR_FOLDER=/tmp/trunk/UIToolkit1.1.2
+ rm -rf /tmp/trunk/UIToolkit1.1.2/calendar.war /tmp/trunk/
UIToolkit1.1.2/Pay.war /tmp/trunk/UIToolkit1.1.2/OrgChart.war
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-
cli.jar -s http://jenkins-host:8082/jenkins/ build XYZ_Platform-
Calendar -p SVN_PATH=trunk -p STATIC_PATH=UIToolkit1.1.2
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-
cli.jar -s http://jenkins-host:8082/jenkins/ build XYZ_Platform-Pay -p
SVN_PATH=trunk -p STATIC_PATH=UIToolkit1.1.2
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-
cli.jar -s http://jenkins-host:8082/jenkins/ build XYZ_Platform-
OrgChart -p SVN_PATH=trunk -p STATIC_PATH=UIToolkit1.1.2
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-
cli.jar -s http://jenkins-host:8082/jenkins/ build XYZ_Platform-
PolicyAck -s -p SVN_PATH=trunk -p STATIC_PATH=UIToolkit1.1.2
Sep 16, 2011 4:35:37 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://jenkins-host:8082/jenkins/cli
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093)
Caused by: java.io.EOFException
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(ObjectInputStream.java:2498)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:
348)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
Exception in thread "main" hudson.remoting.RequestAbortedException:
hudson.remoting.RequestAbortedException: java.io.IOException:
Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:681)
at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:
158)
at $Proxy2.main(Unknown Source)
at hudson.cli.CLI.execute(CLI.java:171)
at hudson.cli.CLI._main(CLI.java:297)
at hudson.cli.CLI.main(CLI.java:216)
Caused by: hudson.remoting.RequestAbortedException:
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1117)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093)
Caused by: java.io.EOFException
at java.io.ObjectInputStream
$BlockDataInputStream.peekByte(ObjectInputStream.java:2498)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:
348)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
Build step 'Execute shell' marked build as failure
Notifying upstream projects of job completion
Finished: FAILURE
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153731#comment-153731 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
A few random thoughts:
* The 15 second timeout that you mention sounds vaguely like a webserver HTTP timeout.
* Have you tried running the same job/script outside of Jenkins? i.e. does it behave differently when run from the normal shell?
* Is there a proxy between the build script and Jenkins? I'm guessing not because jenkins-host:8082 sounds like it will be your Tomcat instance.
* Have you tried running with the builtin jenkins webserver instead of Tomcat? It maybe that the Tomcat HTTP layer is interacting badly with the cli library.
* Have you captured the traffic to port 8082 with tcpdump or wireshark? Looking at the raw traffic may give some clues.
I'm no expert on the CLI library (indeed I have never used it myself) but those are the places where I would start looking.
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153740#comment-153740 ]
mark streit commented on JENKINS-11130:
---------------------------------------
Thanks so much for the comments. Here is what we know relative to your ideas:
There are no proxies set up at all for either the Hudson instance or the Jenkins instance. 6
Both run on 2 separate tomcat servers .... Hudson on Tomcat 6.0.18. And Jenkins on Tomcat7.0.14
They Do run on the same physical sever.
Each runs on diff port 8081 Hudson and 8082 Jenkins
There is no master/slave configuration on either.
The build process works fine using the Hudson CLI jar and older Hudson 1.339.....no issues
The same process using the Jenkins CLI jar and Jenkins 1.431 fails....we copied over the same set of jobs
...cleared workspaces and attempted to run the exact same set on Jenkins and it fails with this error.
The 15 second time seems only relevant to how many are run....we scaled back to just a few jobs and can see it blow up in 4 seconds
Perhaps we need to drop back to Tomcat 6 and try ... I don't know.
We have been at a loss as to why this would happen with the newer versions of everything
Thank you again for taking time to comment
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153741#comment-153741 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
It would definitely be good to eliminate the difference between Tomcat 6 and 7. If it fails in one but not the other that gives a good start for analysis of the underlying cause.
I would also try the command from the commandline (your log above gives the contents of the script so that should be easy) and again that eliminates the running from within tomcat.
As for why things would break with newer versions it could be a simple bug; Tomcat could be being more aggressive with regards to terminating connections (to guard against denial of service attacks); or Jenkins might just be using the connections in a slightly different way.
We need to try to eliminate causes from your environment and hopefully come up with a simple reproducible testcase that makes it easy for someone to investigate/resolve the underlying problem.
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153882#comment-153882 ]
mark streit commented on JENKINS-11130:
---------------------------------------
OK...an update. We have eliminated Tomcat 6 vs Tomcat 7 as the cause. We have tried 6.0.33, 7.0.14, 7.0.22 - same error occurs on all versions.
Will update shortly on the results of a direct command line call outside of Jenkins using the bash shell command.
One other item of note: Our Hudson instance (older 1.339 running on Tomcat 6 using JDK 5 only). The Jenkins instance, 1.431 (on either Tomcat 6 or Tomcat 7 is using JDK 6). Unsure if the JDK would be playing any role here.
Thanks
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153884#comment-153884 ]
mark streit commented on JENKINS-11130:
---------------------------------------
More info: We tried your suggestion of running outside from the command line rather than from within Tomcat. Still facing the same error:
/opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://XX.XX.XX.XX:8082/jenkins/ build -s Build01_Build_App_WARs -p SVN_PATH=trunk -p STATIC_PATH=Rev_1.2
Output after about 15 secs or so:
PID_Hudson:/opt/jenkins/scripts> /opt/jenkins/scripts/build_rev_1.2_wars.sh
Oct 12, 2011 2:29:02 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://XX.XX.XX.XX:8082/jenkins/cli
java.io.IOException: _Unexpected termination of the channel_
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1093)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
Caused by: _java.io.EOFException_
at _java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)_
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=153885#comment-153885 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
I just tried an experiment here and confirmed that things seem to work ok for me.
* launching a single job from the cli works. I observed that a job taking 5+ minutes went ok.
java -jar ./jenkins-cli.jar -s http://XX.XX.XX.XX:8082/ build --username XXX --password-file ./password -s XXjobnameXX
* I created a freestyle job that executes a shell script that launches 2 other jobs via the cli. Then I launched the build from an external commandline invocation similar to the first example.
{code}
Started by command line
Building remotely on XXXXX
[jenkins11130] $ /bin/sh -xe /tmp/hudson5694867157318468244.sh
+ /usr/bin/java -jar /tmp/jenkins11130/jenkins-cli.jar -s http://XX.XX.XX.XX:8082/ build --username XXXX --password-file /tmp/jenkins11130/password -s XXjob1XXX
Completed XXjob1XXX #12 : SUCCESS
+ /usr/bin/java -jar /tmp/jenkins11130/jenkins-cli.jar -s http://XX.XX.XX.XX:8082/ build --username XXXX --password-file /tmp/jenkins11130/password -s XXjob2XXX
Completed XXjob2XXX #1213 : SUCCESS
Finished: SUCCESS
{code}
That was observed to work correctly.
Notes:
* Running jenkins 1.436 (actually Jenkins ver. 1.436-SNAPSHOT (private-10/11/2011 00:09 GMT-jenkins downloaded from ci.jenkins-ci.org earlier today)
* Running the native winstone jenkins container.
* Also tested against an apache reverse proxy to the same container
* Not tested with Tomcat at all
* Various java versions on each machine (Oracle 1.6.0_26-b03, IcedTea 6b20-1.9.9-0ubuntu1-10.04.2, IcedTea 6b22-1.10.2-0ubuntu1-11.04.1)
* jenkins-cli.jar downloaded from
wget http://XX.XX.XX.XX:8082/jnlpJars/jenkins-cli.jar
Unfortunately it does not help to solve your problem but it does show that it works at some level.
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154059#comment-154059 ]
mark streit commented on JENKINS-11130:
---------------------------------------
Richard,
Thank you very much for suggesting using Winstone as an alternative.
* We shut down our Tomcat 7 sever (we had also tried Tomcat 6 too - same Jenkins failure)...the only thing ever running on Tomcat for us was Jenkins.
* Started the jenkins.war using the embedded Winstone container instead. (Specified same port and number and context prefix)..so URLs would not change...
* Making NO CHANGES to our job configurations - IT WORKED CORRECTLY this time...as it does when run from the older Hudson 1.339 on Tomcat 6.
* We are now using Jenkins 1.434 (before trying Winstone, we tried Tomcat 7 with that version and it still fails)
Obviously there is something wrong when trying to do this when running Jenkins on Tomcat where the jenkins-cli is being used. Used with the Winstone container, it ran flawlessly. I would have to assume something was introduced into Jenkins that is causing this. The fact that we changed NOTHING but the runtime container points to something involving running on a different container.
Thank you again for your help...
Mark
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154084#comment-154084 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
I was experimenting with the new Ubuntu 1110 release yesterday and gave the new Jenkins/tomcat(6) packaging a try. It installed easily and I gave the test that I tried last week another try. It worked fine both with 1.409.1 and 1.434 Jenkins. Note I used a single master setup with no slaves.
I didn't try tomcat 7 because the Ubuntu packaging does not automatically setup Jenkins for that.
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154088#comment-154088 ]
mark streit commented on JENKINS-11130:
---------------------------------------
We have discovered something else specific to Jenkins CLI jar. Though your suggestion of running on Winstone worked well, we did discover these errors in the Console output of the build .... HOWEVER - they did NOT stop the build process from continuing:
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://xx.xx.xx.xx:8082/jenkins/ build XYZ_Project-Workflow -p SVN_PATH=branches/XYZ1.4 -p STATIC_PATH=Componentv1.1.2
Oct 19, 2011 9:01:59 AM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://xx.xx.xx.xx:8082/jenkins/cli
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
We CANNOT reproduce on the older Hudson 1.339 instance that uses the hudson-cli.jar. So in conclusion:
# Tomcat 6 or 7 WILL kill the process when we use the jenkins-cli.jar
# Winstone WILL NOT kill the process, but still is showing these EOF errors on every call being made, yet each job finishes and the controlling job waits properly until ALL are completed.
# Like you stated, we only have a single Master setup as well, w/ no slaves set up
# Right now, Ubuntu is not an option for us - we are using SLES9 - which will be upgraded to soon SLES11. We are constrained with this one at the moment.
# It seems that something in the CLI JAR between the older Hudson and newer Jenkins has changed that is causing this to occur.
Thanks
jordanh@zootweb.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jordanh updated JENKINS-11130:
------------------------------
Attachment: PrematureEofTest.java
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
jordanh@zootweb.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154867#comment-154867 ]
jordanh commented on JENKINS-11130:
-----------------------------------
I was going to open a bug but this sounds eerily like my problem. I have a little swing app that uses the Jenkins remote api/xml stuff to grab build statuses and then turn on/off our lavalamps. Well, this little swing app started throwing an IOException out of the blue the other day (hadn't upgraded Jenkins in weeks).
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2512)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2507)
at PrematureEofTest.testDownloadXml(PrematureEofTest.java:24)
...
It appears this exception occurs only when Jenkins sends an HTTP response with the "Transfer-Encoding=[chunked]" header field. When Jenkins does not send that header everything works fine. A little testing shows that when I grab the data for a single project Jenkins does not send that header. But when I grab the data for the entire Jenkins dashboard Jenkins sends that header.
It may be that if the XML being sent is over 8k that the chucked transfer encoding header is used and anything smaller than that it is not... but that is just some speculation based on some google searching.
Just a little bit more data: Using Jenkins 1.434, Apache 1.3.37, Tomcat 5.5 running on Java 1.6.0_20.
I am attaching a little unit test I have been using to repro the issue.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
jordanh@zootweb.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jordanh updated JENKINS-11130:
------------------------------
Attachment: PrematureEofTest.java
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
jordanh@zootweb.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154867#comment-154867 ]
jordanh edited comment on JENKINS-11130 at 11/2/11 10:14 PM:
-------------------------------------------------------------
I was going to open a bug but this sounds eerily like my problem. I have a little swing app that uses the Jenkins remote api/xml stuff to grab build statuses and then turn on/off our lavalamps. Well, this little swing app started throwing an IOException out of the blue the other day (hadn't upgraded Jenkins in weeks).
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2512)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2507)
at PrematureEofTest.testDownloadXml(PrematureEofTest.java:24)
...
It appears this exception occurs only when Jenkins sends an HTTP response with the "Transfer-Encoding=[chunked]" header field. When Jenkins does not send that header everything works fine. A little testing shows that when I grab the data for a single project Jenkins does not send that header. But when I grab the data for the entire Jenkins dashboard Jenkins sends that header.
It may be that if the XML being sent is over 8k that the chucked transfer encoding header is used and anything smaller than that it is not... but that is just some speculation based on some google searching.
Just a little bit more data: Using Jenkins 1.434, Apache 1.3.37, Tomcat 5.5 running on Java 1.6.0_20, no proxies, no jenkins slaves.
I am attaching a little unit test I have been using to repro the issue.
was (Author: jordanh):
I was going to open a bug but this sounds eerily like my problem. I have a little swing app that uses the Jenkins remote api/xml stuff to grab build statuses and then turn on/off our lavalamps. Well, this little swing app started throwing an IOException out of the blue the other day (hadn't upgraded Jenkins in weeks).
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2512)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2507)
at PrematureEofTest.testDownloadXml(PrematureEofTest.java:24)
...
It appears this exception occurs only when Jenkins sends an HTTP response with the "Transfer-Encoding=[chunked]" header field. When Jenkins does not send that header everything works fine. A little testing shows that when I grab the data for a single project Jenkins does not send that header. But when I grab the data for the entire Jenkins dashboard Jenkins sends that header.
It may be that if the XML being sent is over 8k that the chucked transfer encoding header is used and anything smaller than that it is not... but that is just some speculation based on some google searching.
Just a little bit more data: Using Jenkins 1.434, Apache 1.3.37, Tomcat 5.5 running on Java 1.6.0_20.
I am attaching a little unit test I have been using to repro the issue.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154869#comment-154869 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
Thanks for the unit test. I'll give it a try and see if I can make things break here.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154900#comment-154900 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
@jordanh I couldn't reproduce a failure with your testcase. But it did give me a little bit of food for thought and I have now reproduced @mark streit's problem using the cli. Basically if I force the connection to use a HTTP connection (rather than the cli port) then I can reproduce the problem. So I'm guessing that the cli port is not visible to the external cli program and hence it falls back to a HTTP based connection.
I can make the HTTP based connection work properly if I reduce the PingThread timeout from 30 seconds to 5 or 10 seconds. I'll generate a patch/pull request later today.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
jordanh@zootweb.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154902#comment-154902 ]
jordanh commented on JENKINS-11130:
-----------------------------------
That sounds like some progress. I want to reiterate that the use of the "chunked" transfer encoding header appears to be dynamic. In my testing it is always used when I try to read xml for the main Jenkins dashboard. That is probably due to the fact that I have around 90-100 projects in my instance of Jenkins right now. It seems that if the xml to be sent is large enough then the "chunked" header is used. Why it things go wonky when "chunked" is used is probably more the issue and it sounds like maybe you have an idea about that. Cheers.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154904#comment-154904 ]
mark streit commented on JENKINS-11130:
---------------------------------------
So my group's workaround was to move off Tomcat and use Winstone and the problem was solved. I don't like having to run on something with less control than Tomcat... but it works for now. As I have stated - something changed with Jenkins CLI because with Hudson CLI and older Hudson 1.339 and Tomcat 6.0.18 ... no problems...it worked. We moved to Jenkins and its CLI and started seeing this.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154919#comment-154919 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
@jordanh yes the "chunked" transfer is dynamic and that is part of HTTP/1.1. I'm going to describe my findings in detail and hopefully that will allow you to determine if you are seeing the same issue as here.
@mark streit I do not intend to dig around and find out how/why it works for older Hudson or Tomcat instances. I am sure that a few things have changed that could have avoided timeouts in the past. It could be as simple as the CLI client being able to connect via a direct socket rather than having to use HTTP.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154920#comment-154920 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
Short story - Tomcat times out the upload data connection. Jenkins sees this as the CLI client going away. Then Jenkins closes the download connection. CLI sees the download connection unexpectedly close and it generates the error message that we see.
Long story
# First Cli downloads the regular HTML base webpage of the Jenkins instance.
#* from this it looks at the HTTP headers and tries to find the {{X-Jenkins-CLI-Port}} or {{X-Hudson-CLI-Port}} headers.
#* in this case it does not find the header or maybe it fails to connect due to firewall/ip address binding issues.
# At this point the CLI client falls-back to a full duplex connection using two HTTP connections: one for upload and one for download of data. Both of these use HTTP POST requests with chunked encoding out of necessity.
#* the download side POSTs a small amount of setup data in the HTTP headers and a zero length data stream. Then the server sends a 200 response code and starts sending remote protocol data to the CLI client.
#* The upload side does a HTTP POST and sends the chunked remote protocol data as the post data body.
# From this point the remote protocol uses the stream to pass serialized code/data between CLI client and Jenkins server.
# The thing that we must remember is that these are just regular webserver HTTP connections and the Tomcat/Winstone/whatever is not aware that there is any special significance in these streams and hence it applies the same logic that any HTTP connection would get. In particular inactivity timeouts are applied.
# In this case the upload side (which is technically just sending HTTP POST data) is subject to timeouts waiting for the full data to arrive. Without these timeouts clients could mount a denial of service attack against the webserver.
So once the build has started the CLI client has no need to send any data to the server so the upload side sits idle. After 30 seconds Tomcat timesout the upload and we end up with an error.
This is where thing get complicated because the error reported is that the download side closed unexpectedly. To explain this we have to consider the whole sequence of failure events:
* cli upload side. Is sat idle and does/sees nothing. If it did try to send data it would find that the upstream side is closed and that it has a HTTP 200 response/body to deal with.
* server upload side. Jenkins sees an EOF from the client in the upload data stream. This is effectively faked by Tomcat because it has timed-out out the upload. Indeed we get an error in the Jenkins log to that effect.
{quote}
03-Nov-2011 19:19:31 hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel HTTP full-duplex channel 6e5b90d4-767d-4f8b-b1d8-9c
ec92d539d2
java.net.SocketTimeoutException: Read timed out
{quote}
* At this point Jenkins starts to close down the download side of the connection because the client has gone away.
* Now the CLI download side sees what to it is an unexpected EOF because Jenkins just closes the connection.
Hence the net result is that the CLI client thinks that the server has broken but this was actually caused by inactivity in the upload datastream from itself!
So to resolve the issue Jenkins needs to issue a remoting keepalive ping or the timeouts in Tomcat need to be increased.
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154921#comment-154921 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
One workaround is to set the Tomcat Connector connectionUploadTimeout and disableUploadTimeout attributes in the Tomcat server.xml file. Setting it to 60 seconds allows the existing ping at 30 seconds duration to work properly.
{code}
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
disableUploadTimeout="false"
connectionUploadTimeout="60000"
URIEncoding="UTF-8"
redirectPort="8443" />
{code}
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Mortimer reassigned JENKINS-11130:
------------------------------------------
Assignee: Richard Mortimer
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Assignee: Richard Mortimer
> Attachments: PrematureEofTest.java, PrematureEofTest.java
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
oldelvet@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=154924#comment-154924 ]
Richard Mortimer commented on JENKINS-11130:
--------------------------------------------
https://github.com/jenkinsci/jenkins/pull/304
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Assignee: Richard Mortimer
> Attachments: PrematureEofTest.java, PrematureEofTest.java
>
>
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
scm_issue_link@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
scm_issue_link resolved JENKINS-11130.
--------------------------------------
Resolution: Fixed
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Assignee: Richard Mortimer
> Attachments: PrematureEofTest.java, PrematureEofTest.java
>
>
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
scm_issue_link@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=155126#comment-155126 ]
scm_issue_link commented on JENKINS-11130:
------------------------------------------
Code changed in jenkins
User: Richard Mortimer
Path:
changelog.html
cli/src/main/java/hudson/cli/CLI.java
http://jenkins-ci.org/commit/jenkins/92496f99e4c941de9d7dab628466cadcde70d808
Log:
[FIXED JENKINS-11130] SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar
When using HTTP full duplex connections the upstream channel can be idle for
long periods of time and the webserver will time the connection out. Jenkins
sees this and terminates the downstream channel causing the unexpected closure
of the read channel in the CLI client.
Timeout of 15 seconds is based on the 20 second default timeout as seen in
Ubuntu Oneiric Tomcat 6 installations.
Compare: https://github.com/jenkinsci/jenkins/compare/9730359...92496f9
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Assignee: Richard Mortimer
> Attachments: PrematureEofTest.java, PrematureEofTest.java
>
>
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
dogfood@java.net (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=155137#comment-155137 ]
dogfood commented on JENKINS-11130:
-----------------------------------
Integrated in !http://ci.jenkins-ci.org/images/16x16/yellow.png! [jenkins_main_trunk #1271|http://ci.jenkins-ci.org/job/jenkins_main_trunk/1271/]
[FIXED JENKINS-11130] SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar
Kohsuke Kawaguchi : [92496f99e4c941de9d7dab628466cadcde70d808|https://github.com/jenkinsci/jenkins/commit/92496f99e4c941de9d7dab628466cadcde70d808]
Files :
* changelog.html
* cli/src/main/java/hudson/cli/CLI.java
> java.io.IOException: Unexpected termination of the channel - SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar (works on older Hudson version)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JENKINS-11130
> URL: https://issues.jenkins-ci.org/browse/JENKINS-11130
> Project: Jenkins
> Issue Type: Bug
> Components: cli
> Affects Versions: current
> Environment: * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
> * Tomcat 7.0.14
> * JDK 1.6.0_14
> * Only ONE Master configuration - no slaves are configured
> * 3 Executors - (one less than the max number of CPUs)
> Reporter: mark streit
> Assignee: Richard Mortimer
> Attachments: PrematureEofTest.java, PrematureEofTest.java
>
>
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
mcs130@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-11130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=156349#comment-156349 ]
mark streit commented on JENKINS-11130:
---------------------------------------
We have downloaded and installed the LATEST 1.441 release that lists the fix for this problem. Currently we were running 1.438 on Winstone only (since with Tomcat 6 or 7, we had experienced the error HOWEVER yet under Winstone, it worked OK so that was our workaround - while running 1.438).
Now with Jenkins 1.441 - we are getting the ERROR again and NOW WITH BOTH Winstone and the Tomcat configurations). We have left the Jenkins 1.441 WAR file in place running on Winstone, and reverted the CLI jar file back to the 1.438 version for now and that appears to work again with Winstone.
Checked Manifest of CLI jar downloaded with the 1.441 WAR installation:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: kohsuke
Build-Jdk: 1.6.0_26
Main-Class: hudson.cli.CLI
Jenkins-CLI-Version: 1.441
Under Tomcat 7, we get this stacktrace:
Started by command line
[workspace] $ /bin/bash -xe /opt/apache-tomcat-7.0.22_jenkins/temp/hudson32817888834817830.sh
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
Dec 5, 2011 12:59:11 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:681)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
at $Proxy2.main(Unknown Source)
at hudson.cli.CLI.execute(CLI.java:200)
at hudson.cli.CLI._main(CLI.java:330)
at hudson.cli.CLI.main(CLI.java:245)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Build step 'Execute shell' marked build as failure
Notifying upstream projects of job completion
Finished: FAILURE
Under Winstone, we get this stacktrace - it's somewhat different:
Started by command line
[workspace] $ /bin/bash -xe /tmp/hudson10791816374444704.sh
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
Dec 5, 2011 1:18:22 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:681)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
at $Proxy2.main(Unknown Source)
at hudson.cli.CLI.execute(CLI.java:200)
at hudson.cli.CLI._main(CLI.java:330)
at hudson.cli.CLI.main(CLI.java:245)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
Caused by: java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)