[JIRA] [core] (JENKINS-30869) After upgrade from 1.574 to 1.609.3, cannot connect to Linux slave

4 views
Skip to first unread message

shannonckerr@gmail.com (JIRA)

unread,
Oct 9, 2015, 2:55:02 PM10/9/15
to jenkinsc...@googlegroups.com
Shannon Kerr created an issue
 
Jenkins / Bug JENKINS-30869
After upgrade from 1.574 to 1.609.3, cannot connect to Linux slave
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 09/Oct/15 6:54 PM
Environment: Host: Ubuntu 12.04 x64 Server
Slave: Linux freescale 2.6.31 armv5tejl
Labels: Slave connection failure
Priority: Major Major
Reporter: Shannon Kerr

Sometimes I see this error:

[10/09/15 14:38:27] [SSH] Opening SSH connection to 10.6.5.9:22.
[10/09/15 14:38:28] [SSH] Authentication successful.
[10/09/15 14:38:28] [SSH] The remote users environment is:
HOME='/home/myaccount'
IFS='
'
LOGNAME='myaccount'
OPTIND='1'
PATH='/usr/bin:/bin'
PPID='9230'
PS1='\w \$ '
PS2='> '
PS4='+ '
PWD='/home/myaccount'
SHELL='/bin/sh'
USER='myaccount'
[10/09/15 14:38:28] [SSH] Starting sftp client.
[10/09/15 14:38:28] [SSH] SFTP failed. Copying via SCP.
[10/09/15 14:38:28] [SSH] Copying latest slave.jar...
Expanded the channel window size to 4MB
[10/09/15 14:38:28] [SSH] Starting slave process: cd "/home/myaccount" && /opt/jre/bin/java -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 2.52
This is a Unix slave
Segmentation fault
ERROR: Connection terminated
java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
ERROR: Failed to install restarter
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:575)
at hudson.remoting.Request.call(Request.java:130)
at hudson.remoting.Channel.call(Channel.java:777)
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller.install(JnlpSlaveRestarterInstaller.java:52)
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller.access$000(JnlpSlaveRestarterInstaller.java:33)
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$1.call(JnlpSlaveRestarterInstaller.java:39)
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$1.call(JnlpSlaveRestarterInstaller.java:36)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Slave successfully connected and online
ERROR: Failed to update jenkins-slave.exe
Slave JVM has terminated. Exit code=139
java.io.IOException: remote file operation failed: /home/myaccount/jenkins-slave.exe at hudson.remoting.Channel@3902ea91:Freescale-5.9: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.FilePath.act(FilePath.java:985)
at hudson.FilePath.act(FilePath.java:967)
at hudson.FilePath.exists(FilePath.java:1435)
at org.jenkinsci.modules.windows_slave_installer.SlaveExeUpdater$1.call(SlaveExeUpdater.java:46)
at org.jenkinsci.modules.windows_slave_installer.SlaveExeUpdater$1.call(SlaveExeUpdater.java:37)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:575)
at hudson.remoting.Request.call(Request.java:130)
at hudson.remoting.Channel.call(Channel.java:777)
at hudson.FilePath.act(FilePath.java:978)
... 9 more
Caused by: java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
[10/09/15 14:39:03] [SSH] Connection closed.

Sometimes I see this, though:

[10/09/15 14:45:28] [SSH] Starting sftp client.
[10/09/15 14:45:28] [SSH] SFTP failed. Copying via SCP.
[10/09/15 14:45:28] [SSH] Copying latest slave.jar...
Expanded the channel window size to 4MB
[10/09/15 14:45:28] [SSH] Starting slave process: cd "/home/myaccount" && /opt/jre/bin/java -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 2.52
This is a Unix slave
Segmentation fault
ERROR: Connection terminated
java.io.IOException: Unexpected EOF
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
java.lang.NullPointerException
at org.jenkinsci.modules.systemd_slave_installer.SlaveInstallerFactoryImpl.createIfApplicable(SlaveInstallerFactoryImpl.java:33)
at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createIfApplicable(SlaveInstallerFactory.java:29)
at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createFor(SlaveInstallerFactory.java:46)
at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:30)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:563)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:371)
at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:945)
at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:133)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:711)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:696)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[10/09/15 14:46:04] Launch failed - cleaning up connection
[10/09/15 14:46:04] [SSH] Connection closed.

Before I updated to 1.609.3, it worked without issue.

Java is: java version 1.7.0_60
Java SE Embedded Runtime Environment (build 1.7.0_60-b19, headless)
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

dbeck@cloudbees.com (JIRA)

unread,
Oct 15, 2015, 4:59:01 AM10/15/15
to jenkinsc...@googlegroups.com
Daniel Beck updated an issue
Change By: Daniel Beck
Component/s: remoting
Component/s: core

jsposato@mobiquityinc.com (JIRA)

unread,
Oct 15, 2015, 10:52:02 AM10/15/15
to jenkinsc...@googlegroups.com
John Sposato commented on Bug JENKINS-30869
 
Re: After upgrade from 1.574 to 1.609.3, cannot connect to Linux slave

I am seeing similar behavior on a CentOS 7 slave from Jenkins 1.633. Job errors during archiving with this:

ERROR: Failed to archive artifacts: android/app/build/outputs/apk/*.apk
java.io.IOException: java.io.IOException: Failed to extract /var/lib/jenkins/workspace/Wawa_Android_QA/transfer of 1 files
	at hudson.FilePath.readFromTar(FilePath.java:2301)
	at hudson.FilePath.copyRecursiveTo(FilePath.java:2210)
	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
	at hudson.model.Build$BuildExecution.post2(Build.java:185)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
	at hudson.model.Run.execute(Run.java:1766)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: java.io.IOException: Unexpected EOF
	at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:169)
	at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
	at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175)
	at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614)
	at java.io.InputStream.read(InputStream.java:101)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
	at hudson.util.IOUtils.copy(IOUtils.java:40)
	at hudson.FilePath.readFromTar(FilePath.java:2291)
	... 13 more
Caused by: java.io.IOException: Unexpected EOF
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

	at hudson.FilePath.copyRecursiveTo(FilePath.java:2217)
	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:219)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
	at hudson.model.Build$BuildExecution.post2(Build.java:185)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
	at hudson.model.Run.execute(Run.java:1766)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:408)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
	at hudson.remoting.Request$1.get(Request.java:276)
	at hudson.remoting.Request$1.get(Request.java:207)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
	at hudson.FilePath.copyRecursiveTo(FilePath.java:2213)
	... 12 more
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
	at hudson.remoting.Request.abort(Request.java:297)
	at hudson.remoting.Channel.terminate(Channel.java:844)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected EOF
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Build step 'Archive the artifacts' changed build result to FAILURE

and slave log has this:

ERROR: Connection terminated
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

jsposato@mobiquityinc.com (JIRA)

unread,
Oct 22, 2015, 9:52:02 AM10/22/15
to jenkinsc...@googlegroups.com

we use the EC2 plugin and I was able to solve my problems by checking the option 'Connect by SSH Process'. The slaves have been 100% stable since that point.

shannonckerr@gmail.com (JIRA)

unread,
Oct 22, 2015, 10:47:01 AM10/22/15
to jenkinsc...@googlegroups.com

Thanks for the updated John. We do not use EC2 here. Still have this problem. All slaves except for the freescale one are stable. My team is blocked from using Jenkins for testing until this is fixed. I hope to see something soon.

dbeck@cloudbees.com (JIRA)

unread,
Oct 23, 2015, 2:06:21 AM10/23/15
to jenkinsc...@googlegroups.com

1.574 -> 1.609.3 is a bit too much of a jump to investigate the possible cause without having access to the affected machine – it's ~3.5 LTS releases!

Please try to determine the specific release that broke this, and provide that information.

shannonckerr@gmail.com (JIRA)

unread,
Jan 26, 2016, 9:48:01 AM1/26/16
to jenkinsc...@googlegroups.com

This is now fixed when using Jenkins 1.642.1. I verified again that it was still broken with 1.609.3. I'm satisfied that the issue is resolved and I'm good to close the ticket. Daniel, are you good with me closing the ticket? I will not have time to isolate this down to a more-specific version of Jenkins.

dbeck@cloudbees.com (JIRA)

unread,
Jan 26, 2016, 10:04:03 AM1/26/16
to jenkinsc...@googlegroups.com
Daniel Beck resolved as Cannot Reproduce
 

Shannon Kerr Thanks for the update!

Not an issue on current LTS, therefore resolving as Cannot Reproduce.
Change By: Daniel Beck
Status: Open Resolved
Resolution: Cannot Reproduce
Reply all
Reply to author
Forward
0 new messages