SSH Slave crashing immediately after start

[Peter Berghold]

I'm trying to re-establish connectivity to an SSH slave that used to work when I was running a much older version of Jenkins.  Currently I'm using v2.65. What I'm seeing (somewhat trimmed) in the log is the following:

[06/13/17 20:04:05] [SSH] Checking java version of java
[06/13/17 20:04:05] [SSH] java -version returned 1.7.0_75.
[06/13/17 20:04:05] [SSH] Starting sftp client.
[06/13/17 20:04:05] [SSH] Copying latest slave.jar...
[06/13/17 20:04:05] [SSH] Copied 719,269 bytes.
Expanded the channel window size to 4MB
[06/13/17 20:04:05] [SSH] Starting slave process: cd "/data/jenkins-slave" && java  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave JVM has not reported exit code. Is it still running?
[06/13/17 20:04:09] Launch failed - cleaning up connection
[06/13/17 20:04:09] [SSH] Connection closed.
ERROR: Connection terminated
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)

At a loss...

[Mark Waite]

Is your older slave machine running the same Java version as your master?  If not, you may see those types of problems.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAArvnv2jpH-nq1fxzarHa%2BZGMoCdN4nfCFqhf8%2BVbxRcgGGSqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[James Nord]

> java -version returned 1.7.0_75.

Jenkins 2.58+ only support java8 or higher, this includes your agents JVM.

/James

[Peter Berghold]

Okay I guess I'm going to have to investigate how to get the newer version of Java on Redhat. Redhat 6 at that. Thanks for the pointers.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ea6f2611-da6f-448d-9065-2e23fdf29a85%40googlegroups.com.

[Peter Berghold]

Changing the version of Java to 1.8 did the trick.  The slave is back online.

[yoyuli...@gmail.com]

Hi guys, 

     I have the same problem on Jenkins ver. 2.164.1, Selenium Grid 3.12.0 and when start the slave in console will throw me this error:

Expanded the channel window size to 4MB
[04/02/19 09:05:15] [SSH] Starting slave process: cd "/var/jenkins" && /usr/lib/jvm/jdk1.8.0_191/bin/java -Xmx128m -Xms128m -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.29
This is a Unix agent
Evacuated stdout
Starting Selenium nodes on automation-jenkins-node
Apr 02, 2019 9:05:17 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class hudson.plugins.selenium.configuration.DirectJsonInputConfiguration$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
hudson.util.IOException2: Slave JVM has not reported exit code. Is it still running?
	at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:984)
	at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:137)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:725)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:706)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to copy /var/lib/jenkins/plugins/selenium/WEB-INF/lib/selenium-server-standalone-3.12.0.jar to /var/jenkins/selenium-server-standalone-3.12.0.jar
	at hudson.FilePath.copyTo(FilePath.java:2191)


Could someone help me? 

Thank you!

[yoyuli...@gmail.com]

Note:  The user with which make ssh have the rights on that folders

Java Version: On Master(where is the hub) is: openjdk version "1.8.0_191"

and on Slave: java version "1.8.0_191"

On Tuesday, June 13, 2017 at 11:05:40 PM UTC+3, Peter Berghold wrote:

[Ivan Fernandez Calvo]

It is not the same kind of issue, in you case it failed to copy a file to you agent, check that you have disk space on the agent in the path /var/jenkins, check if that path exists, check that the user used by Jenkins to connect has permissions on that folder, check the folder is writable

[Ivan Fernandez Calvo]

I never used selenium grid plugin, but maybe 128MB it is too right, try larger (512 or 256) values and decrease it to the minimum that works in a stable way

[Ivan Fernandez Calvo]

Also check the agent working directory you should have log files and perhaps some error_pid files

[yoyuli...@gmail.com]

Hi Ivan, Thank you for your answer.

    

     I have checked several stuff, but with no success. The user who runs under this ssh slave has permissions on that folder, the folder is writable. I have tried to copy by myself under that user and it is working: scp -r /var/lib/jenkins/plugins/selenium/WEB-INF/lib/selenium-server-standalone-3.12.0.jar jenkins@<slaveHost>:/var/jenkins/selenium-server-standalone-3.12.0.jar.

I increased and decreased the xmx and xms and the same.
I also checked the logs.

[kuisathaverat]

Is nothing in the AGENT_WORKDIR/remoting folder logs? without more info, I cannot help you too much, check the troubleshooting guide https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md there you have details about what files are needed to check an issue.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/13mD0qN5rD4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/41bf369c-0fe0-4588-ac14-7b625c0505cb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Un Saludo

Iván Fernández Calvo

https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033

[yoyuli...@gmail.com]

Hi kuisathaverat,

   

    I upgraded the ssh slave plugin from 1.11 to 1.29 and now the error has changed. In the AGENT_WORKDIR/logs/remoting diplay this error:

Apr 03, 2019 9:46:01 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

WARNING: Attempt to (de-)serialize anonymous class hudson.plugins.selenium.configuration.DirectJsonInputConfiguration$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/

Apr 03, 2019 9:46:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel channel

java.io.IOException: Unexpected termination of the channel

        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
        at hudson.remoting.Command.readFrom(Command.java:140)
        at hudson.remoting.Command.readFrom(Command.java:126)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

The ssh on the agent was successfully done and it also copied the remoting jar, but I have some doubts that the problem is when it tries to start selenium-server, and uses json configuration for a node.

Thank you.

[kuisathaverat]

Which versions of the selenium grid plugin do you use before and after? maybe in the release notes, there is some change that affects you.

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/13mD0qN5rD4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a66ba857-e9df-4dad-879f-556687e29ed1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[yoyuli...@gmail.com]

On both I am using selenium Grid 3.12.0.

[yoyuli...@gmail.com]

But what is tricky is that on the other environment I am using other version of jenkins (2.64), selenium 3.12 and SSH slave 1.11 and it's working......

[kuisathaverat]

to discard things, Could you connect the agent as a regular ssh-agent and run a silly job in it, something like the following pipeline ;), if it works it is something in the selenium grid plugin

```

node('my-rebel-agent'){

   sh(label: "the most complex thing in the world", script: "hello world")

}

```

El mié., 3 abr. 2019 a las 15:29, <yoyuli...@gmail.com> escribió:

But what is tricky is that on the other environment I am using other version of jenkins (2.64), selenium 3.12 and SSH slave 1.11 and it's working......

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/13mD0qN5rD4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0c42a2dc-c4a6-46d5-96ba-e1c4c52511e0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[yoyuli...@gmail.com]

Yes, if I don't put any selenium json configuration, the connection through ssh, between master and node works :)

[Ivan Fernandez Calvo]

I've found something weird, you 've said that your ssh-slaves-plugin version is 1.29, but the stack trace points to a method named SSHLauncher.startSlave, this method was renamed before that version and now is this one https://github.com/jenkinsci/ssh-slaves-plugin/blob/ssh-slaves-1.29.0/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L1018 SSHLauncher.startAgent, the line of code also seems not correct to that version, Did you restart Jenkins after installing the plugins? if so, check 
ssh-slaves.jpi file date in JENKINS_HOME/plugins folder 

[yoyuli...@gmail.com]

The first error that I put here is with SSH Slave version 1.11

Expanded the channel window size to 4MB
[04/02/19 09:05:15] [SSH] Starting slave process: cd "/var/jenkins" && /usr/lib/jvm/jdk1.8.0_191/bin/java -Xmx128m -Xms128m -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.29
This is a Unix agent
Evacuated stdout
Starting Selenium nodes on automation-jenkins-node
Apr 02, 2019 9:05:17 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class hudson.plugins.selenium.configuration.DirectJsonInputConfiguration$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/

hudson.util.IOException2: Slave JVM has not reported exit code. Is it still running?
	at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:984)
	at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:137)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:725)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:706)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.

java:748)
Caused by: java.io.IOException: Failed to copy /var/lib/jenkins/plugins/selenium/WEB-INF/lib/selenium-server-standalone-3.12.0.jar to /var/jenkins/selenium-server-standalone-3.12.0.jar
	at hudson.FilePath.copyTo(FilePath.java:2191)


And the last error, is with SSH Slave 1.29.4

Apr 03, 2019 9:46:01 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

WARNING: Attempt to (de-)serialize anonymous class hudson.plugins.selenium.configuration.DirectJsonInputConfiguration$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Apr 03, 2019 9:46:06 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel channel

java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
        at hudson.remoting.Command.readFrom(Command.java:140)
        at hudson.remoting.Command.readFrom(Command.java:126)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

After I updated the ssh-slave plugin to 1.29.4, I have checked in JENKINS_HOME/plugins folder.

[yoyuli...@gmail.com]

I fixed this problem through disable: "Manage Jenkins" > "Configure Global Security", and check "Enable Agent → Master Access Control" (as it's said in Jenkin documentation)

https://wiki.jenkins.io/display/JENKINS/Slave+To+Master+Access+Control

"On the other hand, if all your agents are trusted to the same degree as your master, then it is safe to leave this subsystem off"

Thanks for support

[kuisathaverat]

I will add it to the troubleshooting guide for future users, thx for sharing

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/13mD0qN5rD4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/204b7bda-fd2d-4564-960d-6df9e1a30518%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[yoyuli...@gmail.com]

Thank you too :)