Cannot start slave on IBM i due to java.io.EOFException

279 views
Skip to first unread message

Thomas R

unread,
Jun 16, 2019, 10:33:13 AM6/16/19
to Jenkins Users
Hi all,

I am new to Jenkins and I am lost. The problem is that I cannot start a Jenkins slave on an IBM i (former: AS/400, iSeries, etc.) due to a java.io.EOFException.

The log file and the console output does not contain anything meaningful to me.

Jenkins version: 2.164.3 (same problem with 2.176.1)

Jenkins log when starting the slave node:

SSHLauncher{host='obfuscated.host.com', port=22, credentialsId='beb60c8b-21d2-4f36-8f17-b594fb421d96', jvmOptions='-Dfile.encoding=ISO8859_1', javaPath='/QIBM/ProdData/Java400/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd=' -text', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[06/16/19 10:14:13] [SSH] Opening SSH connection to obfuscated.host.com:22.
[06/16/19 10:14:15] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
[06/16/19 10:14:15] [SSH] Authentication successful.
[06/16/19 10:14:16] [SSH] The remote user's environment is:
HOME=/home/raddatz
IFS=


LOGIN=raddatz
LOGNAME=raddatz
MAIL=/var/spool/mail/raddatz
MAILCHECK=600
OPTIND=1
PATH=/QOpenSys/usr/bin:/usr/ccs/bin:/QOpenSys/usr/bin/X11:/usr/sbin:.:/usr/bin
SHELL=/QOpenSys/usr/bin/bsh
SSH_CLIENT=10.214.1.158 59593 22
SSH_CONNECTION=10.214.1.158 59593 10.200.36.33 22
TZ=<CET>-1<CEST>,M3.5.0,M10.5.0/03:00:00
USER=raddatz
[06/16/19 10:14:17] [SSH] Starting sftp client.
[06/16/19 10:14:18] [SSH] Copying latest remoting.jar...
Source agent hash is D2D1A740134BD20D6F0855B356344342. Installed agent hash is D2D1A740134BD20D6F0855B356344342
Verified agent jar. No update is necessary.
Expanded the channel window size to 4MB
[06/16/19 10:14:40] [SSH] Starting agent process: cd "/home/Raddatz/jenkins" && /QIBM/ProdData/Java400/bin/java -Dfile.encoding=ISO8859_1 -jar remoting.jar -workDir /home/raddatz/jenkins/work -text
Slave JVM has terminated. Exit code=1
[06/16/19 10:15:12] Launch failed - cleaning up connection
[06/16/19 10:15:13] [SSH] Connection closed.


Job log 1 on IBM i:

File  . . . . . :   QPRINT                                                                               Page/Line   1/6
Control . . . . .                                                                                        Columns     1 - 130
Find  . . . . . .
*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3
Running in text mode
Both error and output logs will be printed to /home/raddatz/jenkins/work/remoting
WARNING: Are you running agent from an interactive console?
If so, you are probably using it incorrectly.
<===[JENKINS REMOTING CAPACITY]===>rO0ABXNyABpodWRzb24ucmVtb3RpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAAAAAAf4=<===[HUDSON
RANSMISSION BEGINS]===>


Job log 2 on IBM i:

File  . . . . . :   QPRINT                                                                               Page/Line   1/6
Control . . . . .                                                                                        Columns     1 - 130
Find  . . . . . .
*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3
Jun 16, 2019 10:14:50 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/raddatz/jenkins/work/remoting as a remoting work directory
Exception in thread "main" java.io.EOFException: unexpected stream termination
 at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
 at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
 at hudson.remoting.Launcher.main(Launcher.java:743)
 at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:691)
 at hudson.remoting.Launcher.run(Launcher.java:373)
 at hudson.remoting.Launcher.main(Launcher.java:283)


Java version on IBM i:

java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 8.0.5.30 - pap3280sr5fp30-20190207_01(SR5 FP30))
IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc-32-Bit 20190124_408237 (JIT enabled, AOT enabled)
OpenJ9   - 9c77d86
OMR      - dad8ba7
IBM      - e2996d1)
JCL - 20190207_01 based on Oracle jdk8u201-b09


Any comments, thoughts or ideas would be greatly appreciated.

Thanks,

Thomas.

Ivan Fernandez Calvo

unread,
Jun 16, 2019, 2:18:27 PM6/16/19
to Jenkins Users
Which JDK do you use on the Jenkins master?
Did you try to execute the following command manually on the agent?
cd "/home/Raddatz/jenkins" && /QIBM/ProdData/Java400/bin/java -Dfile.encoding=ISO8859_1 -jar remoting.jar -workDir /home/raddatz/jenkins/work -text

check the directory /home/raddatz/jenkins and /home/raddatz/jenkins/work for log files or hs_err_pid files

Finally check the troubleshooting guide https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md#troubleshooting

Ivan Fernandez Calvo

unread,
Jun 16, 2019, 2:25:12 PM6/16/19
to Jenkins Users
Another question, what is the use of ‘-text’ option you pass to the JVM? I did no find it on the Oracle JDK either on the IBM JDK

Ivan Fernandez Calvo

unread,
Jun 16, 2019, 2:41:42 PM6/16/19
to Jenkins Users
I've found it, is a remoting option to use base64 to encode the comunication, I've never use it, it is not needed
https://github.com/jenkinsci/remoting/blob/005dd67af809881ff0c79992583661fa2cde4229/src/main/java/hudson/remoting/Launcher.java#L130

Thomas R

unread,
Jun 16, 2019, 3:27:35 PM6/16/19
to Jenkins Users
There are only the following files:

a) \\ghentw.gfd.de\home\Raddatz\jenkins\remoting.jar
b) \\ghentw.gfd.de\home\Raddatz\jenkins\work\remoting\logs\remoting.log.0    (0-bytes = empty)

The "cd "/home/Raddatz/jenkins" && /QIBM/ProdData/Java400/bin/java -Dfile.encoding=ISO8859_1 -jar remoting.jar -workDir /home/raddatz/jenkins/work -text" command produces the following output, when being executed from Qshell on the host:

> cd "/home/Raddatz/jenkins" && /QIBM/ProdData/Java400/bin/java -Dfile.encoding=ISO8859_1 -jar remoting.jar -workDir               
     /home/raddatz/jenkins/work -text                                                                                              
  Running in text mode                                                                                                             
  Jun 16, 2019 9:18:52 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir                                           
  INFO: Using /home/raddatz/jenkins/work/remoting as a remoting work directory                                                     
  Both error and output logs will be printed to /home/raddatz/jenkins/work/remoting                                                
  WARNING: Are you running agent from an interactive console?                                                                      
  If so, you are probably using it incorrectly.                                                                                    
  See https://wiki.jenkins.io/display/JENKINS/Launching+agent+from+console                                                         
  <===[JENKINS REMOTING CAPACITY]===>rO0ABXNyABpodWRzb24ucmVtb3RpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAAAAAAf4=<===[HUDSO
  N TRANSMISSION BEGINS]===>                                                                                                       

It also produces an empty 'remoting.log.0'.

There are no hs_err_pid files in folder /home/raddatz/jenkis and any deeper folder.

I am going to check the troubleshooting page , now.

Thank you for your quick reply.

Thoma.s

Thomas R

unread,
Jun 16, 2019, 3:34:15 PM6/16/19
to Jenkins Users
I forgot to mention that I start the Jenkins server with the following command:

"c:\Program Files\IBM\SDP_096\jdk\bin\java.exe" -jar jenkins-2.164.3.war --httpPort=9090

The Java version number is:

C:\Users\traddatz>"c:\Program Files\IBM\SDP_096\jdk\bin\java.exe" -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 8.0.5.20 - pwa6480sr5fp20-20180802_01(SR5 FP20))
IBM J9 VM (build 2.9, JRE 1.8.0 Windows 7 amd64-64-Bit Compressed References 20180731_393394 (JIT enabled, AOT enabled)
OpenJ9   - bd23af8
OMR      - ca1411c
IBM      - 98805ca)
JCL - 20180719_01 based on Oracle jdk8u181-b12


Thomas.


Am Sonntag, 16. Juni 2019 20:18:27 UTC+2 schrieb Ivan Fernandez Calvo:

Thomas R

unread,
Jun 16, 2019, 4:01:58 PM6/16/19
to Jenkins Users
Changing Java propertis Xms and Xms (trouble shooting guide) did not help:

[06/16/19 21:52:55] [SSH] Starting agent process: cd "/home/Raddatz/jenkins" && /QOpenSys/usr/bin/java -Dfile.encoding=ISO8859_1  -Xms256M  -Xmx256M -jar remoting.jar -workDir /home/Raddatz/jenkins/work

Slave JVM has terminated. Exit code=1


Also enabling

Thomas R

unread,
Jun 16, 2019, 4:13:10 PM6/16/19
to Jenkins Users
Enabling "Master Access Control" (Enable Agent → Master Access Control)  and removing the '-text parameter did not help:

[06/16/19 22:10:24] [SSH] Starting agent process: cd "/home/Raddatz/jenkins" && /QOpenSys/usr/bin/java -Dfile.encoding=ISO8859_1  -Xms256M  -Xmx256M -jar remoting.jar -workDir /home/Raddatz/jenkins/work

Slave JVM has terminated. Exit code=1


Thomas.

Ivan Fernandez Calvo

unread,
Jun 16, 2019, 4:35:53 PM6/16/19
to Jenkins Users

Could you enable verbose loggin for the package "hudson.remoting"? FINE log level should enough see how at https://github.com/jenkinsci/remoting/blob/master/docs/logging.md
Which version os SSH do you use? execute `ssh -V` 

Thomas R

unread,
Jun 17, 2019, 7:29:41 AM6/17/19
to Jenkins Users
The SSH version on the IBM i server, that is where the slave should run, is:

> ssh -V
  OpenSSH_6.9p1, OpenSSL 1.0.2r  26 Feb 2019
  $

Enabling logging on the slave did not help. All I get is an empty log file 'remoting.log.0', when using the following logger configuration file:

# Configuration file that controls the slave logging ... if we pass in a JVM
# arg -Djava.util.logging.config.file=jenkins-slave.logging.properties

# Send output to file as well as console.
handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# Log everything equally
java.util.logging.FileHandler.level=ALL
java.util.logging.ConsoleHandler.level=ALL
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.SimpleFormatter.format=%1$tFT%1$tT.%1$tL%1$tz %4$s %2$s: %5$s%6$s%n

# On the filesystem, we keep up to 9 logs of 10meg each.
java.util.logging.FileHandler.pattern=jenkins-slave.%g.log
java.util.logging.FileHandler.encoding=UTF-8
java.util.logging.FileHandler.limit=10485760
java.util.logging.FileHandler.count=9
java.util.logging.FileHandler.append=false

# Log everything ...
.level= ALL
hudson.remoting.level=FINE


This is who I specified the logger configuration of the slave:

Source agent hash is D2D1A740134BD20D6F0855B356344342. Installed agent hash is D2D1A740134BD20D6F0855B356344342
Verified agent jar. No update is necessary.
Expanded the channel window size to 4MB
[06/17/19 13:24:16] [SSH] Starting agent process: cd "/home/Raddatz/jenkins" && /QOpenSys/usr/bin/java -Dfile.encoding=ISO8859_1   -jar remoting.jar -workDir /home/Raddatz/jenkins/work -loggingConfig jenkins-slave.logging.properties

Slave JVM has terminated. Exit code=1
[06/17/19 13:24:41] Launch failed - cleaning up connection
[06/17/19 13:24:41] [SSH] Connection closed.

Thomas.

Ivan Fernandez Calvo

unread,
Jun 17, 2019, 9:41:38 AM6/17/19
to Jenkins Users
Can you execute a command on a remote ssh session on the agent? something like the following commands `ssh USER@JENKINS_AGENT echo "OK"`, `ssh -t USER@JENKINS_AGENT echo "OK"`, `ssh -T USER@JENKINS_AGENT echo "OK"`

Thomas R

unread,
Jun 17, 2019, 3:53:56 PM6/17/19
to Jenkins Users
Sure I can. Here is the result of the commands executed from a Windows Bash shell:

traddatz@CNDEA041500 MINGW64 ~
$ ssh rad...@10.200.36.33 echo "OK"
OK

traddatz@CNDEA041500 MINGW64 ~
$ ssh -t rad...@10.200.36.33 echo "OK"
OK
Connection to 10.200.36.33 closed.

traddatz@CNDEA041500 MINGW64 ~
$ ssh -T rad...@10.200.36.33 echo "OK"
OK

traddatz@CNDEA041500 MINGW64 ~
$ ssh rad...@10.200.36.33 ssh -V
OpenSSH_6.9p1, OpenSSL 1.0.2r  26 Feb 2019

traddatz@CNDEA041500 MINGW64 ~
$

Ivan Fernandez Calvo

unread,
Jun 17, 2019, 5:21:56 PM6/17/19
to Jenkins Users
I run out of ideas, I could be some configuration option in /etc/ssh/sshd_config but I am not sure

Thomas R

unread,
Jun 18, 2019, 1:38:38 AM6/18/19
to Jenkins Users
Your help was most welcome. Thank you very much for that. Most likely I will contact IBM support, now. Maybe they can help. At least it seems that it is not directly a Jenkins problem.

Again, thank you.

Thomas.

Reply all
Reply to author
Forward
0 new messages