[JIRA] (JENKINS-52204) Jenkins Windows Agent failed to start after upgrade to 2.1.129

157 views
Skip to first unread message

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 27, 2018, 3:08:02 AM6/27/18
to jenkinsc...@googlegroups.com
Dzianis Mazuronak created an issue
 
Jenkins / Bug JENKINS-52204
Jenkins Windows Agent failed to start after upgrade to 2.1.129
Issue Type: Bug Bug
Assignee: Jeff Thompson
Components: remoting, windows-slaves-plugin
Created: 2018-06-27 07:07
Environment: Jenkins 2.1.129
Priority: Major Major
Reporter: Dzianis Mazuronak

Jenkins Windows Agent failed to start after upgrade to version 2.1.129: Jenkins is running behind apache reverse proxy on different host that proxy itself. Parameter “tunnel connection through” is set correctly. From the error message we can see that remote agent was trying to connect to reverse proxy host instead of Jenkins master host.

The version 2.1.128 works without issues.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 27, 2018, 5:07:02 AM6/27/18
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-52204
 
Re: Jenkins Windows Agent failed to start after upgrade to 2.1.129

Most likely it is related to JENKINS-51818, CC Andrei Laptsinski
Dzianis Mazuronak have you updated Remoting on agents to 3.22?

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 27, 2018, 6:41:02 AM6/27/18
to jenkinsc...@googlegroups.com

Remoting 3.22 has the problem. Remoting 3.21 works OK:

INFORMATION: Setting up agent: MY-CLIENT
Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFORMATION: Jenkins agent is running in headless mode.
Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main createEngine
WARNUNG: Certificate validation for HTTPs endpoints is disabled
Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
INFORMATION: Using Remoting version: 3.22
Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
Jun 26, 2018 11:07:54 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
WARNUNG: HTTPs certificate check is disabled for the endpoint.
Jun 26, 2018 11:07:55 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jun 26, 2018 11:07:56 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
WARNUNG: Connection refused: connect
Jun 26, 2018 11:07:56 PM hudson.remoting.jnlp.Main$CuiListener error
SCHWERWIEGEND: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
java.io.IOException: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:284)
	at hudson.remoting.Engine.innerRun(Engine.java:518)
	at hudson.remoting.Engine.run(Engine.java:469)

Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
INFORMATION: Setting up agent: MY-CLIENT
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFORMATION: Jenkins agent is running in headless mode.
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
WARNUNG: Certificate validation for HTTPs endpoints is disabled
Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
INFORMATION: Using Remoting version: 3.21
Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
WARNUNG: HTTPs certificate check is disabled for the endpoint.
Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFORMATION: Agent discovery successful
  Agent address: 10.150.3.9
  Agent port:    44864
  Identity:      f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFORMATION: Handshaking
Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFORMATION: Connecting to 10.150.3.9:44864

 Port 37367 is open on Jenkins server 10.150.3.9 and not on reverse proxy tools.domain.local.

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 27, 2018, 6:59:03 AM6/27/18
to jenkinsc...@googlegroups.com

From what I see in the https://github.com/jenkinsci/remoting/pull/275/files code, the logic should be using the resolved hostname for direct connection:

host = defaultString(first(header(con, "X-Jenkins-JNLP-Host")), salURL.getHost());

Are you sure that "hudson.TcpSlaveAgentListener.hostName" is set on your instance?
If no, setting it up on the Master side would be a workaround until we figure out the tunneling issue

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 27, 2018, 7:10:01 AM6/27/18
to jenkinsc...@googlegroups.com

On master side this slave has the parameter "Tunnel connection through" set to 10.150.3.9.

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 27, 2018, 9:05:02 AM6/27/18
to jenkinsc...@googlegroups.com

Dzianis Mazuronak Could you please set the "-Dhudson.TcpSlaveAgentListener.hostName" system property on master as I suggested above?

crule@aegistg.com (JIRA)

unread,
Jun 27, 2018, 2:41:02 PM6/27/18
to jenkinsc...@googlegroups.com

I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on "emain". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem.

crule@aegistg.com (JIRA)

unread,
Jun 27, 2018, 2:43:01 PM6/27/18
to jenkinsc...@googlegroups.com
Chris Rule edited a comment on Bug JENKINS-52204
I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on " emain eddard ". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem. I receive the same error as before "IOException: [https://eddard/jenkins/|https://emain/jenkins/] provided port:41829 is not reachable".

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 27, 2018, 3:10:01 PM6/27/18
to jenkinsc...@googlegroups.com

I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement.

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 28, 2018, 4:58:02 AM6/28/18
to jenkinsc...@googlegroups.com
Dzianis Mazuronak edited a comment on Bug JENKINS-52204
I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement.


There is no such header "X-Jenkins-JNLP-Host" in http response.

The property "hudson.TcpSlaveAgentListener.hostName" was set in the command line. In which configuration file can this property be set? I'm not sure that the command line was processed.

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 5:38:01 AM6/28/18
to jenkinsc...@googlegroups.com

java -Dhudson.TcpSlaveAgentListener.hostName=fooBar -jar jenkins.war should do the job.
Cannot suggest config file since I do not know Jenkins configuration

gedenami@gmail.com (JIRA)

unread,
Jun 28, 2018, 7:15:03 AM6/28/18
to jenkinsc...@googlegroups.com

You can receive value of "hudson.TcpSlaveAgentListener.hostName". By Jenkins  script console (System.getProperty("hudson.TcpSlaveAgentListener.hostName") ).  Possible it is override by other  JAVA_OPTS/JENKINS_JAVA_OPTIONS)

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 28, 2018, 7:51:01 AM6/28/18
to jenkinsc...@googlegroups.com

No, the other options are empty, so no override here.

If I type

println(System.getProperty("hudson.TcpSlaveAgentListener.hostName") )

in Jenkins script console, I get only

null

as result.

 

There is an other interresting result:

hudson.TcpSlaveAgentListener.hostName

produces following exception:

groovy.lang.MissingPropertyException: No such property: hostName for class: hudson.TcpSlaveAgentListener
	at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1002)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1857)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1833)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
	at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
	at Script1.run(Script1.groovy:1)

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 7:55:02 AM6/28/18
to jenkinsc...@googlegroups.com

> There is an other interresting result:

As designed. System properties won't be visible in Groovy in such way.

Dzianis Mazuronak System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work.

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 7:56:01 AM6/28/18
to jenkinsc...@googlegroups.com
Oleg Nenashev edited a comment on Bug JENKINS-52204
> There is an other interresting result:

As designed. System properties won't be visible in Groovy in such way.

[~dmazuronak] System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work until the property is passed correctly .

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 9:03:02 AM6/28/18
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Oleg Nenashev
 
Change By: Oleg Nenashev
Assignee: Jeff Thompson Oleg Nenashev

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 9:03:03 AM6/28/18
to jenkinsc...@googlegroups.com
Oleg Nenashev started work on Bug JENKINS-52204
 
Change By: Oleg Nenashev
Status: Open In Progress

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 9:07:02 AM6/28/18
to jenkinsc...@googlegroups.com
Change By: Oleg Nenashev
Status: In Progress Review

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 28, 2018, 9:09:02 AM6/28/18
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-52204
 
Re: Jenkins Windows Agent failed to start after upgrade to 2.1.129

See my comment in https://github.com/jenkinsci/remoting/pull/279 . After the review, this issue happens IFF "-tunnel" is set. I believe that "-tunnel" flag should be deprecated and replaced by offering a custom port in TcpAgentListener response headers like it's already done for hosts

So I propose to just disable checks in that case if Andrei Laptsinski agrees. We could implement a more advanced behavior and checks for tunnel options as a part of JENKINS-52246.

dzianis.mazuronak@swisspost.com (JIRA)

unread,
Jun 28, 2018, 1:51:01 PM6/28/18
to jenkinsc...@googlegroups.com

You do it right, and it works: I found my error and placed the option in front of jenkins.jar. Now Windows slave is accessible again.

crule@aegistg.com (JIRA)

unread,
Jun 28, 2018, 2:22:02 PM6/28/18
to jenkinsc...@googlegroups.com

Dzianis Mazuronak :

Agreed. I had to modify the Jenkins startup script for the define (-D) to take effect. It appears you can't use the Jenkins console to make the change, even though it showed up in the Jenkins System Information page.

Since I run on Linux Mint (Ubuntu derivative) I modified the file /etc/defaults/jenkins, JAVA_ARGS line, to add the -D as described above. My final line looks like (the first -D... was already there):

JAVA_ARGS="-Djava.awt.headless=true -Dhudson.TcpSlaveAgentListener.hostName=j-jenkins"

o.v.nenashev@gmail.com (JIRA)

unread,
Jun 29, 2018, 6:10:03 PM6/29/18
to jenkinsc...@googlegroups.com

Remoting 3.23 is out. Created https://github.com/jenkinsci/jenkins/pull/3530

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 3, 2018, 1:57:02 PM7/3/18
to jenkinsc...@googlegroups.com
Change By: Oleg Nenashev
Status: In Review Resolved
Resolution: Fixed
Released As: 2.130

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 3, 2018, 1:58:04 PM7/3/18
to jenkinsc...@googlegroups.com
 
Re: Jenkins Windows Agent failed to start after upgrade to 2.1.129

No need to backport the fix, it won't get into LTS

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 3, 2018, 2:03:03 PM7/3/18
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
 
Change By: Oleg Nenashev
Released As: Jenkins 2.130

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 7, 2018, 6:16:02 AM7/7/18
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Released As: Jenkins 2.130 , Remoting 3.23

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 7, 2018, 6:16:03 AM7/7/18
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Component/s: core
Component/s: windows-slaves-plugin
Reply all
Reply to author
Forward
0 new messages