[JIRA] (JENKINS-57143) JNLP4 error: "Connection closed before acknowledgement sent"

37 views
Skip to first unread message

conf (JIRA)

unread,
Apr 22, 2019, 5:08:02 PM4/22/19
to jenkinsc...@googlegroups.com
Alexey Shein created an issue
 
Jenkins / Bug JENKINS-57143
JNLP4 error: "Connection closed before acknowledgement sent"
Issue Type: Bug Bug
Assignee: Jeff Thompson
Components: remoting
Created: 2019-04-22 21:07
Environment: Agent: OSX Mojave 10.14.4
Jenkins: Linux (Jenkins LTS 2.164.2)

Java on agent:
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

Java on master:
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)
Priority: Blocker Blocker
Reporter: Alexey Shein

Hello!

We run JNLP-powered agents on 4 Mac machines, after Jenkins update to 2.164.2 (from 2.150.3), those machines are not able to connect using JNLP-4 (see exception below), although JNLP-3 connection works.

Agent is run as:
{{
/usr/bin/java -Djava.awt.headless=true -jar /Applications/Jenkins/agent.jar -jnlpUrl https://<redacted-master-address>/computer/<redacted-agent>/slave-agent.jnlp -secret <redacted-secret> -workDir /Users/jenkins/jenkins_root
}}

Exception on agent:
{{INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:614)
at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.access$000(AckFilterLayer.java:43)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer$1.run(AckFilterLayer.java:176)
at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:964)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)}}

On jenkins master:

{{
Apr 22, 2019 3:42:42 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #1512 failed
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Apr 22, 2019 3:42:42 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1,513 from /<redacted-agent-ip>:58595
Apr 22, 2019 3:42:54 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #1514 failed
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)
}}

Things that I checked:

  • agent.jar is up-to-date and matches the one on master
  • JNLP port on master is open and available from the agent
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

conf (JIRA)

unread,
Apr 22, 2019, 5:10:03 PM4/22/19
to jenkinsc...@googlegroups.com
Alexey Shein updated an issue
Change By: Alexey Shein
Things that I checked:
- agent.jar is up-to-date and matches the one on master
- JNLP port on master is open and available from the agent

jthompson@cloudbees.com (JIRA)

unread,
Apr 24, 2019, 5:54:03 PM4/24/19
to jenkinsc...@googlegroups.com
Jeff Thompson commented on Bug JENKINS-57143
 
Re: JNLP4 error: "Connection closed before acknowledgement sent"

That's a weird one. I have no idea what's going on there. There weren't any changes between 2.150 and 2.164 that should have affected anything like that.

I suggest double-checking your configuration. Make sure the version 4 protocol is allowed. Take a look at any other settings that might be relevant.

Many agent issues occur because of system, network, or environmental issues, but this one doesn't seem to fall into that class.

Which agent version do you have?

conf (JIRA)

unread,
Apr 27, 2019, 3:15:02 PM4/27/19
to jenkinsc...@googlegroups.com

I suggest double-checking your configuration. Make sure the version 4 protocol is allowed. Take a look at any other settings that might be relevant.

It is enabled. I've enabled also JNLP 3 and it is used now as a fallback when JNLP4 connect fails.

Which agent version do you have?

 INFO: Using Remoting version: 3.29
 
Any idea how to debug that? I tried to use loggingConfig option, but couldn't make it work (also not sure what to write in the file).

Maybe some plugins that can affect that?

 

 

jthompson@cloudbees.com (JIRA)

unread,
Apr 29, 2019, 4:14:07 PM4/29/19
to jenkinsc...@googlegroups.com
Jeff Thompson updated an issue
 
Change By: Jeff Thompson
Attachment: jenkins-slave.logging.properties

jthompson@cloudbees.com (JIRA)

unread,
Apr 29, 2019, 4:14:07 PM4/29/19
to jenkinsc...@googlegroups.com
Jeff Thompson commented on Bug JENKINS-57143
 
Re: JNLP4 error: "Connection closed before acknowledgement sent"

I've gotten the logging config file to work before and sometimes gotten some useful information from it. There's some basic information on Java util logging config files here: http://tutorials.jenkov.com/java-logging/configuration.html . I'll add one I got from somewhere that I've used before.

Plugins can change a lot of interactions in Jenkins. It can be hard to figure out what they're doing or how different plugins interact with each other.

There seems to be something weird going on when the agent tries to connect with JNLP-4. It seems like it's getting an EOFException, like it's expecting to read more data than is actually supplied. There isn't enough information to figure out what's going on. You're going to have to see if you can gather more information. More detailed logging may help. Master and agent.

 

conf (JIRA)

unread,
Apr 30, 2019, 5:09:02 PM4/30/19
to jenkinsc...@googlegroups.com
Alexey Shein updated an issue
 
Change By: Alexey Shein
Component/s: github-oauth-plugin

conf (JIRA)

unread,
Apr 30, 2019, 5:10:04 PM4/30/19
to jenkinsc...@googlegroups.com
Alexey Shein commented on Bug JENKINS-57143
 
Re: JNLP4 error: "Connection closed before acknowledgement sent"

It turns out the issue was in Github OAuth plugin. After downgrading 0.32 version to 0.31, everything seems to work fine.

jthompson@cloudbees.com (JIRA)

unread,
Apr 30, 2019, 5:25:02 PM4/30/19
to jenkinsc...@googlegroups.com
Jeff Thompson updated an issue
 
Change By: Jeff Thompson
Component/s: remoting

jthompson@cloudbees.com (JIRA)

unread,
Apr 30, 2019, 5:25:02 PM4/30/19
to jenkinsc...@googlegroups.com
Jeff Thompson assigned an issue to Unassigned
Change By: Jeff Thompson
Assignee: Jeff Thompson

jthompson@cloudbees.com (JIRA)

unread,
Apr 30, 2019, 5:25:02 PM4/30/19
to jenkinsc...@googlegroups.com
Jeff Thompson commented on Bug JENKINS-57143
 
Re: JNLP4 error: "Connection closed before acknowledgement sent"

I'm glad you were able to isolate it. I'll remove Remoting from the components.

sam.mxracer@gmail.com (JIRA)

unread,
Aug 5, 2019, 11:08:02 PM8/5/19
to jenkinsc...@googlegroups.com

I'm not able to replicate this issue. We do have authorization bugs in github-oauth 0.32 release but I upgraded to the 0.32 release locally and still can't replicate the issue.

sam.mxracer@gmail.com (JIRA)

unread,
Aug 10, 2019, 10:32:02 AM8/10/19
to jenkinsc...@googlegroups.com
Sam Gleske resolved as Cannot Reproduce
 

Since I'm not able to reproduce the issue on 0.32 (originally reported) and 0.33 (recently released) version of the plugin I'm going to close this.

However, if you can provide reproduction steps I'll look into it further.
Change By: Sam Gleske
Status: Open Resolved
Assignee: Sam Gleske
Resolution: Cannot Reproduce

nneul@neulinger.org (JIRA)

unread,
Dec 3, 2019, 7:47:02 PM12/3/19
to jenkinsc...@googlegroups.com
Nathan Neulinger commented on Bug JENKINS-57143
 
Re: JNLP4 error: "Connection closed before acknowledgement sent"

I realize this has already been closed - but just had to go through a big diag effort on my site with a bunch of windows JNLP agents that stopped working sometime in past few days.

After a long set of rabbit trails - what I found in my case – the problem wound up being reverse DNS resolution stopped working (unrelated local problem – "It's always DNS", right?) - but that inability to reverse resolve the connecting client (windows box) was causing the JNLP server side to drop the connection. 

Just wanted to add this here in case it helps anyone else with future similar issues if it happens to be related.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jthompson@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 1:01:02 PM12/4/19
to jenkinsc...@googlegroups.com

Nathan Neulinger, thanks for sharing your experience. It may help give others some ideas on where to investigate if they see similar problems.

Were you seeing similar log messages as originally reported? If not, it might be a somewhat different issue.

DNS configuration issues can cause lots of weird, problems that don't appear related.

 

nneul@neulinger.org (JIRA)

unread,
Dec 4, 2019, 1:08:03 PM12/4/19
to jenkinsc...@googlegroups.com

Yep, almost exact same failure, with addition of it indicating that all protocols were rejected. Re-enabling JNLP3 fixed it as a test. It was just dumb luck viewing an strace of the jenkins master that I saw the hangs/failures of reverse dns lookups on the master.

 

 

jthompson@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 1:15:02 PM12/4/19
to jenkinsc...@googlegroups.com

Any ideas what we could do to better handle this situation? Or provide better diagnostics?

nneul@neulinger.org (JIRA)

unread,
Dec 4, 2019, 1:18:03 PM12/4/19
to jenkinsc...@googlegroups.com

There was near zero useful reporting server side - if the JNLP server side handler has "remote client has to resolve in DNS" at the very least reporting that explicitly in the failure logs. I can understand it not being able to necessarily make it back to the client side but at the least the server side should report it usefully in logs. (Just guessing there, but I can certainly see that possibility from JNLP4 protocol limitations.) 

 

Would be great if server side could report it in the agent status on the node page, but at the very least getting it in the logs. 

nneul@neulinger.org (JIRA)

unread,
Dec 4, 2019, 1:19:02 PM12/4/19
to jenkinsc...@googlegroups.com

I'd also ask question of why it has to reverse resolve - i.e. why is that requirement in there in the first place. 

jthompson@cloudbees.com (JIRA)

unread,
Dec 4, 2019, 1:21:02 PM12/4/19
to jenkinsc...@googlegroups.com

It would be nice to get some time to see about getting some of that in. Better diagnosability would be a nice addition.

Yeah, I wonder why it needs to reverse resolve. Would take some time to figure out what and why it's doing there.

Thanks!
Reply all
Reply to author
Forward
0 new messages