[JIRA] (JENKINS-61253) Remoting using WebSocket fails with "Handshake response not received" Exception

201 views
Skip to first unread message

ethorsa@inbox.lv (JIRA)

unread,
Feb 27, 2020, 4:51:02 AM2/27/20
to jenkinsc...@googlegroups.com
ethorsa created an issue
 
Jenkins / Bug JENKINS-61253
Remoting using WebSocket fails with "Handshake response not received" Exception
Issue Type: Bug Bug
Assignee: Jeff Thompson
Components: remoting
Created: 2020-02-27 09:50
Environment: - Remoting v4.2
- AdoptOpenJDK v11.0.5 (Windows 10)
- Jenkins v2.204.2
Priority: Major Major
Reporter: ethorsa

Using remoting v4.2 with WebSocket crashes after ~ 30 seconds with a "Handshake response not received" exception:
 

io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager <init>
CONFIG: Provider class loaded: io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.JdkClientContainer
hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Handshake response not received.
io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
	at hudson.remoting.Engine.runWebSocket(Engine.java:624)
	at hudson.remoting.Engine.run(Engine.java:469)

 

Remoting is launched through hudson.remoting.jnlp.Main.main() with arguments: 

[<Secret ID>, <Agent>, -url, https://<Master Host>:<Port>/, -webSocket, -workDir, <WorkDir Path> -headless, -noreconnect]
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

ethorsa@inbox.lv (JIRA)

unread,
Feb 27, 2020, 4:52:04 AM2/27/20
to jenkinsc...@googlegroups.com

ethorsa@inbox.lv (JIRA)

unread,
Feb 27, 2020, 5:49:03 AM2/27/20
to jenkinsc...@googlegroups.com
ethorsa updated an issue
 
Change By: ethorsa
Environment: - Remoting v4.2
- AdoptOpenJDK v11.0.5 , Oracle Java v1.8.0-181 (Windows 10)
- Jenkins v2.204.2

ethorsa@inbox.lv (JIRA)

unread,
Feb 27, 2020, 5:50:02 AM2/27/20
to jenkinsc...@googlegroups.com

jthompson@cloudbees.com (JIRA)

unread,
Feb 27, 2020, 2:07:03 PM2/27/20
to jenkinsc...@googlegroups.com
Jeff Thompson commented on Bug JENKINS-61253
 
Re: Remoting using WebSocket fails with "Handshake response not received" Exception

You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217.

jthompson@cloudbees.com (JIRA)

unread,
Feb 27, 2020, 2:07:04 PM2/27/20
to jenkinsc...@googlegroups.com
Jeff Thompson closed an issue as Not A Defect
 
Change By: Jeff Thompson
Status: Open Closed
Resolution: Not A Defect

ethorsa@inbox.lv (JIRA)

unread,
Mar 2, 2020, 3:36:02 AM3/2/20
to jenkinsc...@googlegroups.com
ethorsa commented on Bug JENKINS-61253
 
Re: Remoting using WebSocket fails with "Handshake response not received" Exception

I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

jthompson@cloudbees.com (JIRA)

unread,
Mar 2, 2020, 10:31:03 AM3/2/20
to jenkinsc...@googlegroups.com

As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

ethorsa@inbox.lv (JIRA)

unread,
Mar 3, 2020, 1:53:02 AM3/3/20
to jenkinsc...@googlegroups.com
ethorsa commented on Bug JENKINS-61253

Great, thanks!

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 3:37:02 AM3/30/20
to jenkinsc...@googlegroups.com
ethorsa reopened an issue
 
Change By: ethorsa
Resolution: Not A Defect
Status: Closed Reopened

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 3:45:02 AM3/30/20
to jenkinsc...@googlegroups.com
ethorsa commented on Bug JENKINS-61253
 
Re: Remoting using WebSocket fails with "Handshake response not received" Exception

Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3) too. My minimal steps to reproduce the error:

  1. Create a new static agent on the master (test-websocket in this example)
  2. Enable "Use WebSocket" in it's configuration
  3. Start the agent:
java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"

After ~ 30 sec the agent fails by exception:

Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Handshake response not received.
io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
	at hudson.remoting.Engine.runWebSocket(Engine.java:627)
	at hudson.remoting.Engine.run(Engine.java:469)

The master log shows a single related entry:

INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 3:52:09 AM3/30/20
to jenkinsc...@googlegroups.com
ethorsa edited a comment on Bug JENKINS-61253
Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3) too. My minimal steps to reproduce the error:
# Create a new static agent on the master (_test-websocket_ in this example)
# Enable _"Use WebSocket"_ in it's configuration
# Start the agent:

{noformat}
java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"{noformat}


After ~ 30 sec the agent fails by exception:


{noformat}

Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Handshake response not received.
io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
at hudson.remoting.Engine.runWebSocket(Engine.java:627)
at hudson.remoting.Engine.run(Engine.java:469)
{noformat}


The master log shows a single related entry:


{noformat}

INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException
{noformat}

 Error on the agent after connection failed:

{noformat}
java.nio.channels.ClosedChannelException
at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141)
at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90)
at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317)
at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
{noformat}

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 3:52:19 AM3/30/20
to jenkinsc...@googlegroups.com
[...]
 {noformat}

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 4:09:03 AM3/30/20
to jenkinsc...@googlegroups.com
ethorsa edited a comment on Bug JENKINS-61253
Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3) too . My minimal steps to reproduce the error:

ethorsa@inbox.lv (JIRA)

unread,
Mar 30, 2020, 5:32:02 AM3/30/20
to jenkinsc...@googlegroups.com
ethorsa commented on Bug JENKINS-61253

I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main(). The agent launches successfully, but it's marked as offline on the master.

 

Remoting log (on Agent):

hudson.remoting.jnlp.Main$CuiListener status
INFO: WebSocket connection open
hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Agent log (Icon of the node):

Inbound agent connected from <Ip>

Master Log:

WARNING jenkins.agents.WebSocketAgents$Session error
null
java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
Caused: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.base/java.lang.Thread.run(Thread.java:834)

INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException

jthompson@cloudbees.com (JIRA)

unread,
Mar 30, 2020, 11:07:04 AM3/30/20
to jenkinsc...@googlegroups.com

Try Remoting (agent) 4.2.

You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh.

Remoting 4.2:

   Weekly releases 2.217 through 2.228

   LTS release 2.222.1

Remoting 4.3:

  Weekly releases 2.229+

  Upcoming LTS

 

If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

ethorsa@inbox.lv (JIRA)

unread,
Mar 31, 2020, 1:10:02 AM3/31/20
to jenkinsc...@googlegroups.com
ethorsa commented on Bug JENKINS-61253

Thanks Jeff Thompson, downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets. 

 

The WebSocket implementation is still in Beta, because we need to get extended real-world testing.

I'll let you know if there are other issues.

ethorsa@inbox.lv (JIRA)

unread,
Mar 31, 2020, 1:11:04 AM3/31/20
to jenkinsc...@googlegroups.com
Change By: ethorsa
Status: Reopened Fixed but Unreleased
Resolution: Fixed
Released As: Jenkins v2.222.1 / Remoting v4.2

ethorsa@inbox.lv (JIRA)

unread,
Mar 31, 2020, 1:11:04 AM3/31/20
to jenkinsc...@googlegroups.com
Change By: ethorsa
Status: Fixed but Unreleased Resolved
Reply all
Reply to author
Forward
0 new messages