[JIRA] (JENKINS-61409) Websocket connections crash if message size is greater than 64Kb

310 views
Skip to first unread message

karthik.jayaraman@fiserv.com (JIRA)

unread,
Mar 9, 2020, 9:00:04 PM3/9/20
to jenkinsc...@googlegroups.com
Karthik Jayaraman created an issue
 
Jenkins / Bug JENKINS-61409
Websocket connections crash if message size is greater than 64Kb
Issue Type: Bug Bug
Assignee: Jeff Thompson
Components: core, remoting
Created: 2020-03-10 00:59
Environment: Jenkins version 2.223 - Built from docker container jenkins/jenkins:centos
Labels: websocket
Priority: Critical Critical
Reporter: Karthik Jayaraman

Websocket connections fail if the response from a pipeline step is greater than 64kb. 

Steps to recreate the problem:

  1. Run a jenkins server using the docker container jenkins/jenkins:centos
  2. Create a New Node with webSocket enabled
  3. On another Linux VM/slave machine, run the agent.jar with -webSocket option to connect to the Jenkins Master
  4. Create a pipeline with a step that would output text >64kb. For example, create a text file with size > 64Kb. 'cat' the file as a pipeline step. 
pipeline {
  agent {
    label 'test-channelclose'
  }  
  stages {
    stage ('Debug Issue') {
      steps {
        sh 'cat build_output.log'
        //archiveArtifacts 'build_output.log'
      }
    }
  }
}

5. Running the pipeline will break with the following error.

Mar 07, 2020 2:31:28 PM jenkins.agents.WebSocketAgents$Session error
WARNING: null
org.eclipse.jetty.websocket.api.MessageTooLargeException: Binary message size [524494] exceeds maximum size [65536]
        at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidBinaryMessageSize(WebSocketPolicy.java:128)
        at org.eclipse.jetty.websocket.common.Parser.assertSanePayloadLength(Parser.java:133)
        at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:494)
        at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:253)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:460)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)

Note: One of the workaround is to write the output to a file and archive the file. However, not all pipeline steps support an easy way to write the output to file.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 10, 2020, 6:28:02 AM3/10/20
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 10, 2020, 6:28:02 AM3/10/20
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Jesse Glick
 
Change By: Oleg Nenashev
Assignee: Jeff Thompson Jesse Glick

jglick@cloudbees.com (JIRA)

unread,
Mar 10, 2020, 2:52:03 PM3/10/20
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-61409
 
Re: Websocket connections crash if message size is greater than 64Kb

Possibly also affects WebSocket-based CLI.

jglick@cloudbees.com (JIRA)

unread,
Mar 10, 2020, 4:12:02 PM3/10/20
to jenkinsc...@googlegroups.com

There seems to be a configurable maximum size for binary packets. (WebSocketPolicy.setMaxBinaryMessageSize) There is no particular reason to impose a limit in Jenkins that I know of, so it could just be set to some large value.

Or Remoting may already have a maximum frame size; would have to dig into it. In this case the packet is coming via a RemoteOutputStream, I think, though the behavior may vary depending on whether or not org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING is set.

dbeck@cloudbees.com (JIRA)

unread,
Mar 11, 2020, 3:16:05 AM3/11/20
to jenkinsc...@googlegroups.com
Daniel Beck updated an issue
 
Change By: Daniel Beck
Labels: lts-candidate websocket

karthik.jayaraman@fiserv.com (JIRA)

unread,
Mar 11, 2020, 3:59:02 AM3/11/20
to jenkinsc...@googlegroups.com
Karthik Jayaraman commented on Bug JENKINS-61409
 
Re: Websocket connections crash if message size is greater than 64Kb

Observation after enabling FINEST logging on the remoting component:

The buffer size @ hudson.remoting.Engine#writeBlock() does not increase above 8192b. It seems that the remoting component is breaking it into multiple packets. However, the server errors out and closes the connection if there is more than 64kb of data received for a step.

Also, note that archiving large files (>30MB) on the Jenkins master does not have an impact.

vincent@latombe.net (JIRA)

unread,
Mar 16, 2020, 11:12:04 AM3/16/20
to jenkinsc...@googlegroups.com

https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/websocket/api/WebSocketPolicy.html#getMaxBinaryMessageSize()

Get the maximum size of a binary message during parsing.

This is a memory conservation option, memory over this limit will not be allocated by Jetty for handling binary messages. This applies to individual frames, whole message handling, and partial message handling.

Binary messages over this maximum will result in a close code 1009 StatusCode.MESSAGE_TOO_LARGE

Using too big values could cause memory issues.

karthik.jayaraman@fiserv.com (JIRA)

unread,
Mar 16, 2020, 11:28:03 AM3/16/20
to jenkinsc...@googlegroups.com

Using a large value can potentially cause memory issues on the server. Note that the slave connection is crashing anyway.

I think the size should be configurable so that we can change it based on the environment that we run our Jenkins servers on.

jglick@cloudbees.com (JIRA)

unread,
Mar 16, 2020, 11:47:02 AM3/16/20
to jenkinsc...@googlegroups.com

I do not wish to require any configuration. The Remoting communication should just work, reliably, regardless of workload.

karthik.jayaraman@fiserv.com (JIRA)

unread,
Mar 16, 2020, 12:34:03 PM3/16/20
to jenkinsc...@googlegroups.com

vincent@latombe.net (JIRA)

unread,
Mar 18, 2020, 11:59:03 AM3/18/20
to jenkinsc...@googlegroups.com
Vincent Latombe assigned an issue to Vincent Latombe
 
Change By: Vincent Latombe
Assignee: Jesse Glick Vincent Latombe

vincent@latombe.net (JIRA)

unread,
Mar 18, 2020, 11:59:03 AM3/18/20
to jenkinsc...@googlegroups.com
Vincent Latombe started work on Bug JENKINS-61409
 
Change By: Vincent Latombe
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Mar 20, 2020, 2:20:04 PM3/20/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Progress Review

jglick@cloudbees.com (JIRA)

unread,
Mar 26, 2020, 9:40:03 AM3/26/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Review Fixed but Unreleased
Resolution: Fixed
Released As: 2.229

jglick@cloudbees.com (JIRA)

unread,
Mar 26, 2020, 9:45:02 AM3/26/20
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-61409
 
Re: Websocket connections crash if message size is greater than 64Kb

Oliver Gondža remember that if backporting to 2.222.x, we would want to create a component backport so as to get something like remoting.version=4.2.1. Otherwise we would be including https://github.com/jenkinsci/remoting/pull/369 in LTS, which includes a couple of production source code changes, albeit small (Util.mkdirs → Files.createDirectories).

ogondza@gmail.com (JIRA)

unread,
Mar 30, 2020, 4:01:03 AM3/30/20
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
 
Change By: Oliver Gondža
Labels: lts-candidate non-trivial-lts-backporting websocket

dbeck@cloudbees.com (JIRA)

unread,
Mar 30, 2020, 7:20:04 AM3/30/20
to jenkinsc...@googlegroups.com
Change By: Daniel Beck
Status: Fixed but Unreleased Resolved
Released As: Jenkins 2.229

jthompson@cloudbees.com (JIRA)

unread,
Apr 3, 2020, 2:34:04 PM4/3/20
to jenkinsc...@googlegroups.com
Jeff Thompson commented on Bug JENKINS-61409
 
Re: Websocket connections crash if message size is greater than 64Kb

Should I create that intermediate version of Remoting, without those minor cleanup changes?

ogondza@gmail.com (JIRA)

unread,
Apr 7, 2020, 6:13:03 AM4/7/20
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Labels: 2.222.2-rejected lts-candidate non-trivial-lts-backporting websocket

ogondza@gmail.com (JIRA)

unread,
Apr 7, 2020, 6:21:03 AM4/7/20
to jenkinsc...@googlegroups.com
Oliver Gondža commented on Bug JENKINS-61409
 
Re: Websocket connections crash if message size is greater than 64Kb

Jesse Glick,Jeff Thompson, I chose not to backport this mostly for being too new. Since the unrelated changes are far less scary than the body of the fix, I suggest to target 2.222.3 without special remoting release.

Note to self: this needs to be backported as well https://github.com/jenkinsci/jenkins/pull/4601

o.v.nenashev@gmail.com (JIRA)

unread,
Apr 7, 2020, 6:28:04 PM4/7/20
to jenkinsc...@googlegroups.com

Oliver Gondža JEP-222 is a pretty important feature. Even if it is experimental, it would be great to have it fixed for LTS users so that we can get more adoption and feedback from early adopters.

For me backporting 4,3 is a no-go option due to removal of the deprecated code in https://github.com/jenkinsci/remoting/pull/369 . This change also adds serial version IDs, and it may lead to a funny behavior if classes are serialized over the Remoting channel. I would advice against taking the risk even in 2.222.3. CC Raihaan Shouhell who submitted a patch.

At the same time https://github.com/jenkinsci/remoting/pull/373 would be considerable for 2.222.2/3 if it was released as Remoting 4.2.1. It is still a high-risk change due to patches under the hood of Remoting engine (JNLP4 might be potentially affected by changes), but personally I would give it a try for 2.222.2 assuming that extensive testing of the patch is performance for the JNLP4 mode.

 

 

 

 

jglick@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 6:59:05 PM4/7/20
to jenkinsc...@googlegroups.com

It is still a high-risk change due to patches under the hood of Remoting engine (JNLP4 might be potentially affected by changes)

I cannot see how. None of the affected code is outside the WebSocket-specific handler method used in the -webSocket mode new in 2.222.x, except for a new method in AbstractByteBufferCommandTransport which is newly called in that modified code.

jthompson@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 8:11:03 PM4/7/20
to jenkinsc...@googlegroups.com

I agree with Jesse. I don't see any risk to previously existing functionality for this change to better support WebSockets.

I'm in the middle of preparing a 4.2.1 patch containing just the WebSockets fix over 4.2. That eliminates any concern over https://github.com/jenkinsci/remoting/pull/369 .

jthompson@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 9:10:03 PM4/7/20
to jenkinsc...@googlegroups.com

The Remoting 4.2.1 release is out and available in repo.jenkins.io. The release notes are at https://github.com/jenkinsci/remoting/releases/tag/remoting-4.2.1 . I put together a PR to integrate that version into the 2.222.x LTS Jenkins branch at https://github.com/jenkinsci/jenkins/pull/4635 .

ogondza@gmail.com (JIRA)

unread,
May 5, 2020, 2:25:03 AM5/5/20
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Labels: 2. 2. 222.2-rejected lts-candidate non-trivial-lts-backporting websocket

ogondza@gmail.com (JIRA)

unread,
May 5, 2020, 2:25:04 AM5/5/20
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Labels: 2. 2. 222.2-rejected 2.222.4-fixed non-trivial-lts-backporting websocket
Reply all
Reply to author
Forward
0 new messages