[JIRA] (JENKINS-58700) java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

[cw.chu@asmpt.com (JIRA)]

Chu CW created an issue

Jenkins / JENKINS-58700 java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent Issue Type: Bug Assignee: Unassigned Components: core Created: 2019-07-29 08:50 Environment: Jenkins version: 2.121.3 OS: Windows Server 2012 R2 Standard Java version 1.8.0_111 Labels: jnlp slave slave-agent network Priority: Blocker Reporter: Chu CW Hi our Jenkins server is facing following issues for around one month,but there is no updates/installation of plugins for Jenkins. The issue is the slaves(~10-20) can be connected to server normally at the beginning, with calling command 'java -jar agent.jar -jnlpurl ......'. And all of them will suddenly disconnected at the same time. But it is confirmed that the network connection of both server and slaves are normal, and 'ping'ing each other are success too.   Here is the logging. Server logging: Jul 26, 2019 6:19:17 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel JNLP4-connect connection from vis_test25_10_1.ahkex.asmpt.com/10.0.120.128:58639. java.util.concurrent.TimeoutException: Ping started at 1564136117631 hasn't completed by 1564136357652 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Jul 26, 2019 6:19:17 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: Computer.threadPoolForRemoting 9322 for vis_test25_10_1 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:799) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:103) at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:715) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Slave Logging: Jul 26, 2019 6:21:18 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Failed to connect to ahknts34b:63560 java.io.IOException: Failed to connect to ahknts34b:63560 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242) at hudson.remoting.Engine.connect(Engine.java:686) at hudson.remoting.Engine.innerRun(Engine.java:547) at hudson.remoting.Engine.run(Engine.java:469) Caused by: java.net.ConnectException: Connection refused: connect at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Unknown Source) at sun.nio.ch.Net.connect(Unknown Source) at sun.nio.ch.SocketChannelImpl.connect(Unknown Source) at java.nio.channels.SocketChannel.open(Unknown Source) at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203) ... 3 more   Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[cw.chu@asmpt.com (JIRA)]

[cw.chu@asmpt.com (JIRA)]

Chu CW updated an issue

Jenkins / JENKINS-58700 java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

Change By: Chu CW Hi our Jenkins server is facing following issues for around one month, but there is no updates/installation of plugins for Jenkins from one per week to currently occur once in 1-2 days .

The issue is the slaves(~10-20) can be connected to server normally at the beginning, with calling command 'java -jar agent.jar -jnlpurl ......'. And all of them will suddenly disconnected at the same time. But it is confirmed that the network connection of both server and slaves are normal, and 'ping'ing each other are success too.   Here is the logging. Server logging: Jul 26, 2019 6:19:17 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel JNLP4-connect connection from vis_test25_10_1.ahkex.asmpt.com/10.0.120.128:58639. java.util.concurrent.TimeoutException: Ping started at 1564136117631 hasn't completed by 1564136357652 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Jul 26, 2019 6:19:17 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed

WARNING: Computer.threadPoolForRemoting [#9322] for vis_test25_10_1 terminated

Meanwhile, it is found that all port for listening slaves on server are all status "CLOSE_WAIT".   FYI, here is the installed plugin list: JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 Ant Plugin (ant): 1.8 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5 Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.5-3.0 Authentication Tokens API Plugin (authentication-tokens): 1.3 Autofavorite for Blue Ocean (blueocean-autofavorite): 1.2.4 Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline): 1.16.0 Common API for Blue Ocean (blueocean-commons): 1.16.0 Config API for Blue Ocean (blueocean-config): 1.16.0 Blue Ocean Core JS (blueocean-core-js): 1.16.0 Dashboard for Blue Ocean (blueocean-dashboard): 1.16.0 Display URL for Blue Ocean (blueocean-display-url): 2.2.0 Events API for Blue Ocean (blueocean-events): 1.16.0 Git Pipeline for Blue Ocean (blueocean-git-pipeline): 1.16.0 GitHub Pipeline for Blue Ocean (blueocean-github-pipeline): 1.16.0 i18n for Blue Ocean (blueocean-i18n): 1.16.0 JIRA Integration for Blue Ocean (blueocean-jira): 1.16.0 JWT for Blue Ocean (blueocean-jwt): 1.16.0 Personalization for Blue Ocean (blueocean-personalization): 1.16.0 Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl): 1.16.0 Blue Ocean Pipeline Editor (blueocean-pipeline-editor): 1.16.0 Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api): 1.16.0 REST Implementation for Blue Ocean (blueocean-rest-impl): 1.16.0 REST API for Blue Ocean (blueocean-rest): 1.16.0 Web for Blue Ocean (blueocean-web): 1.16.0 Blue Ocean (blueocean): 1.16.0 bouncycastle API Plugin (bouncycastle-api): 2.16.3 Branch API Plugin (branch-api): 2.3.0 Build Timeout (build-timeout): 1.19 user build vars plugin (build-user-vars-plugin): 1.5 Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 2.4.4 Folders Plugin (cloudbees-folder): 6.5.1 Command Agent Launcher Plugin (command-launcher): 1.2 Conditional BuildStep (conditional-buildstep): 1.3.6 Credentials Binding Plugin (credentials-binding): 1.16 Credentials Plugin (credentials): 2.1.17 Dashboard View (dashboard-view): 2.10 Display URL API (display-url-api): 2.3.1 Docker Commons Plugin (docker-commons): 1.14 Docker Pipeline (docker-workflow): 1.18 Durable Task Plugin (durable-task): 1.29 Email Extension Plugin (email-ext): 2.66 External Monitor Job Type Plugin (external-monitor-job): 1.7 Favorite (favorite): 2.3.2 Git client plugin (git-client): 2.7.2 GIT server Plugin (git-server): 1.7 Git plugin (git): 3.9.1 GitHub API Plugin (github-api): 1.92 GitHub Branch Source Plugin (github-branch-source): 2.3.6 GitHub plugin (github): 1.29.2 Gradle Plugin (gradle): 1.29 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.6-1.0 HTML Publisher plugin (htmlpublisher): 1.16 Jackson 2 API Plugin (jackson2-api): 2.9.8 Javadoc Plugin (javadoc): 1.4 JDK Tool Plugin (jdk-tool): 1.1 Design Language (jenkins-design-language): 1.16.0 JIRA plugin (jira): 3.0.7 Join plugin (join): 1.21 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 jQuery plugin (jquery): 1.12.4-0 JSch dependency plugin (jsch): 0.1.54.2 JUnit Plugin (junit): 1.24 LDAP Plugin (ldap): 1.20 Locale plugin (locale): 1.3 Mailer Plugin (mailer): 1.21 MapDB API Plugin (mapdb-api): 1.0.9.0 Matrix Authorization Strategy Plugin (matrix-auth): 2.3 Matrix Project Plugin (matrix-project): 1.13 Maven Integration plugin (maven-plugin): 3.1.2 Mercurial plugin (mercurial): 2.4 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 MSBuild Plugin (msbuild): 1.29 Node and Label parameter plugin (nodelabelparameter): 1.7.2 PAM Authentication plugin (pam-auth): 1.4 Parallel Test Executor Plugin (parallel-test-executor): 1.10 Parameterized Scheduler (parameterized-scheduler): 0.6.2 Parameterized Trigger plugin (parameterized-trigger): 2.35.2 Pipeline: Build Step (pipeline-build-step): 2.9 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.9 Pipeline: Input Step (pipeline-input-step): 2.10 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Pipeline: Model API (pipeline-model-api): 1.3.8 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 Pipeline: Declarative (pipeline-model-definition): 1.3.8 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.3.8 Pipeline: REST API Plugin (pipeline-rest-api): 2.11 Pipeline: Stage Step (pipeline-stage-step): 2.3 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.3.8 Pipeline: Stage View Plugin (pipeline-stage-view): 2.11 Plain Credentials Plugin (plain-credentials): 1.4 PowerShell plugin (powershell): 1.3 Priority Sorter Plugin (PrioritySorter): 3.6.0 Pub-Sub "light" Bus (pubsub-light): 1.12 Python Plugin (python): 1.3 Rebuilder (rebuild): 1.29 Redmine Metrics Report (redmine-metrics-report): 1.0.1 Redmine plugin (redmine): 0.21 Run Condition Plugin (run-condition): 1.0 SCM API Plugin (scm-api): 2.4.1 Script Security Plugin (script-security): 1.58 Server Sent Events (SSE) Gateway Plugin (sse-gateway): 1.17 SSH Agent Plugin (ssh-agent): 1.15 SSH Credentials Plugin (ssh-credentials): 1.14 SSH Slaves plugin (ssh-slaves): 1.26 Structs Plugin (structs): 1.19 Subversion Plug-in (subversion): 2.11.1 Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.0.1 Timestamper (timestamper): 1.8.10 Token Macro Plugin (token-macro): 2.5 Active Choices Plug-in (uno-choice): 2.1 Variant Plugin (variant): 1.1 WebSphere Deployer Plugin (websphere-deployer): 1.6.1 Windows Exe Runner Plugin (windows-exe-runner): 1.2 Windows Slaves Plugin (windows-slaves): 1.3.1 Pipeline (workflow-aggregator): 2.5 Pipeline: API (workflow-api): 2.33 Pipeline: Basic Steps (workflow-basic-steps): 2.15 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.13 Pipeline: Groovy (workflow-cps): 2.68 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.28 Pipeline: Job (workflow-job): 2.32 Pipeline: Multibranch (workflow-multibranch): 2.21 Pipeline: SCM Step (workflow-scm-step): 2.7 Pipeline: Step API (workflow-step-api): 2.19 Pipeline: Supporting APIs (workflow-support): 3.3 IncrediBuild Plugin (IncrediBuild): 1.0.0

Add Comment

[cw.chu@asmpt.com (JIRA)]

Chu CW started work on JENKINS-58700

Change By: Chu CW Status: Open In Progress

Add Comment

[cw.chu@asmpt.com (JIRA)]

Chu CW stopped work on JENKINS-58700

Change By: Chu CW Status: In Progress Open

Add Comment

[cw.chu@asmpt.com (JIRA)]

Chu CW updated JENKINS-58700

Jenkins / JENKINS-58700 java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

Change By: Chu CW Status: In Review Progress

Add Comment

[cw.chu@asmpt.com (JIRA)]

Chu CW updated JENKINS-58700

Jenkins / JENKINS-58700 java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

Change By: Chu CW Status: In Progress Review

Add Comment