[JIRA] (JENKINS-62181) Remoting deadlock when launching remote agent

17 views
Skip to first unread message

bart.heupers@gmail.com (JIRA)

unread,
May 6, 2020, 2:43:02 PM5/6/20
to jenkinsc...@googlegroups.com
Bart Heupers created an issue
 
Jenkins / Bug JENKINS-62181
Remoting deadlock when launching remote agent
Issue Type: Bug Bug
Assignee: Jeff Thompson
Attachments: jenkins-remoting-deadlock.log
Components: remoting
Created: 2020-05-06 18:42
Environment: Jenkins 2.231
Remoting version 4.3
Manifest-Version: 1.0
Application-Name: Jenkins Remoting Agent
Permissions: all-permissions
Codebase: *
Trusted-Library: true
Build-Jdk-Spec: 1.8
Version: 4.3
Created-By: Maven Jar Plugin 3.2.0
Main-Class: hudson.remoting.Launcher
Labels: deadlock
Priority: Critical Critical
Reporter: Bart Heupers

After launching an agent it keeps hangin with the follwing text :

 

[05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

 

jstack reports a deadlock in attachment :

 

Found one Java-level deadlock:
=============================
"pool-1-thread-4 for channel id=17236":
waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
which is held by "Channel reader thread: channel"
"Channel reader thread: channel":
waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
which is held by "pool-1-thread-4 for channel id=17236"

 

On april 3 2020 in commit https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d  

a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

 

 

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

bart.heupers@gmail.com (JIRA)

unread,
May 6, 2020, 2:46:02 PM5/6/20
to jenkinsc...@googlegroups.com
Bart Heupers updated an issue
Change By: Bart Heupers
After launching an agent it keeps hangin hanging with the follwing following text :


 

[05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

 

jstack reports a deadlock in attachment :

 

Found one Java-level deadlock:
=============================
"pool-1-thread-4 for channel id=17236":
waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
which is held by "Channel reader thread: channel"
"Channel reader thread: channel":
waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
which is held by "pool-1-thread-4 for channel id=17236"

 

On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

 

 

 

bart.heupers@gmail.com (JIRA)

unread,
May 7, 2020, 3:05:02 AM5/7/20
to jenkinsc...@googlegroups.com
Bart Heupers updated an issue
After launching an agent it keeps hanging with the following text :


 

[05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

 

This happens since a couple of weeks

 

jstack reports a deadlock in attachment :

 

Found one Java-level deadlock:
=============================
"pool-1-thread-4 for channel id=17236":
waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
which is held by "Channel reader thread: channel"
"Channel reader thread: channel":
waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
which is held by "pool-1-thread-4 for channel id=17236"

 

On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

 

 

 

dbeck@cloudbees.com (JIRA)

unread,
May 7, 2020, 4:39:03 AM5/7/20
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 10:54:04 AM5/7/20
to jenkinsc...@googlegroups.com

The “new” method was just an override of an existing synchronized method, though the deadlock does smell like a regression in the new code before the super call.

You have a custom logger on hudson.remoting.Channel (or hudson.remoting) at FINE or below I think? As a workaround, try removing that logger.

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:00:05 AM5/7/20
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
 
Change By: Jesse Glick
Labels: deadlock regression

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:00:14 AM5/7/20
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Jesse Glick
Change By: Jesse Glick
Assignee: Jeff Thompson Jesse Glick

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:00:14 AM5/7/20
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Component/s: core
Component/s: remoting

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:00:16 AM5/7/20
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-62181
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:01:02 AM5/7/20
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Labels: deadlock lts-candidate regression

jglick@cloudbees.com (JIRA)

unread,
May 7, 2020, 11:06:03 AM5/7/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Progress Review
Reply all
Reply to author
Forward
0 new messages