[JIRA] (JENKINS-59119) Terminating xvnc takes several minutes when running on Kubernetes agents

0 views

Skip to first unread message

fredg02@hotmail.com (JIRA)

unread,

Aug 28, 2019, 7:49:03 AM8/28/19

to jenkinsc...@googlegroups.com

Fred G created an issue

Jenkins /

JENKINS-59119

Terminating xvnc takes several minutes when running on Kubernetes agents

Issue Type:	Bug
Assignee:	Levon Saldamli
Components:	core, xvnc-plugin
Created:	2019-08-28 11:48
Environment:	Jenkins version 2.176.2 LTS OpenShift 3.9 cluster
Priority:	Minor
Reporter:	Fred G

At the end of a build, it seems to hang for several minutes while Xvnc is shutting down ("Terminating xvnc.").

Several observations:

This only happens on dynamic agents running on our OpenShift cluster. It does not happen on static/dedicated build agents or when running on a Jenkins master.
It takes ~6 minutes to terminate when the option "Create a dedicated Xauthority file per build" is enabled. It takes "only" ~2 minutes when this option is disabled. This was reproducible across several builds switching the option on and off.
I was able to narrow down the problem to this line in the code:

 
                                                                launcher.kill(xauthorityEnv); 
                                                            

https://github.com/jenkinsci/xvnc-plugin/blob/master/src/main/java/hudson/plugins/xvnc/Xvnc.java#L240

Since it involves core functionality (Launcher, ProcessTree, etc) I also added "core" as component.
Please find thread dumps below:

 
                                                                "pool-1-thread-16 for JNLP4-connect connection to <cluster-hostname>/<cluster-IP>:50000 id=9736" Id=31 Group=main TIMED_WAITING
	at java.lang.Thread.sleep(Native Method)
	at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:784)
	at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:818)
	at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:805)
	at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:709)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
	at hudson.remoting.Engine$1$$Lambda$3/2070832539.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@195341cb 
                                                            

 
                                                                "Executor #0 for migration-agent-tvwtb : executing xvnc-terminating-test #16 / waiting for JNLP4-connect connection from <agent-IP>/<agent-IP>:35020 id=9736" Id=34283 Group=main TIMED_WAITING on hudson.remoting.UserRequest@480bc704 (in native)
	at java.lang.Object.wait(Native Method)
	-  waiting on hudson.remoting.UserRequest@480bc704
	at java.lang.Object.wait(Unknown Source)
	at hudson.remoting.Request.call(Request.java:177)
	-  locked hudson.remoting.UserRequest@480bc704
	at hudson.remoting.Channel.call(Channel.java:956)
	at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1086)
	at hudson.plugins.xvnc.Xvnc$DisposerImpl.tearDown(Xvnc.java:248)
	at jenkins.tasks.SimpleBuildWrapper$EnvironmentWrapper.tearDown(SimpleBuildWrapper.java:175)
	at hudson.model.Build$BuildExecution.doRun(Build.java:174)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1818)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
 
                                                            