[JIRA] (JENKINS-61103) LinkageError on agent following RemotingSystemException(InterruptedException)

923 views
Skip to first unread message

tom.gl@free.fr (JIRA)

unread,
Feb 15, 2020, 4:07:04 PM2/15/20
to jenkinsc...@googlegroups.com
Thomas de Grenier de Latour created an issue
 
Jenkins / Bug JENKINS-61103
LinkageError on agent following RemotingSystemException(InterruptedException)
Issue Type: Bug Bug
Assignee: Jeff Thompson
Components: remoting
Created: 2020-02-15 21:06
Environment: Jenkins 2.204.2 (remoting 3.36.1)
Linux OpenJDK 1.8.0_242
Priority: Major Major
Reporter: Thomas de Grenier de Latour

I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.

I'm trying to improve resilience of the agent's RemoteClassLoader when it receives a RemotingSystemException caused by an InterruptedException.

The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a LinkageError.
This is not something new, see JENKINS-36991 / PR #94. This one got fixed by adding some try/catch/retry logic to the affected part of the RemoteClassLoader#findClass code. This way, the interruption can be deferred until after the class has been successfully loaded.

But there are other parts of the RemoteClassLoader code which are not protected yet, despite being on the critical path of classes loading/initialization.

See for instance JENKINS-56767: the exception occurs in a call to RemoteClassLoader#getResource made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.

I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.

  • error (slave log) when this first Maven job execution gets interrupted: 
    Feb 10, 2020 5:51:39 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@38569db8
java.lang.ExceptionInInitializerError
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1090)
	at java.lang.Class.getResource(Class.java:2267)
	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:128)
	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
	at hudson.model.Run.<clinit>(Run.java:2535)
	... 11 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 22 more

see RemoteClassLoader.java:426

  • error (slave log) when trying to run any other Maven job on the same agent after that:
Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c
java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

And with the same manual procedure, I also sometimes get the fatal exception in an other part of RemoteClassLoader#findClass which is not protected covered by the retry logic either (at least, not for RemotingSystemException):

  • error (slave log) when this first Maven job gets interrupted:
2020-02-11T12:16:49.933+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@601f4e25
java.lang.ExceptionInInitializerError
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:89)
	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:79)
	at org.eclipse.sisu.plexus.ClassRealmManager.<clinit>(ClassRealmManager.java:66)
	... 18 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 25 more

see RemoteClassLoader.java:315

  • error (slave log) when trying to run any other Maven job on the same agent after that:
2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46
java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

 

Both these cases could be fixed by adding more "try / catch / retry with deferred interruption" logic on the relevant parts of the RemoteClassLoader code,
but instead I propose to do it at a slightly lower level, on methods of the IClassLoader proxy (that's the com.sun.proxy.$Proxy5.getResource2(Unknown Source)
and com.sun.proxy.$Proxy5.fetch(Unknown Source) in the above stack traces).

I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any LinkageError reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any NoClassDefFoundError, as one would expect.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

tom.gl@free.fr (JIRA)

unread,
Feb 15, 2020, 4:47:04 PM2/15/20
to jenkinsc...@googlegroups.com
Thomas de Grenier de Latour updated an issue
Change By:

tom.gl@free.fr (JIRA)

unread,
Feb 15, 2020, 5:53:04 PM2/15/20
to jenkinsc...@googlegroups.com
Thomas de Grenier de Latour updated an issue

whoops, wrong issue link, fixed
I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.

I'm trying to improve resilience of the agent's {{RemoteClassLoader}} when it receives a {{RemotingSystemException}} caused by an {{InterruptedException}}.

The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a {{LinkageError}}.
This is not something new, see JENKINS-36991 / [PR #94|https://github.com/jenkinsci/remoting/pull/94]. This one got fixed by adding some try/catch/retry logic to the affected part of the {{RemoteClassLoader#findClass}} code. This way, the interruption can be deferred until after the class has been successfully loaded.


But there are other parts of the {{RemoteClassLoader}} code which are not protected yet, despite being on the critical path of classes loading/initialization.

See for instance JENKINS- 56767 51854 : the exception occurs in a call to {{RemoteClassLoader#getResource}} made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.


I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.
- error (slave log) when this first Maven job execution gets interrupted:
{code:java}
{code}

see [RemoteClassLoader.java:426|https://github.com/jenkinsci/remoting/blob/remoting-3.36.1/src/main/java/hudson/remoting/RemoteClassLoader.java#L426]
- error (slave log) when trying to run any other Maven job on the same agent after that:

{code:java}

Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c
java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
And with the same manual procedure, I also sometimes get the fatal exception in an other part of {{RemoteClassLoader#findClass}} which is not protected covered by the retry logic either (at least, not for {{RemotingSystemException}}):
- error (slave log) when this first Maven job gets interrupted:

{code:java}
{code}
see [RemoteClassLoader.java:315|https://github.com/jenkinsci/remoting/blob/remoting-3.36.1/src/main/java/hudson/remoting/RemoteClassLoader.java#L315]
- error (slave log) when trying to run any other Maven job on the same agent after that:

{code:java}

2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46
java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager
at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
 

Both these cases could be fixed by adding more "_try / catch / retry with deferred interruption_" logic on the relevant parts of the {{RemoteClassLoader}} code,

but instead I propose to do it at a slightly lower level, on methods of the {{IClassLoader}} proxy (that's the {{com.sun.proxy.$Proxy5.getResource2(Unknown Source)}}
and {{com.sun.proxy.$Proxy5.fetch(Unknown Source)}} in the above stack traces).

I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any {{LinkageError}} reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any {{NoClassDefFoundError}}, as one would expect.

o.v.nenashev@gmail.com (JIRA)

unread,
Feb 19, 2020, 12:17:04 PM2/19/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Component/s: maven-plugin

tom.gl@free.fr (JIRA)

unread,
Feb 20, 2020, 4:01:04 AM2/20/20
to jenkinsc...@googlegroups.com
Thomas de Grenier de Latour commented on Bug JENKINS-61103
 
Re: LinkageError on agent following RemotingSystemException(InterruptedException)

For information, another occurrence of this issue we've had in production a few days ago (not involving the Maven plugin):

  • user launch a job, and then cancel it a few seconds later (edited Apache logs): 
    IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:24 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/build?delay=0sec HTTP/1.1" 201 241 "http://SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:32 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/1/stop HTTP/1.1" 302 267 "http://SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
  • slave.log at the time of interruption (it's similar to one of my previous examples, in RemoteClassLoader#findResource() / RemoteClassLoader.java:426): 
    Feb 17, 2020 4:17:32 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@8a955ea
java.lang.ExceptionInInitializerError
	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:44)
	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
	at hudson.model.Computer.<clinit>(Computer.java:1761)
	at hudson.util.ProcessTree.get(ProcessTree.java:414)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
	at java.lang.Class.getResource(Class.java:2267)
	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
	at hudson.security.Permission.<clinit>(Permission.java:328)
	... 14 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 24 more
  • slave.log when trying to execute jobs again (we've then had to disconnect/reconnect the agent): 
    Feb 17, 2020 4:20:21 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@2813c722
java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
	at hudson.util.ProcessTree.get(ProcessTree.java:414)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Feb 17, 2020 4:23:41 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@73bb794a
java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
	at hudson.util.ProcessTree.get(ProcessTree.java:414)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

romanek.adam@gmail.com (JIRA)

unread,
Mar 6, 2020, 7:33:03 AM3/6/20
to jenkinsc...@googlegroups.com

Me and my team have been experiencing this issue for more than a year now. In our environment it breaks our Amazon EC2 slaves.

Here's the initial ExceptionInInitializerError found in remoting.log on one of our slaves:

Mar 06, 2020 11:52:33 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@75f3b9a6
java.lang.ExceptionInInitializerError
	at hudson.util.ProcessTree.get(ProcessTree.java:414)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at hudson.model.Computer.<clinit>(Computer.java:1362)
	... 11 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 16 more

From there one can only see NoClassDefFoundError:

Mar 06, 2020 11:56:46 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@1c3a1ce1
java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
	at hudson.util.ProcessTree.get(ProcessTree.java:414)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
 
                                                            

                                                        
 
The issue occurs when a build finishes and ProcessTreeKiller is requested by Jenkins Master to clean up any left-over processes running on a given slave.
 
Unfortunately due to a bug in JDK the issue is rather hard to track down, as the logs in our builds have always had this NoClassDefFoundError exception logged, never the one causing all the trouble - ExceptionInInitializerError.
 
I'm really looking forward to get it fixed. Please let me know if I could help somehow.
 		 
                                                
 
                                            
                                        
 		 
                                

                        
 		 
                
 
                
 
                     
                        
 
                            
                                
 
                                     This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)  
                                     
                                        
 
                                            
                                                
 
                                                     Atlassian logo  
                                                
 
                                            
                                        
 		 
                                
 
                            
                        
 		 
                
 
            
        
   
    


tom.gl@free.fr (JIRA)
unread,
Mar 6, 2020, 7:08:03 PM3/6/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Thanks Adam for the pointer to JDK bug 8051847, interesting reading. I didn't know about https://github.com/kohsuke/eiie-logger, that's pretty clever!
 		 
                                                
 
                                            
                                        
 		 
                                
tom.gl@free.fr (JIRA)
unread,
Apr 28, 2020, 11:53:03 AM4/28/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     Thomas de Grenier de Latour updated an issue  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
                                
 
                                     
                                      
                                
 
                                
 
                                     
                                        
 
                                            

                                        
 		 
                                
 
                                
 
                                     
                                        
 
                                            
                                                
 
                                                    Change By:
Reply all
Reply to author
Forward
0 new messages