[JIRA] (JENKINS-51191) hudson.remoting.Channel$CallSiteStackTrace: Remote call to

656 views
Skip to first unread message

tomasz.dudala@nsn.com (JIRA)

unread,
May 8, 2018, 3:58:02 AM5/8/18
to jenkinsc...@googlegroups.com
tdudala created an issue
 
Jenkins / Bug JENKINS-51191
hudson.remoting.Channel$CallSiteStackTrace: Remote call to
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-client-plugin, remoting
Created: 2018-05-08 07:57
Environment: Jenkins 2.114
Java 1.8.0_131
Remoting 3.19
git-client-plugin 2.7.1
Priority: Critical Critical
Reporter: tdudala

My jobs fails randomly (about 10% are failed), they are defined as pipelines. Master and jenkins slaves have exactly the same environment.

 
Generally, build fails when it is going to load/execute pipeline script in remote machine, it ends with following message:
 
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to 192.168.3.27
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
        at hudson.remoting.Channel.call(Channel.java:955)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy98.withRepository(Unknown Source)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
        at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1245)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1218)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
        at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
        at hudson.security.ACL.impersonate(ACL.java:290)
        at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.NoClassDefFoundError: hudson.model.Computer
    at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:72)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
    at hudson.remoting.UserRequest.perform(UserRequest.java:212)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:369)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused: java.io.IOException: Remote call on 192.168.3.27 failed
    at hudson.remoting.Channel.call(Channel.java:961)
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
    at com.sun.proxy.$Proxy98.withRepository(Unknown Source)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
    at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1245)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1218)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
    at hudson.security.ACL.impersonate(ACL.java:290)
    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
 
 
On Jenkins node log I can find:
 
 
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.19
This is a Unix agent
Evacuated stdout
May 04, 2018 12:23:05 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class net.bull.javamelody.RemoteCallHelper$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
May 04, 2018 12:23:27 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Agent successfully connected and online
May 04, 2018 9:16:41 AM jenkins.model.Jenkins <clinit>
SEVERE: Failed to load Jenkins.class
com.thoughtworks.xstream.InitializationException: Could not instantiate converter : com.thoughtworks.xstream.converters.extended.DurationConverter : null
    at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:904)
    at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:867)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:574)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:496)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:465)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:411)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:378)
    at hudson.util.XStream2.<init>(XStream2.java:112)
    at jenkins.model.Jenkins.<clinit>(Jenkins.java:5103)
    at hudson.model.Computer.<clinit>(Computer.java:1761)
    at hudson.util.ProcessTree.getKillers(ProcessTree.java:157)
    at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:220)
    at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:662)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:670)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
    at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:591)
    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090)
    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081)
    at hudson.remoting.UserRequest.perform(UserRequest.java:212)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:369)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:897)
    ... 26 more
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
    at com.sun.proxy.$Proxy5.getResources2(Unknown Source)
    at hudson.remoting.RemoteClassLoader.findResources(RemoteClassLoader.java:481)
    at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
    at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
    at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
    at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
    at javax.xml.datatype.FactoryFinder$1.run(FactoryFinder.java:296)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.xml.datatype.FactoryFinder.findServiceProvider(FactoryFinder.java:292)
    at javax.xml.datatype.FactoryFinder.find(FactoryFinder.java:268)
    at javax.xml.datatype.DatatypeFactory.newInstance(DatatypeFactory.java:145)
    at com.thoughtworks.xstream.converters.extended.DurationConverter$1.getFactory(DurationConverter.java:39)
    at com.thoughtworks.xstream.converters.extended.DurationConverter.<init>(DurationConverter.java:36)
    ... 31 more
Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at hudson.remoting.Request.call(Request.java:177)
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
    ... 44 more
 
May 04, 2018 9:16:41 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@14f02673
java.lang.ExceptionInInitializerError
    at hudson.model.Computer.<clinit>(Computer.java:1761)
    at hudson.util.ProcessTree.getKillers(ProcessTree.java:157)
    at hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:220)
    at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:662)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:670)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
    at hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:669)
    at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:591)
    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090)
    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081)
    at hudson.remoting.UserRequest.perform(UserRequest.java:212)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:369)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.thoughtworks.xstream.InitializationException: Could not instantiate converter : com.thoughtworks.xstream.converters.extended.DurationConverter : null
    at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:904)
    at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:867)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:574)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:496)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:465)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:411)
    at com.thoughtworks.xstream.XStream.<init>(XStream.java:378)
    at hudson.util.XStream2.<init>(XStream2.java:112)
    at jenkins.model.Jenkins.<clinit>(Jenkins.java:5103)
    ... 18 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:897)
    ... 26 more
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
    at com.sun.proxy.$Proxy5.getResources2(Unknown Source)
    at hudson.remoting.RemoteClassLoader.findResources(RemoteClassLoader.java:481)
    at java.lang.ClassLoader.getResources(ClassLoader.java:1142)
    at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
    at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
    at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
    at javax.xml.datatype.FactoryFinder$1.run(FactoryFinder.java:296)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.xml.datatype.FactoryFinder.findServiceProvider(FactoryFinder.java:292)
    at javax.xml.datatype.FactoryFinder.find(FactoryFinder.java:268)
    at javax.xml.datatype.DatatypeFactory.newInstance(DatatypeFactory.java:145)
    at com.thoughtworks.xstream.converters.extended.DurationConverter$1.getFactory(DurationConverter.java:39)
    at com.thoughtworks.xstream.converters.extended.DurationConverter.<init>(DurationConverter.java:36)
    ... 31 more
Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at hudson.remoting.Request.call(Request.java:177)
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
    ... 44 more

 

 

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
May 8, 2018, 1:47:01 PM5/8/18
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
May 8, 2018, 1:47:02 PM5/8/18
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-51191
 
Re: hudson.remoting.Channel$CallSiteStackTrace: Remote call to

Any chance that the agent process is running in Java 9, instead of Java 8? Jenkins masters and Jenkins agents are not supported with Java 9. The xstream code has known problems with the Java 9 module system.

tomasz.dudala@nsn.com (JIRA)

unread,
May 9, 2018, 2:01:03 AM5/9/18
to jenkinsc...@googlegroups.com
tdudala commented on Bug JENKINS-51191

Nope, both slave and master are on Java 1.8.0_131 

jthompson@cloudbees.com (JIRA)

unread,
Nov 1, 2018, 3:49:02 PM11/1/18
to jenkinsc...@googlegroups.com

I'm at a loss to explain this one.

I've seen a few other reports where people using pipeline were getting an InterruptedException. I haven't yet seen any indication of what caused it. Maybe an interaction between different plugins.

I see timestamps in the agent logs but none in the master. Do these different events and logs match up?

My only suggestion is to try and isolate or reproduce the problem somehow. 

Have you seen any changes in the behavior or the frequency?

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

romanzwi000@gmx.at (JIRA)

unread,
Nov 16, 2018, 3:12:03 AM11/16/18
to jenkinsc...@googlegroups.com

Is the correct version of slave.jar running on the slave? You can download it from <yourjenkinsurl>/jnlpJars/slave.jar
You can use the Versions Node Monitors plugin to have it checked automatically. It also adds the versions to the node list.

jthompson@cloudbees.com (JIRA)

unread,
Dec 11, 2018, 2:11:02 PM12/11/18
to jenkinsc...@googlegroups.com
Jeff Thompson closed an issue as Cannot Reproduce
 

Closing for lack of sufficient diagnostics and information to reproduce after no response for quite a while.

Change By: Jeff Thompson
Status: Open Closed
Resolution: Cannot Reproduce

tomasz.dudala@nsn.com (JIRA)

unread,
Oct 17, 2019, 2:12:01 AM10/17/19
to jenkinsc...@googlegroups.com
tdudala commented on Bug JENKINS-51191

Issue has been solved after upgrading JAVA to 1.8.0_171

 

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jthompson@cloudbees.com (JIRA)

unread,
Oct 17, 2019, 10:51:33 AM10/17/19
to jenkinsc...@googlegroups.com

Interesting. I don't know what would have caused that change. I just scanned the Java changelog and didn't see any obvious causes.

Thanks for sharing the info here!

Reply all
Reply to author
Forward
0 new messages