[JIRA] (JENKINS-38072) remote file operation failed: due to org.jenkinsci.plugins.gitclient.Git.getClient

151 views
Skip to first unread message

jp@jpsim.com (JIRA)

unread,
Sep 8, 2016, 2:50:03 PM9/8/16
to jenkinsc...@googlegroups.com
JP Simard created an issue
 
Jenkins / Bug JENKINS-38072
remote file operation failed: due to org.jenkinsci.plugins.gitclient.Git.getClient
Issue Type: Bug Bug
Assignee: Mark Waite
Attachments: jenkins_systeminfo.txt
Components: git-client-plugin, git-plugin
Created: 2016/Sep/08 6:49 PM
Environment: Jenkins 2.7.2
Priority: Critical Critical
Reporter: JP Simard

Getting the following exception and stack trace:

java.io.IOException: remote file operation failed: /Users/realm/workspace/objc_pr at hudson.remoting.Channel@4ec66dd4:sf_host1_3: java.io.IOException: Remote call on sf_host1_3 failed
	at hudson.FilePath.act(FilePath.java:1014)
	at hudson.FilePath.act(FilePath.java:996)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:741)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:733)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1080)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1741)
	at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Remote call on sf_host1_3 failed
	at hudson.remoting.Channel.call(Channel.java:789)
	at hudson.FilePath.act(FilePath.java:1007)
	... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy11
	at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
	at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:143)
	at hudson.remoting.Channel.export(Channel.java:621)
	at hudson.remoting.Channel.export(Channel.java:590)
	at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)
	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:157)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:166)
	at hudson.remoting.UserRequest.perform(UserRequest.java:128)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:62)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to sf_host1_3(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:781)
	... 15 more
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 8, 2016, 3:05:01 PM9/8/16
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 8, 2016, 3:10:02 PM9/8/16
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-38072
 
Re: remote file operation failed: due to org.jenkinsci.plugins.gitclient.Git.getClient

The jenkins_systeminfo.txt indicates that your master node is running JDK 8 (good), but the report does not indicate what JDK version is running on the agent where the failure is reported. Is it also running JDK 8 or is it running an earlier JDK?

The jenkins_systeminfo.txt indicates that your master node is running Linux kernel 4.4.0, which probably means that you're running Ubuntu 16. Is that the operating system of the server? What operating system is running on the agent which shows the failure?

Are there any other hints you can suggest which might point to differences between the agent and the master?

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 8, 2016, 3:13:02 PM9/8/16
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-38072
The  [^jenkins_systeminfo.txt] indicates that your master node is running JDK 8 (good), but the report does not indicate what JDK version is running on the agent where the failure is reported.  Is it also running JDK 8 or is it running an earlier JDK?

The  [^jenkins_systeminfo.txt] indicates that your master node is running Linux kernel 4.4.0, which probably means that you're running Ubuntu 16.  Is that the operating system of the server?  What operating system is running on the agent which shows the failure?

 The  [^jenkins_systeminfo.txt] shows that you're running git plugin 2.5.3 and git client plugin 1.21.0.  Since you're also running blue-ocean on that machine, I assume that means you're willing to experiment.  Could you try installing git plugin 2.6.0 to see if it resolves the issue?  I'm not aware of any reason it would resolve the issue, but it is a newer version and was specifically built to depend on git client plugin 1.21.0.

Are there any other hints you can suggest which might point to differences between the agent and the master?

jp@jpsim.com (JIRA)

unread,
Sep 8, 2016, 4:44:04 PM9/8/16
to jenkinsc...@googlegroups.com

The agent is running the following java version:

$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

Not sure what the server is running. Agent is running Mac OS X 10.11.6 (El Capitan).

I'll install git plugin 2.6.0 and report back with whether or not the issue is resolved.

jp@jpsim.com (JIRA)

unread,
Sep 8, 2016, 4:47:02 PM9/8/16
to jenkinsc...@googlegroups.com

After installing git plugin 2.6.0 and restarting, the issue still occurs.

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 8, 2016, 5:26:02 PM9/8/16
to jenkinsc...@googlegroups.com

stackoverflow indicates that NoClassDefFoundError indicates a problem reading the class definition of a class which was found, and notes that is different than if the bytecode for the class could not be found. Unfortunately, I don't know what could be done with that information.

Is there a difference in JVM implementations which could be tested? For example, if you run the same operations on an agent temporarily started on the master, does the same problem occur? That would then tend to use the same JVM as the master.

jp@jpsim.com (JIRA)

unread,
Sep 8, 2016, 5:48:01 PM9/8/16
to jenkinsc...@googlegroups.com

So the fact that the server has JDK 1.8.0_101 and the agent has 1.8.0_102 might be the issue, you think?

The release notes don't seem to indicate that anything related to this would be different: http://www.oracle.com/technetwork/java/javase/8u102-relnotes-3021767.html

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 8, 2016, 6:09:01 PM9/8/16
to jenkinsc...@googlegroups.com

No, I don't think the difference is between JDK 8 1.101 and JDK 8 1.102. Those two versions are very, very close to one another (I believe mostly a packaging change between those two versions). I was making a wild guess that it might be a difference between the JDK 8 implementation on Linux and the JDK 8 implementation on MacOS. I must admit freely that I am making a wild guess, since I don't know how to duplicate the problem you're seeing.

I don't have ready access to a MacOS machine, so I can't run my own tests to see if it works. You might run a temporary Jenkins server on MacOS and run a temporary agent on the same MacOS machine.

Sorry that I'm making wild guesses, but I don't have other ideas to suggest...

jp@jpsim.com (JIRA)

unread,
Sep 8, 2016, 6:17:01 PM9/8/16
to jenkinsc...@googlegroups.com

I appreciate your help...

ingunawa@cisco.com (JIRA)

unread,
Sep 27, 2016, 6:15:02 PM9/27/16
to jenkinsc...@googlegroups.com

I have same exact remote file operation failure still on Jenkins 1.625.3.
The Jenkins master uses Oracle JDK jdk1.8.0_92 on RHEL 6 and the agent/node is on Ubuntu 14.04. Master is connecting to the slave using SSH and I set slave to run Oracle JDK 1.7.0_79.

This error often occurs and I can't pinpoint what is the problem.
The git-client plugin version is 1.19.6 and git plugin version 2.4.4

The stacktrace output:
Building remotely on sjc-bld139-lnx (cf-verify LinuxDataCenterUbuntu14.04.2) in workspace /scratch/workspace/workspace/VMS-Cloud-Deployments/deploy-vms-2.2.0-daily-P-4

Deleting project workspace... java.io.IOException: remote file operation failed: /scratch/workspace/workspace/VMS-Cloud-Deployments/deploy-vms-2.2.0-daily-P-4 at hudson.remoting.Channel@44bd60f8:sjc-bld139-lnx: java.io.IOException: Remote call on sjc-bld139-lnx failed
at hudson.FilePath.act(FilePath.java:987)
at hudson.FilePath.act(FilePath.java:969)
at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
at hudson.plugins.git.GitSCM.createClient(GitSCM.java:752)
at hudson.plugins.git.GitSCM.createClient(GitSCM.java:744)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1091)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1275)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: Remote call on sjc-bld139-lnx failed
at hudson.remoting.Channel.call(Channel.java:789)
at hudson.FilePath.act(FilePath.java:980)
... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.reflect.Proxy.newInstance(Proxy.java:764)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:755)


at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:143)
at hudson.remoting.Channel.export(Channel.java:621)
at hudson.remoting.Channel.export(Channel.java:590)
at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1075)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)


at hudson.remoting.UserRequest._serialize(UserRequest.java:157)
at hudson.remoting.UserRequest.serialize(UserRequest.java:166)
at hudson.remoting.UserRequest.perform(UserRequest.java:128)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to sjc-bld139-lnx(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)

Is there anything I should upgrade? Should all node/agent also run Oracle JDK 1.8?
Thank you

-Indra

ingunawa@cisco.com (JIRA)

unread,
Sep 27, 2016, 6:18:01 PM9/27/16
to jenkinsc...@googlegroups.com

Shall OpenJDK 8 good enough to be used on all slaves and that should match the Oracle JDK 1.8.0_92 on the master?

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 27, 2016, 7:26:01 PM9/27/16
to jenkinsc...@googlegroups.com

My best advice is to run the master and the agents on the same Java version.

jody.shumaker@gmail.com (JIRA)

unread,
May 30, 2018, 11:56:02 AM5/30/18
to jenkinsc...@googlegroups.com

We just had an issue occur that seems similar or identical to this this issue occur on 2.107.2.  Slave was an openstack cloud image that has not been changed, and same configuration + source worked fine earlier in the day on a different instance of the image.  I scheduled termination on the slave where the issue occurred and we tried re-running on a new instance of the slave, same base image, and it worked. Suggests to me this is some state issue on the slave?

Stack trace:

hudson.remoting.Channel$CallSiteStackTrace: Remote call to docker-host-1188
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
		at hudson.remoting.Channel.call(Channel.java:952)
		at hudson.FilePath.act(FilePath.java:998)
		at hudson.FilePath.act(FilePath.java:987)
		at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:795)
		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:786)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1154)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
		at hudson.security.ACL.impersonate(ACL.java:290)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
	at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:162)
	at hudson.remoting.Channel.export(Channel.java:765)
	at hudson.remoting.Channel.export(Channel.java:729)
	at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:198)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1218)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:250)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:259)
	at hudson.remoting.UserRequest.perform(UserRequest.java:221)
	at hudson.remoting.UserRequest.perform(UserRequest.java:53)
	at hudson.remoting.Request$2.run(Request.java:364)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
Caused: java.io.IOException: Remote call on docker-host-1188 failed
	at hudson.remoting.Channel.call(Channel.java:958)
	at hudson.FilePath.act(FilePath.java:998)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/T_Custom-Stack-Start_master-W57EAG3UPWEGWJMNS3HMILIWXQKKPAL5WT4DF7IMUS7ZH2YSE56Q at hudson.remoting.Channel@4701282f:docker-host-1188
	at hudson.FilePath.act(FilePath.java:1005)
	at hudson.FilePath.act(FilePath.java:987)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:795)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:786)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1154)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

br.madhu336@gmail.com (JIRA)

unread,
Jan 9, 2019, 6:26:03 AM1/9/19
to jenkinsc...@googlegroups.com

Sometimes the issue is because, 

You might have recently updated the version of JAVA in the slave machine. hence there will be a mis-match in the JAVA version of the slave machine and the JAVA version of the slave which master is storing.

In that case, disconnect the slave and reconnect it. 

i faced this issue couple of times and it worked for me after doing the above mentioned stuff
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
Reply all
Reply to author
Forward
0 new messages