[JIRA] (JENKINS-40437) Git cloning remote repo fails randomly

3 views
Skip to first unread message

jonas@pix4d.com (JIRA)

unread,
Dec 14, 2016, 4:40:02 AM12/14/16
to jenkinsc...@googlegroups.com
Jonas created an issue
 
Jenkins / Bug JENKINS-40437
Git cloning remote repo fails randomly
Issue Type: Bug Bug
Assignee: Stephen Connolly
Components: core, credentials-plugin, git-client-plugin, git-plugin
Created: 2016/Dec/14 9:39 AM
Environment: OS: debian testing
git-plugin version: 3.0.1 (was happening on a much older version as well)
git-client-plugin: 2.1.0 (was happening on an older version as well)
credentials-plugin: 2.1.10 (was happening on an older version as well)
Jenkins is running in an lxc container
Priority: Minor Minor
Reporter: Jonas

We randomly get `GitException`s, caused by a `java.io.IOException` when running `CliGitAPIImpl.createSshKeyFile` (see stacktrace below). This happens randomly on different nodes (i.e. it does not always happen, and not always on the same node).

In the job configuration, we simply tell Jenkins to use the corresponding git credentials. The error seems to mean that sometimes, getting the credentials fails for some reason. We thought about a permission problem, but the thing is that it sometimes works without changing the node at all.

09:01:44 > git --version # timeout=10
09:01:44 using GIT_SSH to set credentials Gitmirror key
09:01:44 ERROR: Error cloning remote repo 'origin'
09:01:44 hudson.plugins.git.GitException: Failed to setup credentials
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1491)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:512)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
09:01:44 at hudson.remoting.UserRequest.perform(UserRequest.java:153)
09:01:44 at hudson.remoting.UserRequest.perform(UserRequest.java:50)
09:01:44 at hudson.remoting.Request$2.run(Request.java:332)
09:01:44 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
09:01:44 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:01:44 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:01:44 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:01:44 at java.lang.Thread.run(Thread.java:745)
09:01:44 at ......remote call to tmp-build-debian-jessie(Native Method)
09:01:44 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
09:01:44 at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
09:01:44 at hudson.remoting.Channel.call(Channel.java:781)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
09:01:44 at sun.reflect.GeneratedMethodAccessor416.invoke(Unknown Source)
09:01:44 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:01:44 at java.lang.reflect.Method.invoke(Method.java:606)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
09:01:44 at com.sun.proxy.$Proxy80.execute(Unknown Source)
09:01:44 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1054)
09:01:44 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094)
09:01:44 at hudson.scm.SCM.checkout(SCM.java:495)
09:01:44 at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
09:01:44 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
09:01:44 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
09:01:44 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
09:01:44 at hudson.model.Run.execute(Run.java:1720)
09:01:44 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
09:01:44 at hudson.model.ResourceController.execute(ResourceController.java:98)
09:01:44 at hudson.model.Executor.run(Executor.java:410)
09:01:44 Caused by: java.io.IOException: No such file or directory
09:01:44 at java.io.UnixFileSystem.createFileExclusively(Native Method)
09:01:44 at java.io.File.createTempFile(File.java:2001)
09:01:44 at java.io.File.createTempFile(File.java:2047)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1501)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1430)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:512)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
09:01:44 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
09:01:44 at hudson.remoting.UserRequest.perform(UserRequest.java:153)
09:01:44 at hudson.remoting.UserRequest.perform(UserRequest.java:50)
09:01:44 at hudson.remoting.Request$2.run(Request.java:332)
09:01:44 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
09:01:44 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
09:01:44 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
09:01:44 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
09:01:44 at java.lang.Thread.run(Thread.java:745)
09:01:44 ERROR: null
09:01:44 Retrying after 10 seconds
09:01:54 Wiping out workspace first.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

o.v.nenashev@gmail.com (JIRA)

unread,
Dec 14, 2016, 7:58:02 AM12/14/16
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Mark Waite
Change By: Oleg Nenashev
Assignee: Stephen Connolly Mark Waite

o.v.nenashev@gmail.com (JIRA)

unread,
Dec 14, 2016, 7:58:02 AM12/14/16
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Component/s: core
Component/s: credentials-plugin

o.v.nenashev@gmail.com (JIRA)

unread,
Dec 14, 2016, 7:58:02 AM12/14/16
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-40437
 
Re: Git cloning remote repo fails randomly

It's an issue in the Git plugin due to its Credentials integration. CC Mark Waite, likely a duplicate

mark.earl.waite@gmail.com (JIRA)

unread,
Dec 14, 2016, 9:39:01 AM12/14/16
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
 
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Dec 14, 2016, 9:40:01 AM12/14/16
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-40437
 
Re: Git cloning remote repo fails randomly

It may be an issue in the git plugin, though I'd be truly surprised if it were. The stack trace shows:

{{09:01:44 at java.io.UnixFileSystem.createFileExclusively(Native Method)


09:01:44 at java.io.File.createTempFile(File.java:2001)
09:01:44 at java.io.File.createTempFile(File.java:2047)
09:01:44 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1501)
}}

If a UnixFileSystem call to createTempFile / createFileExclusively fails with "No such file or directory", I would expect something very interesting is happening to the temporary files on that system. The createTempFile() method works very hard to create a temporary file and "No such file or directory" typically would mean a parent directory component no longer exists.

For example, is there something that is removing temporary directories while the jobs are running?

Is there some form of cleanup process on that computer which ignores the age of a directory or the age of a file and just deletes everything in the temporary directory?

jonas@pix4d.com (JIRA)

unread,
Dec 14, 2016, 11:54:02 AM12/14/16
to jenkinsc...@googlegroups.com
Jonas resolved as Fixed
 
Change By: Jonas
Status: Open Resolved
Resolution: Fixed

jonas@pix4d.com (JIRA)

unread,
Dec 14, 2016, 11:54:02 AM12/14/16
to jenkinsc...@googlegroups.com
Jonas commented on Bug JENKINS-40437
 
Re: Git cloning remote repo fails randomly

Oh, you were right! Something is removing directories while the jobs are running =/. It has to be on our side, then.

Thanks a lot!

mark.earl.waite@gmail.com (JIRA)

unread,
Dec 14, 2016, 3:48:01 PM12/14/16
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Fixed
 
Change By: Mark Waite
Status: Resolved Closed
Reply all
Reply to author
Forward
0 new messages