Git polling fails with authentication errors sometimes

[Dave]

I get this result from polling sometimes. This is a multiple SCMs configuration. I don't believe I see this with single SCM setups. I see that it cannot find the temporary credential path, but how is that even supposed to be created?

Started on Feb 12, 2015 9:26:36 AM
Polling SCM changes on u64internal
Using strategy: Default
[poll] Last Built Revision: Revision 36abd7ba96ef8631377aad256931a4140ea2bcbc (origin/feature/TH-3052-implement-xsd-parser-to-determine)
 > /usr/bin/git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repositories
 > /usr/bin/git config remote.origin.url https://xxx.com/stash/scm/th/those.git # timeout=10
Fetching upstream changes from https://xxx.com/stash/scm/th/those.git
 > /usr/bin/git --version # timeout=10
using .gitcredentials to set credentials
 > /usr/bin/git config --local credential.helper store --file=/tmp/git5332354415487210485.credentials # timeout=10
 > /usr/bin/git -c core.askpass=true fetch --tags --progress https://xxx.com/stash/scm/th/those.git +refs/heads/*:refs/remotes/origin/*
 > /usr/bin/git config --local --remove-section credential # timeout=10
FATAL: hudson.plugins.git.GitException: Failed to fetch from https://xxx.com/stash/scm/th/those.git
hudson.util.IOException2: hudson.plugins.git.GitException: Failed to fetch from https://xxx.com/stash/scm/th/those.git
	at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:496)
	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
	at hudson.scm.SCM.poll(SCM.java:373)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.compareRemoteRevisionWith(MultiSCM.java:92)
	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
	at hudson.scm.SCM.poll(SCM.java:373)
	at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1605)
	at hudson.model.AbstractProject._poll(AbstractProject.java:1575)
	at hudson.model.AbstractProject.poll(AbstractProject.java:1493)
	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:462)
	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:491)
	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: hudson.plugins.git.GitException: Failed to fetch from https://xxx.com/stash/scm/th/those.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:668)
	at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:575)
	at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:494)
	... 17 more
Caused by: hudson.plugins.git.GitException: Command "/usr/bin/git -c core.askpass=true fetch --tags --progress https://xxx.com/stash/scm/th/those.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run git credential-store --file=/tmp/git5332354415487210485.credentials get: No such file or directory
error: cannot run git credential-store --file=/tmp/git5332354415487210485.credentials erase: No such file or directory
fatal: Authentication failed

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1591)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1379)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:701)
Done. Took 1.5 sec
No changes

[Dave]

I should add I'm using Jenkins 1.554.2, Git client plug-in 1.16.1 and Git plug-in 2.2.12. It seems like this used to work, but I'm not sure what version of each plug-in I had.

[Mark Waite]

The message indicates that a file which the plugin expects to be on the file system is not there.   That stack trace might hint that there is a race condition in the multiple SCM plugin / git plugin interaction.   

I don't know if the file is missing because the git plugin performed some cleanup, or because the multiple SCMs plugin did some cleanup, or because something outside performed cleanup.

Thanks,

Mark Waite

On Thu, Feb 12, 2015 at 7:39 AM, Dave <dhum...@gmail.com> wrote:

I should add I'm using Jenkins 1.554.2, Git client plug-in 1.16.1 and Git plug-in 2.2.12. It seems like this used to work, but I'm not sure what version of each plug-in I had.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0ea3197f-4b6b-480d-bd56-344bd3cd4caa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Dave]

I was kinda thinking that, any suggestions on how I could debug this? Or does someone know where the numbered credential file is named? If it is based on a timestamp and the SCM plug-in checks both repos with the same credential file, I could see how that could be a problem if one deletes it and the other tries to use it.

[Mark Waite]

You might be able to test the theory by creating a multi-scm job which uses 10 or 20 authenticated git repositories, then run that job multiple times.  If it fails, then it is probably the file delete condition we're theorizing might be there.

If it does not fail, then it is likely something else.  If it does not fail, then you might try adding multiple repos of other types (Subversion, Mercurial, etc.).

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0b2668f7-945e-49ad-adb9-d7894beb9c95%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Dave]

I forgot that this job tries to check all branches as well and I wonder if that is more of the cause. The credentials are stored within the local repository so multiple Git repos probably wouldn't cause conflicts. However, if a single repo checks out multiple branches, it of course does a git checkout so the same .git folder is used. I'm created some test jobs, one with multiple SCMs and one with a single SCM, both configured to check all branches. So far I've not seen these same issues. I still need to make sure the same version of Git is being used between the test jobs and the problem job.

I would also add that although it is intermittent, once it gets in this weird state it stays there until we manually run the job.

[Dave]

I'm able to reproduce the error with my test job if I have it restricted to the same slave that the problem job is using. It happens with the single SCM job that has multiple branches, but I haven't been able to reproduce with just a single branch configurations, so I'm thinking it has to do with a particular version of Git and the multiple branch issue (or single branches always run on master so I don't see it). The slave in question is running 1.7.9.5 which meets the requirement, but not the recommended version of 1.8.

I don't feel like upgrading that slave right now, but I can at least limit the job to other slaves to see if that works around the problem.

[Mark Waite]

I haven't done the detailed review of the git capabilities, but I thought that credentials support only first arrived in git 1.7.9 (or possibly 1.7.10).  Your 1.7.9 machine may have a git version which is too old for that use model.

Does the 1.7.9 based machine checkout successfully from a single SCM job?

Mark Waite

On Fri, Feb 13, 2015 at 9:52 AM, Dave <dhum...@gmail.com> wrote:

I'm able to reproduce the error with my test job if I have it restricted to the same slave that the problem job is using. It happens with the single SCM job that has multiple branches, but I haven't been able to reproduce with just a single branch configurations, so I'm thinking it has to do with a particular version of Git and the multiple branch issue (or single branches always run on master so I don't see it). The slave in question is running 1.7.9.5 which meets the requirement, but not the recommended version of 1.8.

I don't feel like upgrading that slave right now, but I can at least limit the job to other slaves to see if that works around the problem.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20db5c14-70c1-4c27-b0b2-200c274b291d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Dave]

If I run manually, the slave with 1.7.9.5 is able to fetch the changes. Only the polling is an issue. With the problem job switched to running on a different slave with 1.8+, I am not currently seeing the issue.