Git Plugin Issues
Burrows, John
Hi All,
I have seen a few write-ups about this and just want to get some clarification, as I have not been able to solve this as of yet in our environments.
We are using Jenkins Master v2.89.4 on Windows 2008 R2 Servers. We have the latest Git (3.8.0) & Git Client (2.7.1) plugins installed.
We have multiple Jenkins Nodes/Slave Servers setup, using Windows, Linux, AIX, Solaris as the OS. The Agent Remoting version is v3.14, Java version matches between the Jenkins Masters (v1.8.0) and all of the Jenkins Nodes/Slaves.
We launch the Jenkins Node/Slave Agents using Java Web Start (Windows OS) or via SSH (Linux, AIX, Solaris). The Windows JNLP Agents run using a Service ID that has Administration privileges on the Windows Server. The SSH agents have a local id (same name as the Service ID) with the privileges necessary as well.
BitBucket is setup to utilize SSH for communications.
When we require the use of Git to interact with our BitBucket/Git Projects and Repositories, we set the following up to ensure proper communication between the Jenkins Nodes/Slave Servers and BitBucket/Git:
1) We ensure that the Java version in use has the certificates in the cacerts for the BitBucket server we interact with for source code
2) We create ssh keys for the above mentioned Service ID and ensure that they are in the .ssh folder under the Users Profile and Home directories.
3) We copy the same rsa_id.pub ssh key to BitBucket and add the key to the Service ID account SSH Keys listings under the Service ID’s Profile.
4) We have credentials created on each Jenkins Master (Network ID and Password) for the Service Account, as the BitBucket server utilizes LDAP to authenticate as well as the Windows Servers and on the Linux, AIX and Solaris machines, we setup the local user with the same password. These credentials are chosen in the SCM Section of the Jenkins Job for the Git Repository configuration section.
Recently, we have come across the issue where the interaction with Git fails on some Windows Servers and some Linux or AIX servers. While we can have one Windows server failing and another working just fine on the same Jenkins Master. Same with the other OS based Nodes/Slaves. This was working fine earlier this year, but in the last 6-8 weeks, it has started to fail for some existing servers that worked fine before, and for some servers that have been Jenkins Nodes, but we recently added the interaction to Git.
We have 10 Jenkins Masters all set up identically, and as stated above, follow the same setup of every Jenkins Node/Slave as well.
Here are the errors we are running into (Credentials Unix BitBucket Groovy are the name of the credentials commonly used across all platforms):
Building remotely on XXXJENKINSNODEXXX in workspace d:\scm\checkout\John-Test
> D:\SCM\Apps\Git\2.8.2\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> D:\SCM\Apps\Git\2.8.2\bin\git.exe config remote.origin.url ssh://g...@bitbucket.XXXXXXX.com:7999/projectname/reponame.git # timeout=10
Fetching upstream changes from ssh://g...@bitbucket.XXXXXXX.com:7999/projectname/reponame.git
> D:\SCM\Apps\Git\2.8.2\bin\git.exe --version # timeout=10
using GIT_ASKPASS to set credentials Unix Bitbucket Groovy
> D:\SCM\Apps\Git\2.8.2\bin\git.exe fetch --tags --progress ssh://g...@bitbucket.XXXXXXX.com:7999/projectname/reponame.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from ssh://g...@bitbucket.XXXXXXXX.com:7999/projectname/reponame.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:862)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1129)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "D:\SCM\Apps\Git\2.8.2\bin\git.exe fetch --tags --progress ssh://g...@bitbucket.XXXXXXX.com:7999/projectname/reponame.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1715)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:207)
at hudson.remoting.UserRequest.perform(UserRequest.java:53)
at hudson.remoting.Request$2.run(Request.java:358)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Any insight or help on the issue would be greatly appreciated, as I have searched and tried each and every suggestion that I am able to find.
Thanks,
John
---------------
John Burrows
Principal Engineer, USA
Common Services - SCM
Vacation Alert :
This email message and any attachments may contain confidential, proprietary or non-public information. The information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this email, please notify the sender immediately and destroy this email. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this email are those of the author personally.
Ramanathan Muthaiah
Have you tried running, manually from command line, 'git clone' using the creds in the problematic nodes ?
Burrows, John
Have you tried running, manually from command line, 'git clone' using the creds in the problematic nodes ?
/Ram