Random Authentication Error at git checkout

[Grant Liu]

Hi, 

At random times, I am getting Authentication error with git checkout. This issue would persist through rebuilding, until the job is reassigned to a new slave. The slaves also have git installed via yum:

Installed Packages

git.x86_64                                                                          1.8.3.1-23.el7_8

Attempting to resolve master from remote references...

 > git --version # timeout=10

 > git --version # 'git version 1.8.3.1'

using GIT_SSH to set credentials jenkins ssh creds

 > git ls-remote -h xxxx

Found match: refs/heads/master revision 2ca56ccc896a62bdb199924dccf4ca6eebbcb9cd

Selected Git installation does not exist. Using Default

The recommended git tool is: NONE

using credential jenkins_ssh

 > git rev-parse --is-inside-work-tree # timeout=10

Fetching changes from the remote Git repository

 > git config remote.origin.url xxxx.git # timeout=10

Fetching without tags

Fetching upstream changes from xxxx.git

 > git --version # timeout=10

 > git --version # 'git version 1.8.3.1'

using GIT_SSH to set credentials jenkins ssh creds

 > git fetch --no-tags --progress xxxx* # timeout=10

Checking out Revision 2ca56ccc896a62bdb199924dccf4ca6eebbcb9cd (master)

 > git config core.sparsecheckout # timeout=10

 > git checkout -f 2ca56ccc896a62bdb199924dccf4ca6eebbcb9cd # timeout=10

Commit message: "DEVOPS-4254: update slack channel to #jenkins_maven_alerts (#186)"

[Checks API] No suitable checks publisher found.

[Pipeline] Start of Pipeline

[Pipeline] node

Running on EC2 (Personalcapital) - Jenkins slave centos (i-0fae3510929c96322) in /home/centos/jenkins/workspace/web_pcap-website_master

[Pipeline] {

[Pipeline] stage

[Pipeline] { (Declarative: Checkout SCM)

[Pipeline] checkout

The recommended git tool is: NONE

using credential pcap_github

Fetching changes from the remote Git repository

Fetching without tags

ERROR: Error fetching remote repo 'origin'

hudson.plugins.git.GitException: Failed to fetch from https://github.com/personalcapital/xxxx.git

at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:957)

at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1198)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1258)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)

at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://github.com/personalcapital/pcap-website.git +refs/heads/master:refs/remotes/origin/master" returned status code 128:

stdout: 

stderr: warning: invalid credential line: get

remote: Invalid username or password.

fatal: Authentication failed for 'https://github.com/personalcapital/xxxx.git/'

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2450)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2051)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:573)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:369)

at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

... 4 more

Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (Personalcapital) - Jenkins slave centos (i-0fae3510929c96322)

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:998)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)

at sun.reflect.GeneratedMethodAccessor639.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)

at com.sun.proxy.$Proxy86.execute(Unknown Source)

at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:955)

at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1198)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1258)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)

at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)

at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

... 4 more

I am using the latest git plugin from jenkins. My jenkins version is 2.235.5. Any idea how to resolve this? 

Thanks,

Grant

[Mark Waite]

Usually the message "invalid credentials line" indicates that a git credentials manager has been configured on that agent or in that workspace.  https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage describes credential storage in general.  http://microsoft.github.io/Git-Credential-Manager-for-Windows/Docs/Configuration.html describes more details of the Windows implementation.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1db622bc-39e1-495d-95f9-d1ce7d450000n%40googlegroups.com.

[Grant Liu]

Hi Mark,

This error happens randomly. So probably not a configuration issue, but rather some kind of conflict. Is there a way to narrow this down further when this issue occurs to get to the root cause?

Today when this happened, I did "yum remove git" then "yum install git" then the issue went away when I rebuilt the job. But this still doesn't point me to the root cause.

Any idea is appreciated.

Thanks,

Grant

[Mark Waite]

Are your agents 100% ephemeral?  If not, then a preceding job could have inadvertently activated a git credential manager.

I believe that git credential managers operate at the account level rather than at the workspace level.  If a prior job installed a credential manager or misconfigured git to think there was a credentials manager, then later jobs could be affected by it.

I believe that git credential manager information may be stored in the git configuration for the user.  If so, then you could report the contents of `git config --global --list` at the start of the job and then report `git config --global --list` at the end of the job.  Comparison should show that the global configuration before the job started should be the same as the global configuration at the end of the job.

The git source code suggests that the specific message is coming from either git-credential-libsecret.c, credential.c, or credential-gnome-keyring.c.  If your agent is running on a Linux user desktop, then you might check if the user has enabled the Gnome keyring.  If it is running on Windows on a desktop, you may want to ask the user if they enabled a credential manager.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f7af9d08-9299-40dd-8e35-c6bed2d176d7n%40googlegroups.com.

[Grant Liu]

You are right, Mark. Upon further investigation, we run this script to push tags to git repo:

#!/usr/bin/env bash
set -e
#set -x
git reset --hard $GIT_COMMIT
git tag $1
git config remote.origin.url g...@github.com:$2
git config credential.username ${GIT_USERNAME}
git config credential.helper '!echo password=${GIT_PASSWORD}; echo'
GIT_ASKPASS=true git push origin $1

Without the following unset, any jobs in the same worksapce that run afterwards would consistently fail with authentication issue.

script.sh("git config --unset credential.username")
script.sh("git config --unset credential.helper")

I am not sure why they would need to be unset though, because each run should have the same GIT_USERNAME and GIT_PASSWORD. 

 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/BrtMEPpLxK8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHC0HJV_%2B%2B5Us1--B4BC88nf4d2jhA0wFF2QEoit_6jxQ%40mail.gmail.com.

--

Thanks,

Grant