GIT_SSH not working on one of my windows slaves

[red 888]

Super frustrating because this is working on one of my windows slaves, but not this one- and I cant find any config differences.

On the working slave I see this:

[Pipeline] checkout
Cloning the remote Git repository
Cloning repository g...@bitbucket.org:myteam/myapp.git
 > git init C:\Jenkins\workspace\test-slave123456 # timeout=10
Fetching upstream changes from g...@bitbucket.org:myteam/myapp.git
 > git --version # timeout=10
using GIT_SSH to set credentials mygitcreds
 > git fetch --tags --progress g...@bitbucket.org:myteam/myapp.git +refs/heads/*:refs/remotes/origin/* # timeout=45
 > git config remote.origin.url g...@bitbucket.org:myteam/myapp.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url g...@bitbucket.org:myteam/myapp.git # timeout=10
Fetching upstream changes from g...@bitbucket.org:myteam/myapp.git
using GIT_SSH to set credentials mygitcreds
 > git fetch --tags --progress g...@bitbucket.org:myteam/myapp.git +refs/heads/*:refs/remotes/origin/* # timeout=45
 > git rev-parse "origin/test-slave^{commit}" # timeout=10
Checking out Revision 30f11ef09ab13f73fb9a6b75983e1bf32437f51d (origin/test-slave)
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 30f11ef09ab13f73fb9a6b75983e1bf32437f51d # timeout=45
 > git config --get remote.origin.url # timeout=10
using GIT_SSH to set credentials mygitcreds
 > git lfs pull origin # timeout=45
Commit message: "test slave"
 > git rev-list --no-walk 30f11ef09ab13f73fb9a6b75983e1bf32437f51d # timeout=10

But on the failing slave:

[Pipeline] checkout
Cloning the remote Git repository
Cloning repository g...@bitbucket.org:myteam/myapp.git
 > git init C:\Jenkins\workspace\test-slave123456 # timeout=10
Fetching upstream changes from g...@bitbucket.org:myteam/myapp.git
 > git --version # timeout=10
using GIT_SSH to set credentials mygitcreds
 > git fetch --tags --progress g...@bitbucket.org:myteam/myapp.git +refs/heads/*:refs/remotes/origin/* # timeout=45
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress g...@bitbucket.org:myteam/myapp.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: gituserfrom...@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Its the same pipeline job, same repo, same creds, and the slave should be configured the same but when I change the agent to point to the other slave it cant clone.

On the working slave all i had to do was install git for windows (turn off windows cred store), install java, and then run the jnlp jar.

Tried to do the same thing on the non working slave so I dont know why that one could be failing.

[Slide]

Can you try dumping the environment variables on each node and see if there are any differences?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3afe4362-20f7-40c9-91ac-ac6573d0bd16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Waite]

It could be a "happy accident" that it is working on the first agent.  

When using a command prompt on the first agent, does `git clone` allow you to clone without prompting for remote username or password?  

When using a command prompt on the second agent, does it behave the same as the first agent?

The login context (~/.ssh/ directory contents, environment variables, etc.) affect agents which use that login context.  If the agent is already configured to silently authenticate to bitbucket, then incorrect credentials in the Jenkins environment are ignored and the repository is still retrieved.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPiUgVft0hmukqoCu-JQckBpj%3D72CwCfQp_tTwAz5kAvT_-qRg%40mail.gmail.com.

[red 888]

Dumped env vars on both nodes right before checkout, i dont see any real differences. Anything env vars specific to GIT_SSH? Also both slaves are running the same windows build too.

[red 888]

I can confirm the git ssh key works and has always worked so the creds themselves should not be an issue.

git clone fails on both slaves (when run interactively as a logged in user). The windows task that runs the jnlp executes as the SYSTEM account.

I also made sure to do git config --system --unset credential.helper. Any local config that would break this?

The git jenkins plugin should be totally handling all the git cred setup stuff, but maybe someone modified a local config on the broken slave? the git global config looks identical on both of them

[Mark Waite]

Have you confirmed that the temporary directory on the failing machine does not include any space characters in its path?  There is a known problem on Windows that the credential passing technique required by command line git does not allow a space character in the temporary directory path.

I assume from the log that the workspace does not include a space character in its path.  If it does, that could invoke the same problem with command line git authentication on Windows not really liking temporary paths which contain a space character.

Mark Waite 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7127c010-253a-48c0-bf83-af218988a391%40googlegroups.com.

[red 888]

hmm what is the "temporary directory" in this context? The jenkins workspace is configured the same on both slaves (just a folder under c:) so no spaces there

Another thing i looked at was c:\Windows\System32\config\systemprofile\.ssh\ and on both slaves that folder just has a know_hosts file and its contents is the same on both slaves too.

[red 888]

also, if this is helpful this is the global git config on both slaves:

PS C:\Users\Administrator> git config --list

core.symlinks=false

core.autocrlf=true

core.fscache=true

color.diff=auto

color.status=auto

color.branch=auto

color.interactive=true

help.format=html

rebase.autosquash=true

http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt

http.sslbackend=openssl

diff.astextplain.textconv=astextplain

filter.lfs.clean=git-lfs clean -- %f

filter.lfs.smudge=git-lfs smudge -- %f

filter.lfs.process=git-lfs filter-process

filter.lfs.required=true

[red 888]

Jenkins says "using GIT_SSH to set credential...." What is it actually putting in GIT_SSH?

[Mark Waite]

The name of the temporary directory used for the credentials is based on either the workspace folder path (with @tmp appended) or the environment variables %TEMP% or %TMP%.  If none of those contain a space character, then that is not the problem.

The plugin usually wirtes a message when it detects a space character in a temporary directory path.  Since you didn't report such a message, I assume that is not the issue in this case.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1ab89567-36f1-4dc7-84a7-5dd90da4c2e8%40googlegroups.com.

[red 888]

How can I get jenkins to give me more verbose output than just "using GIT_SSH to set credential...."

I set GIT_CURL_VERBOSE=1 and GIT_TRACE=1 on the node but the output from jenkins doesnt change at all

I'd like to know what its setting the environment var to and maybe get some trace level info too. Is there a git 

[Mark Waite]

You may be able to use GIT_SSH_COMMAND="ssh -vvv" as a job parameter or an agent environment variable.  Refer to https://support.cloudbees.com/hc/en-us/articles/115000618372-How-to-trace-git-connections- for more information

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1db8d13d-ec0f-4292-b85f-d220fec73ebc%40googlegroups.com.

[red 888]

Thanks for all the help/suggestions! Will ssh need to be in the system path for GIT_SSH_COMMAND="ssh -vvv" to work? because ssh isnt in the path

[Mark Waite]

As far as I know, ssh will need to be in the PATH.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d92620df-9f3c-46b0-b58e-1f717b70896c%40googlegroups.com.

[red 888]

I already set trace and curl (for some reason setting the env on the node didnt work I had to actually set envs for the system on the box directly)

It doesnt give me much extra info though:

ERROR: Error cloning remote repo 'origin'

hudson.plugins.git.GitException: Command "git fetch --tags --progress g...@bitbucket.org:myproject/myrepo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: 09:00:41.660477 git.c:344               trace: built-in: git fetch --tags --progress git@bitbucket.org:myproject/myrepo.git '+refs/heads/*:refs/remotes/origin/*'
09:00:41.663480 run-command.c:640       trace: run_command: unset GIT_PREFIX; 'C:\Jenkins\workspace\mybranchabc123@tmp\ssh2197923650539343925.bat' git@bitbucket.org 'git-upload-pack '\''myproject/myrepo.git'\'''
mygi...@bitbucket.org: Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I could try "ssh -vvv" to get more output too (ssh isnt in the path of the working slave either).