[JIRA] (JENKINS-57542) git checkout over ssh fails on some windows agents because .bat file is missing

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke created an issue

Jenkins / JENKINS-57542 git checkout over ssh fails on some windows agents because .bat file is missing Issue Type: Bug Assignee: Mark Waite Components: git-client-plugin Created: 2019-05-18 16:44 Priority: Major Reporter: T Herzke Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke updated an issue

Jenkins / JENKINS-57542 git checkout over ssh fails on some windows agents because .bat file is missing

Change By: T Herzke While setting up a new windows build slave on a rented virtual windows server (tried  server2012 as well as server2016) I keep running into this problem where Jenkins jobs cannot check out from git on the new agent: stderr: error: cannot run c:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7559664601978999582.bat: No such file or directory The required batch file is actually produced and quickly deleted again by the jenkins slave. A script I wrote was able to copy the batch file to a different location before it was deleted. It contains:   {code:java} @echo off "C:\msys64\usr\bin\ssh.exe" -i "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" -l "git" -o "StrictHostKeyChecking=no" %* {code} The referenced .key file also exists for a short time and contains the correct private key for the git repository. It is not clear to me if the file is deleted again before it is invoked, or if it is produced only after it was invoked. I would appreciate pointers how to debug this. Can I introduce wait times after file creation? Since this behaviour is dependent on the windows agent, Windows specific filesystem caching and timing may play a role. I was able to reproduce this with three different virtual windows servers: 2 windows cloud servers rented at ionos.de (2012 and 2016) and a 2016 server rented from a different provider. When executing this in a virtualbox windows 10 machine running on top of ubuntu on my own hardware, this issue does not show up. Steps to reproduce: * Rent a virtual windows server, e.g. from ionos.de * Install msys2 and install git and ssh through msys2 * Install openjdk12 from zip, add to PATH and set JAVA_HOME * Define new build agent NEW-WINDOWS-SERVER in Jenkins * Start Jenkins slave on windows server like this: ** start msys2 bash shell ** inside bash shell, create ssh tunnel to Jenkins host: *** ssh -L 8080:localhost:8080 -L 35900:localhost:35900 -N -f jen...@my.jenkinshost.example.com ** Download slave.jar from Jenkins *** wget [http://localhost:8080/jnlpJars/slave.jar] ** Start Jenkins slave *** java -jar slave.jar -jnlpUrl [http://localhost:8080/computer/NEW-WINDOWS-SERVER/slave-agent.jnlp] * Create test job in Jenkins, which checks out a git repository from a git repository over ssh (somethingj like ssh://g...@example.com/example-repo) * The build steps do not matter, add something simple * Build now attaching the complete console output.

Add Comment

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke updated an issue

Jenkins / JENKINS-57542 git checkout over ssh fails on some windows agents because .bat file is missing

Change By: T Herzke Attachment: console.log

Add Comment

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke updated an issue

Jenkins / JENKINS-57542 git checkout over ssh fails on some windows agents because .bat file is missing Change By: T Herzke

attaching the complete console output. Jenkins version: 2.177. Git client plugin version: 2.7.7 | |

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite updated an issue

Jenkins / JENKINS-57542 git checkout over ssh fails on some windows agents because .bat file is missing

Change By: Mark Waite

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-57542

Re: git checkout over ssh fails on some windows agents because .bat file is missing You might be able to compile a custom version of the git client plugin which includes more diagnostic messages in the batch file with steps like: {noformat} IF EXISTS "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" ECHO Found key file {noformat} in case something is causing those files to not be detected. You might also try google searches for specific cases where programs related to git report "{{No such file or directory}}" on Windows .  That message seems to indicate in at least [one case|https://github.com/fabiospampinato/vscode-open-in-terminal/issues/2] that a directory name was expected to be provided in a different format.  Another hint indicated that message might be coming from {{bash}} rather than from the Windows command processor.  If so, then there may be something incorrect in the git client plugin's detection of which operating system command ({{sh}} or {{bat}}) should be used to launch the process.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-57542

Re: git checkout over ssh fails on some windows agents because .bat file is missing

You might be able to compile a custom version of the git client plugin which includes more diagnostic messages in the batch file with steps like:

IF EXISTS "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" ECHO Found key file

in case something is causing those files to not be detected.

You might also try google searches for specific cases where programs related to git report "No such file or directory". That message seems to indicate in at least one case that a directory name was expected to be provided in a different format. Another hint indicated that message might be coming from bash rather than from the Windows command processor. If so, then there may be something incorrect in the git client plugin's detection of which operating system command (sh or bat) should be used to launch the process.

Add Comment

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke updated an issue

Jenkins / JENKINS-57542

git checkout over ssh fails on some windows agents because .bat file is missing

Change By: T Herzke Priority: Major Minor

Add Comment

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke commented on JENKINS-57542

Re: git checkout over ssh fails on some windows agents because .bat file is missing Thanks for the suggestions. They are reasonable and I would follow but I think I have found a workaround since reporting this issue: I have now observed that if I invoke the "java -jar slave.jar ..."  on the windows server agent not from the msys2 bash shell but from a git-bash shell, then this problem does not occur and the git checkout over ssh succeeds. I have now different problems because the actual Jenkins jobs require the invocation of tools from the msys2 environment which are difficult to invoke from the git-bash environment, but I think I will be able to solve this. The relevant difference for this issue that is introduced by the workaround is probably the different set of environment variables passed on to the java slave process by the different bash shells.

Add Comment

[jenkins2017@gi2.herzkes.de (JIRA)]

T Herzke edited a comment on JENKINS-57542

Re: git checkout over ssh fails on some windows agents because .bat file is missing

Thanks for the suggestions. They are reasonable and I would follow up on them but I think I have found a workaround since reporting this issue:

I have now observed that if I invoke the "java -jar slave.jar ..."  on the windows server agent not from the msys2 bash shell but from a git-bash shell, then this problem does not occur and the git checkout over ssh succeeds. I have now different problems because the actual Jenkins jobs require the invocation of tools from the msys2 environment which are difficult to invoke from the git-bash environment, but I think I will be able to solve this. The relevant difference for this issue that is introduced by the workaround is probably the different set of environment variables passed on to the java slave process by the different bash shells.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite assigned an issue to Unassigned

Jenkins / JENKINS-57542

git checkout over ssh fails on some windows agents because .bat file is missing

Change By: Mark Waite Assignee: Mark Waite

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-57542

Re: git checkout over ssh fails on some windows agents because .bat file is missing The Windows agents where I test and develop the git plugin are run from either: Windows batch file launched from a command processor window (a "JNLP agent") Windows command processor launched through the OpenSSH server provided by Windows (an ssh agent) I prefer the ssh agent because it can be launched from the Jenkins server. I've never attempted to run the agent from a git bash shell on Windows. I have never heard of anyone else using that technique either. Is there a specific reason you chose to start the agent from a git bash shell instead of using the Windows command processor?

Add Comment