gitSCM failed after updates

693 views
Skip to first unread message

Giles

unread,
Jun 11, 2019, 3:54:57 PM6/11/19
to Jenkins Users
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url g...@github.com:*/*.git # timeout=10
Fetching upstream changes from g...@github.com:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress g...@github.com:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from g...@github.com:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress g...@github.com:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs.  I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file?  It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

     -----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           
>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24

I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

Mark Waite

unread,
Jun 11, 2019, 4:36:38 PM6/11/19
to Jenkins Users
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

Giles

unread,
Jun 11, 2019, 5:09:22 PM6/11/19
to Jenkins Users
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.


--
Thanks!
Mark Waite

Mark Waite

unread,
Jun 11, 2019, 5:12:11 PM6/11/19
to Jenkins Users
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3c977d6b-a180-402b-a5f7-bb4825ba2217%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

Giles

unread,
Jun 11, 2019, 6:53:03 PM6/11/19
to Jenkins Users
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:114)
	at jenkins.plugins.git.GitSCMFileSystem$3.invoke(GitSCMFileSystem.java:193)
	at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:81)
	at jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:189)
	at jenkins.plugins.git.GitSCMFileSystem.<init>(GitSCMFileSystem.java:114)
	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:353)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:198)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:293)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

This at least is a known error: it appears to happen when there's a newer version of git, or a misalignment of the git version and the git plugin.  I had initially installed the very new version of Git for Windows 2.22.0.  I downgraded that to 2.17.1.2, and eventually went to 2.21.0 - all resulting in the same failure.  Any thoughts?  It certainly doesn't seem to be the Git version, and the one working pipeline muddies the water further.

Mark Waite

unread,
Jun 11, 2019, 7:02:56 PM6/11/19
to Jenkins Users
On Tue, Jun 11, 2019 at 4:53 PM Giles <gile...@gmail.com> wrote:
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
That may indicate that something has installed a newer version of JGit than JGit 4.5 which is bundled with git client plugin 2.7.x.  JGit 4.5 and prior versions provided the getRef(String) API.  Newer JGit versions (like 4.9, 5.1, 5.2, and 5.3) have deprecated or stopped delivering the getRef(String) API.

Can you confirm that the git client plugin version you're running is 2.7.x and not one of the pre-release 3.0 versions?  If it is not, then you likely need to install the most recent 2.7 version.

Can you confirm that the git plugin version you're running is 3.9.x and not one of the pre-release 4.0 versions?  I expect that is the case since pre-release versions of git plugin 4.0.0 generally do not refer to getRef(String).

That message is unrelated to the version of command line git installed on the master.  That message is related to an internal mismatch between the JGit API that is available in the running Jenkins process and the JGit API that the git plugin requires.  The git plugin is expecting to find JGit 4.5 in the Jenkins process.  It does not seem to be there in this case.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

Giles

unread,
Jun 12, 2019, 11:52:43 AM6/12/19
to Jenkins Users
Hi Mark.

Correct again:

- Git plugin 3.10.0, now downgraded to 3.9.1
- Git client plugin 3.0.0-rc, now downgraded to 2.7.6

Deploys do now appear to all be working (this time I waited until we'd done a fair bit of testing before reporting ...).

I upgraded the above plugins to 4.0.0-rc and 3.0.0-rc back in February because there were security advisories: I don't usually upgrade major versions immediately (especially not RC), instead waiting for them to have time to stabilize.

These downgrades means that I'm now getting a fair bit of messaging from Jenkins about things I should upgrade or change:

    Dependency errors:

    Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.

    Git Parameter Plug-In version 0.9.10
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.
    GitHub Branch Source Plugin version 2.5.3
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.

    Warnings have been published for the following currently installed components.
    Git plugin 3.9.1
    CSRF vulnerability

Given that everything is working, I take it I should just ignore these for now?  Or possibly remove "Git Parameter Plug-In" and "GitHub Branch Source Plugin" as we apparently don't use them at all ...

Once again, thank you very much for your help.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.


--
Thanks!
Mark Waite

Mark Waite

unread,
Jun 12, 2019, 12:05:49 PM6/12/19
to Jenkins Users
On Wed, Jun 12, 2019 at 9:52 AM Giles <gile...@gmail.com> wrote:
Hi Mark.

Correct again:

- Git plugin 3.10.0, now downgraded to 3.9.1

Git plugin 3.10.0 is a good choice.  Upgrade to it.
 
- Git client plugin 3.0.0-rc, now downgraded to 2.7.6


Git client plugin 3.0.0-rc is a bad choice.  Don't upgrade (yet) to any git client plugin newer than 2.7.x
 
Deploys do now appear to all be working (this time I waited until we'd done a fair bit of testing before reporting ...).

I upgraded the above plugins to 4.0.0-rc and 3.0.0-rc back in February because there were security advisories: I don't usually upgrade major versions immediately (especially not RC), instead waiting for them to have time to stabilize.


I think that you may have misread the security advisory.  No security advisory has been issued that would require installation of a release candidate.  Those two releases (git plugin 4.0.0-rc and git client plugin 3.0.0-rc) have been removed from the update center.  They have serious compatibility problems which have been resolved in later pre-releases of the plugins.
 
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2769d445-024a-4ffa-a593-b37958466581%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

Henry Xu

unread,
Sep 29, 2019, 7:12:19 PM9/29/19
to Jenkins Users
I guess I have same issue. And it's wired as it works will on a machine I provisioned a year ago. Can you kindly provide more detail about how to using Global Tool Configuration to fix this problem? Thanks in advance.


Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress xxx +refs/heads/*:refs/remotes/Fiji/* --prune" returned status code 128:
stdout: 
stderr: error: cannot run c:\jws2\workspace\xxx\ssh7984855687134194056.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to SDET-XMN160
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:957)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy109.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:748)

Mark Waite

unread,
Sep 29, 2019, 8:01:52 PM9/29/19
to Jenkins Users
On Sun, Sep 29, 2019 at 5:12 PM Henry Xu  wrote:
I guess I have same issue. And it's wired as it works will on a machine I provisioned a year ago. Can you kindly provide more detail about how to using Global Tool Configuration to fix this problem? Thanks in advance.


The Jenkins global tool configuration won't fix the problem, at least as far as I know.

If you have the same problem, then you are running Windows, have installed cygwin, and have configured Jenkins to use the git program that is available with cygwin.  The solution is to install git for Windows and assure that it is the program which is used by Jenkins, instead of the git program that is available with cygwin.  The Jenkins git plugin does not test with the command line git version that is included with cygwin.  It tests with git for Windows.

Another alternative (for many use cases) is to enable JGit as a git implementation in your Jenkins server, then configure the job to use JGit instead of command line git.  JGit is able to run most use cases, but not all use cases that are supported by command line git.  Refer to https://plugins.jenkins.io/git-client for instructions to enable JGit in your installation.
 
-- 

--
Thanks!
Mark Waite

Henry Xu

unread,
Sep 29, 2019, 8:47:22 PM9/29/19
to Jenkins Users
Hi, thanks for your suggestion. I downgrade the git from 2.21.0 to 2.17.1 (the version I used on the old machine) and anything work fine. I am wondering why the git version may cause issue of this kind.

Mark Waite

unread,
Sep 29, 2019, 9:20:51 PM9/29/19
to Jenkins Users
On Sun, Sep 29, 2019 at 6:47 PM 'Henry Xu' via Jenkins Users <jenkins...@googlegroups.com> wrote:
Hi, thanks for your suggestion. I downgrade the git from 2.21.0 to 2.17.1 (the version I used on the old machine) and anything work fine. I am wondering why the git version may cause issue of this kind.


The cygwin environment may have changed how it launches subprocesses.  The cygwin environment may have changed how it is invoked from Windows.  Command line git may have changed something which causes the damage.

The Jenkins git plugin is not tested with cygwin.  This mail message thread has strong indications (you and one other user) that the most recent command line git that is available with cygwin does not work with the Jenkins git plugin.  I'm glad that it works for you with git 2.17.1, but I don't test with cygwin and won't test with cygwin in the future.
 
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6440ee7e-354d-4362-b6a2-77c59deb714f%40googlegroups.com.


--
Thanks!
Mark Waite
Reply all
Reply to author
Forward
0 new messages