[JIRA] (JENKINS-61408) Git executable is randomly set to empty string

15 views
Skip to first unread message

stephane@odul.com (JIRA)

unread,
Mar 9, 2020, 6:38:03 PM3/9/20
to jenkinsc...@googlegroups.com
Stephane Odul created an issue
 
Jenkins / Bug JENKINS-61408
Git executable is randomly set to empty string
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-plugin
Created: 2020-03-09 22:37
Environment: Jenkins 2.204.5
Git Plugin 4.1.1
Priority: Major Major
Reporter: Stephane Odul

Once in a while, at least several times a week, the name of the Git executable ends up being unset and the Git plugin calls `  init` (whitespace) instead of `git init`.

We use a custom installer for git and it works and install just fine and even with jobs with multiple parallel stages, only one stage hits the bug, randomly. So configuration on our side is just fine but we suspect that there is a bug in the codebase where the variable holding the git executable name can get invalidated, at the stage level.

 

 
                                                            

[2020-03-04T19:28:38.877Z] Failed to get git executable[2020-03-04T19:28:38.877Z] Failed to get git executable[2020-03-04T19:28:42.682Z] using credential xxxxx-github-rw[2020-03-04T19:28:42.691Z] Cloning the remote Git repository[2020-03-04T19:28:42.691Z] Using shallow clone with depth 500[2020-03-04T19:28:42.691Z] Avoid fetching tags[2020-03-04T19:28:42.691Z] Honoring refspec on initial clone[2020-03-04T19:28:42.933Z] ERROR: Error cloning remote repo 'origin'[2020-03-04T19:28:42.933Z] hudson.plugins.git.GitException: Could not init /home/jenkins/agent/workspace/GitHub_xxxx_master[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:989)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:747)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)[2020-03-04T19:28:42.933Z] at hudson.remoting.UserRequest.perform(UserRequest.java:211)[2020-03-04T19:28:42.934Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54)[2020-03-04T19:28:42.934Z] at hudson.remoting.Request$2.run(Request.java:369)[2020-03-04T19:28:42.934Z] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)[2020-03-04T19:28:42.934Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[2020-03-04T19:28:42.934Z] at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)[2020-03-04T19:28:42.934Z] at java.lang.Thread.run(Thread.java:748)[2020-03-04T19:28:42.934Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-xx-xx-xx-xx.us-west-2.compute.internal/xx.xx.xx.xx:40980[2020-03-04T19:28:42.934Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)[2020-03-04T19:28:42.934Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)[2020-03-04T19:28:42.934Z] at hudson.remoting.Channel.call(Channel.java:955)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)[2020-03-04T19:28:42.934Z] at sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)[2020-03-04T19:28:42.934Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[2020-03-04T19:28:42.934Z] at java.lang.reflect.Method.invoke(Method.java:498)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)[2020-03-04T19:28:42.934Z] at com.sun.proxy.$Proxy163.execute(Unknown Source)[2020-03-04T19:28:42.934Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)[2020-03-04T19:28:42.934Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)[2020-03-04T19:28:42.934Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[2020-03-04T19:28:42.934Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[2020-03-04T19:28:42.934Z] ... 1 more[2020-03-04T19:28:42.934Z] Caused by: hudson.plugins.git.GitException: Error performing git command:  init /home/jenkins/agent/workspace/GitHub_xxxx_master[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2436)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2359)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2355)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1915)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:987)[2020-03-04T19:28:42.934Z] ... 12 more[2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: Cannot run program "" (in directory "/home/jenkins/agent/workspace/GitHub_xxxx_master"): error=2, No such file or directory[2020-03-04T19:28:42.934Z] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)[2020-03-04T19:28:42.934Z] at hudson.Proc$LocalProc.<init>(Proc.java:250)[2020-03-04T19:28:42.934Z] at hudson.Proc$LocalProc.<init>(Proc.java:219)[2020-03-04T19:28:42.934Z] at hudson.Launcher$LocalLauncher.launch(Launcher.java:937)[2020-03-04T19:28:42.934Z] at hudson.Launcher$ProcStarter.start(Launcher.java:455)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2422)[2020-03-04T19:28:42.934Z] ... 16 more[2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: error=2, No such file or directory[2020-03-04T19:28:42.934Z] at java.lang.UNIXProcess.forkAndExec(Native Method)[2020-03-04T19:28:42.934Z] at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)[2020-03-04T19:28:42.934Z] at java.lang.ProcessImpl.start(ProcessImpl.java:134)[2020-03-04T19:28:42.934Z] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)[2020-03-04T19:28:42.934Z] ... 21 moreError cloning remote repo 'origin'{noformat}

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

stephane@odul.com (JIRA)

unread,
Mar 9, 2020, 6:39:03 PM3/9/20
to jenkinsc...@googlegroups.com
Stephane Odul updated an issue
Change By: Stephane Odul
Once in a while, at least several times a week, the name of the Git executable ends up being unset and the Git plugin calls `  init` (whitespace) instead of `git init`.

We use a custom installer for git and it works and install just fine and even with jobs with multiple parallel stages, only one stage hits the bug, randomly. So configuration on our side is just fine but we suspect that there is a bug in the codebase where the variable holding the git executable name can get invalidated, at the stage level.

 

{noformat}
[2020-03-04T19:28:38.877Z] Failed to get git executable[2020-03-04T19:28:38.877Z] Failed to get git executable[2020-03-04T19:28:42.682Z] using credential xxxxx-github-rw[2020-03-04T19:28:42.691Z] Cloning the remote Git repository[2020-03-04T19:28:42.691Z] Using shallow clone with depth 500[2020-03-04T19:28:42.691Z] Avoid fetching tags[2020-03-04T19:28:42.691Z] Honoring refspec on initial clone[2020-03-04T19:28:42.933Z] ERROR: Error cloning remote repo 'origin'[2020-03-04T19:28:42.933Z] hudson.plugins.git.GitException: Could not init /home/jenkins/agent/workspace/GitHub_xxxx_master[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:989)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:747)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)[2020-03-04T19:28:42.933Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)[2020-03-04T19:28:42.933Z] at hudson.remoting.UserRequest.perform(UserRequest.java:211)[2020-03-04T19:28:42.934Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54)[2020-03-04T19:28:42.934Z] at hudson.remoting.Request$2.run(Request.java:369)[2020-03-04T19:28:42.934Z] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)[2020-03-04T19:28:42.934Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[2020-03-04T19:28:42.934Z] at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)[2020-03-04T19:28:42.934Z] at java.lang.Thread.run(Thread.java:748)[2020-03-04T19:28:42.934Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-xx-xx-xx-xx.us-west-2.compute.internal/xx.xx.xx.xx:40980[2020-03-04T19:28:42.934Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)[2020-03-04T19:28:42.934Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)[2020-03-04T19:28:42.934Z] at hudson.remoting.Channel.call(Channel.java:955)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)[2020-03-04T19:28:42.934Z] at sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)[2020-03-04T19:28:42.934Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[2020-03-04T19:28:42.934Z] at java.lang.reflect.Method.invoke(Method.java:498)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)[2020-03-04T19:28:42.934Z] at com.sun.proxy.$Proxy163.execute(Unknown Source)[2020-03-04T19:28:42.934Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)[2020-03-04T19:28:42.934Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)[2020-03-04T19:28:42.934Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[2020-03-04T19:28:42.934Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[2020-03-04T19:28:42.934Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[2020-03-04T19:28:42.934Z] ... 1 more[2020-03-04T19:28:42.934Z] Caused by: hudson.plugins.git.GitException: Error performing git command:  init /home/jenkins/agent/workspace/GitHub_xxxx_master[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2436)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2359)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2355)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1915)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:987)[2020-03-04T19:28:42.934Z] ... 12 more[2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: Cannot run program "" (in directory "/home/jenkins/agent/workspace/GitHub_xxxx_master"): error=2, No such file or directory[2020-03-04T19:28:42.934Z] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)[2020-03-04T19:28:42.934Z] at hudson.Proc$LocalProc.<init>(Proc.java:250)[2020-03-04T19:28:42.934Z] at hudson.Proc$LocalProc.<init>(Proc.java:219)[2020-03-04T19:28:42.934Z] at hudson.Launcher$LocalLauncher.launch(Launcher.java:937)[2020-03-04T19:28:42.934Z] at hudson.Launcher$ProcStarter.start(Launcher.java:455)[2020-03-04T19:28:42.934Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2422)[2020-03-04T19:28:42.934Z] ... 16 more[2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: error=2, No such file or directory[2020-03-04T19:28:42.934Z] at java.lang.UNIXProcess.forkAndExec(Native Method)[2020-03-04T19:28:42.934Z] at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)[2020-03-04T19:28:42.934Z] at java.lang.ProcessImpl.start(ProcessImpl.java:134)[2020-03-04T19:28:42.934Z] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)[2020-03-04T19:28:42.934Z] ... 21 moreError cloning remote repo 'origin' \ {noformat}
 

stephane@odul.com (JIRA)

unread,
Mar 9, 2020, 6:40:04 PM3/9/20
to jenkinsc...@googlegroups.com
[2020-03-04T19:28:42.934Z]  ... 21 moreError more
Error
cloning remote repo 'origin'
{noformat}
 

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 9, 2020, 6:50:03 PM3/9/20
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 9, 2020, 6:51:14 PM3/9/20
to jenkinsc...@googlegroups.com

stephane@odul.com (JIRA)

unread,
Mar 11, 2020, 4:45:02 AM3/11/20
to jenkinsc...@googlegroups.com

Under Global Tool Configuration replace the Default Git installation with a custom one that uses a shell script.

In our case we have 2 installers with filters:

  • master: the script is just `true` and Tool Home is `/usr/bin/git`
  • !master: the script is installing a simple shell wrapper to perform retry logic when the git commands fails due to transient network or server side glitches. The Tool Home is the path to the wrapper.

We use the GitHub Organization Plugin to scan our repos and trigger our CI pipelines, and we use the Kubernetes plugin to launch ephemeral workers for each stage.

The kubernetes plugin is configured with a default 'jnlp' pod definition which is nothing more that the default pod definition that comes with the kubernetes plugin.

The Jenkinsfile has parallel stages, it does not matter what the stages do.

Very randomly, and infrequently, with no apparent cause one of the stages will get an empty git executable name with the stack trace provided in the description.

 

Sample Jenkinsfile:

 
pipeline {
    agent none
    options {
        ansiColor('xterm')
        timestamps ()
        buildDiscarder(logRotator(daysToKeepStr: '30'))
        timeout(time: 5, unit: 'MINUTES')
    }
    stages {
        stage('Tests') {
            parallel {
                stage('sleep 1') {
                    agent { kubernetes { label 'jnlp' } }
                    steps { sh 'sleep 1' }
                }
                stage('sleep 2') {
                    agent { kubernetes { label 'jnlp' } }
                    steps { sh 'sleep 2' }
                }
                stage('sleep 3') {
                    agent { kubernetes { label 'jnlp' } }
                    steps { sh 'sleep 3' }
                }
                stage('sleep 4') {
                    agent { kubernetes { label 'jnlp' } }
                    steps { sh 'sleep 4' }
                }
                stage('sleep 5') {
                    agent { kubernetes { label 'jnlp' } }
                    steps { sh 'sleep 5' }
                }
            }
        }
    }
    post { always { chuckNorris() } }
}

stephane@odul.com (JIRA)

unread,
Apr 9, 2020, 7:31:03 PM4/9/20
to jenkinsc...@googlegroups.com

We have some additional data about the issue. The empty string is because our git installer did not have a fallback value and so the git command ended up being null.

The root cause is `Failed to get git executable` which is probably returned by the installer but we have no information as to why it is failing.

In our case the installer is a just a shell script that writes an other shell script which becomes a wrapper for the git commands so that we can have smarter retry logic and better tuned ref specs than possible with the plugin.

Reply all
Reply to author
Forward
0 new messages