Unable to checkout Jenkinsfile from git for a Pipeline Job: getting unable to fork

[Philippe M.]

Hello,

Using Jenkins 2.138.3 and Git client plugin 2.7.4 I get an error when doing the following:

1) Create job of type Pipeline

2) Select pipeline from SCM and enter

- ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git

- Credentials come from a private key

3) Build job , it fails with:

Started by user unknown or anonymous
Checking out git ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git into /var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script to read Jenkinsfile
Cloning the remote Git repository
Cloning repository ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git
> git init /var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script # timeout=10
Fetching upstream changes from ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git
> git --version # timeout=10
using GIT_SSH to set credentials
 > git fetch --tags --progress ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://g...@bitbucket.foo.com:7999/~userXXX/jmt_poc.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: fatal: cannot exec '/var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script@tmp/ssh7220368723599186680.sh': Permission denied
fatal: unable to fork
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:629)
       at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
       at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
       at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:144)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:292)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
ERROR: Error cloning remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting
Finished: FAILURE

Master is running on redhat 7.2, openjdk 1.8.0_65 64 bit server VM

If I create a project from content of Jenkinsfile directly , which checkouts exactly the same repository , it works fine but node is on redhat 6.4.

I adjusted Xmx, Xms and MaxMetaSpaceSize.

I checked limits of jenkins and I see no issues:

• Max processes is at 4096
• Max open files at 10000
• Max locked memory 65536
  

There is nothing more in /var/log/jenkins/jenkins.log

It happens even after a clean restart of Jenkins and there is no job running.

Thanks in advance for your help.

Regards

Philippe M.

[Mark Waite]

On Friday, December 21, 2018 at 6:20:56 AM UTC-7, Philippe M. wrote:

Hello,

Using Jenkins 2.138.3 and Git client plugin 2.7.4 I get an error when doing the following:

1) Create job of type Pipeline

2) Select pipeline from SCM and enter

- ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git

- Credentials come from a private key

3) Build job , it fails with:

Started by user unknown or anonymous

Checking out git ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git into /var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script to read Jenkinsfile

Cloning the remote Git repository

Cloning repository ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git

> git init /var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script # timeout=10

Fetching upstream changes from ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git

> git --version # timeout=10
using GIT_SSH to set credentials

 > git fetch --tags --progress ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git +refs/heads/*:refs/remotes/origin/*

ERROR: Error cloning remote repo 'origin'

hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://g...@bitbucket.example.com:7999/~userXXX/jmt_poc.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:

stdout:
stderr: fatal: cannot exec '/var/lib/jenkins/workspace/POC_JMT_PIPELINE_FROM_GIT2@script@tmp/ssh7220368723599186680.sh': Permission denied
fatal: unable to fork
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:629)
       at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
       at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
       at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:144)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:292)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
ERROR: Error cloning remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting
Finished: FAILURE

Master is running on redhat 7.2, openjdk 1.8.0_65 64 bit server VM

Check that the temporary directory on the master is not mounted with the 'noexec' option.  Check that the workspace directory on the master is not mounted with the 'noexec' option.

The Jenkins git client plugin needs to communicate credentials to command line git.  Red Hat Linux defaults to an old enough version of command line git that the only way to communicate those credentials is through an executable shell script.  The executable shell script is placed in a temporary directory on the master and run from the master.  The preferred location for that temporary directory is adjacent to the workspace directory.  However, there are cases where the system temporary directory is used instead.

Refer to https://issues.jenkins-ci.org/browse/JENKINS-26993 and https://issues.jenkins-ci.org/browse/JENKINS-44951 for much more detail on the topic.

Mark Waite

[Mark Waite]

See also this excellent article from CloudBees Support

https://support.cloudbees.com/hc/en-us/articles/115003082887-Jenkins-with-Git-Client-needs-permission-on-Temporary-directory

[Philippe M.]

Hello Mark Waite,

THANKS A LOT !

The issue was indeed noexec on the tmp and workspace folders mounting.

Once fixed, everything worked fine.

Regards