[JIRA] (JENKINS-52857) Cannot find git-lfs when it is not installed in /usr/bin/

966 views
Skip to first unread message

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 8:59:02 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret created an issue
 
Jenkins / Bug JENKINS-52857
Cannot find git-lfs when it is not installed in /usr/bin/
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-plugin
Created: 2018-08-02 12:58
Environment: OSX
Priority: Minor Minor
Reporter: Serghei Moret

We have an issue on our OSX nodes, where we cannot checkout a repo with git-lfs.

We suppose that it is related to that `checkout scm` listens `/usr/bin/` and it is not possible to install anything there because of Apple's system integrity. Even passing a PATH=$PATH:/usr/local/bin doesn't take any effect. 

Tested this with a ssh nods and `git checkout` works (as we have user credentials and all the simlinks), but as all of our nodes use JNLP we would like to have a possibility to define path to git-lfs like we can do for git.

Stack trace:
hudson.plugins.git.GitException: Command "/usr/local/bin/git checkout -f 304f1c283e1fa22e47ea9d7be1e2582a93021819" returned status code 128:
stdout:
stderr: git-lfs filter-process: git-lfs: command not found
fatal: The remote end hung up unexpectedly

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$800(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2301)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 172.31.128.169/172.31.128.169:49241
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:389)
at hudson.remoting.Channel.call(Channel.java:955)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor638.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy117.execute(Unknown Source)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1236)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: hudson.plugins.git.GitException: Could not checkout 304f1c283e1fa22e47ea9d7be1e2582a93021819
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2325)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:210)
at hudson.remoting.UserRequest.perform(UserRequest.java:53)
at hudson.remoting.Request$2.run(Request.java:358)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Thread.java:844)
Finished: FAILURE
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:00:02 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret updated an issue
Change By: Serghei Moret

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:00:06 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret updated an issue
We have an issue on our OSX nodes, where we cannot checkout a repo with git-lfs.

We suppose that it is related to that `checkout scm` listens `/usr/bin/` and it is not possible to install anything there because of Apple's system integrity. Even passing a PATH=$PATH:/usr/local/bin doesn't take any effect. 

Tested this with a ssh nods and `git checkout` works (as we have user credentials and all the simlinks), but as all of our nodes use JNLP we would like to have a possibility to define path to git-lfs like we can do for git.

Stack trace:
{code:java}

{code}
 

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:12:01 AM8/2/18
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:14:01 AM8/2/18
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-52857
 
Re: Cannot find git-lfs when it is not installed in /usr/bin/

Since I don't have access to a Mac, I can't test the case you're describing. Can you confirm that you installed git LFS on the mac following the git-lfs instructions?

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:21:01 AM8/2/18
to jenkinsc...@googlegroups.com

sure, git-lfs is installed. I can do: 

sh 'git checkout master'

but not

checkout scm

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:22:07 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret assigned an issue to Mark Waite
 
Change By: Serghei Moret
Assignee: Mark Waite

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:22:08 AM8/2/18
to jenkinsc...@googlegroups.com
 
Re: Cannot find git-lfs when it is not installed in /usr/bin/

Also, I don't think that mac is needed to reproduce. What is needed is git-lfs installed in /usr/local/bin and JNLP connection to the node

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:23:02 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret edited a comment on Bug JENKINS-52857
Also, I don't think that mac is needed to reproduce. What is needed is git-lfs installed in /usr/local/bin and JNLP connection to the node . 

 

Mac problem is only that you cannot install anything to /usr/bin even with rooted user.

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:27:03 AM8/2/18
to jenkinsc...@googlegroups.com

What we miss is possibility to define location of git-lfs binary, so `checkout scm` command could work

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:45:01 AM8/2/18
to jenkinsc...@googlegroups.com

I wasn't able to duplicate the problem you're reporting, even on a JNLP based agent with git-lfs (and git) in the /usr/local/bin directory. Steps I took:

  1. Confirm that git and git-lfs are installed in /usr/local/bin in the 'git' jail of my FreeNAS server
  2. Configure a JNLP agent to run from a Jenkins 2.121.2 master based on the docker-lfs image
  3. Declare the location of the Default git tool on that agent as /usr/local/bin/git
  4. Run Freestyle and Pipeline jobs on that agent which confirm it is able to use LFS repositories

See the attached picture for the configuration details

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:45:02 AM8/2/18
to jenkinsc...@googlegroups.com
Mark Waite updated an issue
 
Change By: Mark Waite
Attachment: FreeNAS-JNLP-git-lfs-config.PNG

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:46:02 AM8/2/18
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-52857
I wasn't able to duplicate the problem you're reporting, even on a JNLP based agent with git-lfs (and git) in the /usr/local/bin directory.  Steps I took:

# Confirm that git and git-lfs are installed in /usr/local/bin in the 'git' jail of my FreeNAS server
# Configure a JNLP agent to run from a Jenkins 2.121.2 master based on the [docker-lfs image|https://github.com/MarkEWaite/docker-lfs/tree/lts-with-plugins]
# Declare the location of the Default git tool on that agent as /usr/local/bin/git
# Run Freestyle and Pipeline jobs on that agent which confirm it is able to use LFS repositories


See the attached picture for the configuration details


!FreeNAS-JNLP-git-lfs-config.PNG|thumbnail!

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 2, 2018, 9:48:01 AM8/2/18
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

joesatriani323@gmail.com (JIRA)

unread,
Aug 2, 2018, 10:09:02 AM8/2/18
to jenkinsc...@googlegroups.com
Serghei Moret commented on Bug JENKINS-52857
 
Re: Cannot find git-lfs when it is not installed in /usr/bin/

Mark Waite thanks a lot for looking into it! I have the same settings but it fails, so maybe it is really related to OSX

tom@futurevisual.com (JIRA)

unread,
Oct 18, 2018, 11:49:05 AM10/18/18
to jenkinsc...@googlegroups.com

I am having the same issue on OSX, with my git and git lfs installed through brew. This has happened on two iMacs for me (one fresh install, one existing/in use)
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

andrewlopinto@gmail.com (JIRA)

unread,
Oct 18, 2018, 3:51:02 PM10/18/18
to jenkinsc...@googlegroups.com

I have the same issue as Serghei, OSX, and git-lfs is installed at --system.  Similarly, I cannot move the git-lfs binary to /usr/bin because of Apple System Integrity.   Hoping for some help on this one, we do a lot of xCode builds that we want to move to an OSX Slave.  

 

Anything I can provide to help here?

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 18, 2018, 4:15:02 PM10/18/18
to jenkinsc...@googlegroups.com

Andrew LoPinto I think the best way you could help would be to debug on a macOS machine to see why command line git launched from the Jenkins git client plugin can't find the git LFS binary.

I don't have a macOS machine. It works on FreeBSD as I expected. FreeBSD is the closest approximation I have to macOS.

For example, you might investigate (with Java debugger or otherwise):

  • PATH setting of command line git when run on macOS
  • Homebrew special cases (if any) for git on macOS
  • Workarounds like installing command line git outside homebrew (in case that allows git-lfs to be found)

tom@futurevisual.com (JIRA)

unread,
Oct 19, 2018, 5:09:02 AM10/19/18
to jenkinsc...@googlegroups.com

I believe it's to do with the PATH. The way I got it to work was by disabling the Apple System Integrity and symbolic linking the git-lfs from /usr/local/bin to /usr/bin. When the plugin uses git, it seems to use a PATH that only has /usr/bin in it and not /usr/local/bin, because I even set the PATH to have /usr/local/bin first before using the command 'checkout scm' but it still doesn't find git-lfs. Also git-lfs works if I manually call 'git checkout ...' so it's something specific to the git scm plugin for jenkins.

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 19, 2018, 8:30:01 AM10/19/18
to jenkinsc...@googlegroups.com

Does it work with Apple System Integrity enabled if you in the environment of the macOS agent you add the environment variable:

PATH+GIT_LFS=/usr/local/bin

I use that technique on my agents to set the PATH environment variable for some jobs that aren't equipped to use the Jenkins global tool configuration.

In my configuration, it looks like this:

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 19, 2018, 8:30:01 AM10/19/18
to jenkinsc...@googlegroups.com
Mark Waite updated an issue
Change By: Mark Waite
Attachment: node-envs.png

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 19, 2018, 8:31:02 AM10/19/18
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-52857
Does it work with Apple System Integrity enabled if you in the environment of the macOS agent you add adds the environment variable:

{noformat}
PATH+GIT_LFS=/usr/local/bin
{noformat}


I use that technique on my agents to set the PATH environment variable for some jobs that aren't equipped to use the Jenkins global tool configuration.

In my configuration, it looks like this:

!node-envs.png|thumbnail!

btk3000@gmail.com (JIRA)

unread,
Oct 25, 2018, 4:47:03 AM10/25/18
to jenkinsc...@googlegroups.com

Hi,

I have exactly same issue with Linux node.

My Linux image is predefined without git. I use git and git lfs from custom location. git work, but git lfs doesn`t.

I`ve tried all recommendations about reconfigured PATH from this discussion. It doesn`t work. Git plugin still looks git lfs at /usr/bin. 

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 25, 2018, 7:29:03 AM10/25/18
to jenkinsc...@googlegroups.com

Ilya Evseenkov can you provide a step by step description that will allow someone else to duplicate the problem you're seeing? I can't duplicate this problem in my attempts with FreeBSD. All my Linux installations include git and they all find git lfs whether the git lfs package is provided by the operating system (Debian Testing), provided by a separate operating system package (Debian 8), or separately installed (CentOS 7).

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 25, 2018, 7:30:02 AM10/25/18
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-52857
[~btk3000] can you provide a step by step description that will allow someone else to duplicate the problem you're seeing?   

   I can't duplicate this problem in my attempts with FreeBSD.   I don't have access to macOS.

  All my Linux installations include git and they all find git lfs whether the git lfs package is provided by the operating system (Debian Testing), provided by a separate operating system package (Debian 8), or separately installed (CentOS 7).

btk3000@gmail.com (JIRA)

unread,
Oct 29, 2018, 8:48:01 AM10/29/18
to jenkinsc...@googlegroups.com
Ilya Evseenkov assigned an issue to Ilya Evseenkov
 
Change By: Ilya Evseenkov
Assignee: Ilya Evseenkov

btk3000@gmail.com (JIRA)

unread,
Oct 29, 2018, 9:05:23 AM10/29/18
to jenkinsc...@googlegroups.com
Ilya Evseenkov commented on Bug JENKINS-52857
 
Re: Cannot find git-lfs when it is not installed in /usr/bin/

Hi Mark,

I`m using core 2.107.3 and git plugin 3.9.1.

My Linux node runs CentOS 6.7. 

When I do "git --version" on it, I get "git version 1.7.1".

 

I installed another git into "/anotherfs/git-2.19.0/" and added it "Manage Jenkins->Global Tools configuration->git". I didn`t re-configured PATH, "which git" still point "/usr/bin/git"

 

After that, I added git lfs like "/anotherfs/git-2.19.0/bin/git lfs install". 

Command "/anotherfs/git-2.19.0/bin/git-lfs version" shows "git-lfs/2.4.0 (GitHub; linux amd64; go 1.8.3; git 1a98fa5c)"

 

After that I`m configuring my Pipeline job with Jenkinsfile and "Git LFS pull after checkout". It works on master and get Jenkinsfile, but at node:
hudson.plugins.git.GitException: Command "/anotherfs/git-2.19.0/bin/git checkout -f f52336bc40265a1f8703eac49b89cd6b7a778df7" returned status code 128:


stdout:
stderr: git-lfs filter-process: git-lfs: command not found
 

It happens, because "/anotherfs/git-2.19.0/bin/" is not in the PATH.

If I add/overwrite PATH at node configuration page, git plugin ignores it and use OS version of PATH.

 

If I add "PATH" from console, it works too,

 

Hope. it is enough )

 

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 30, 2018, 6:21:01 PM10/30/18
to jenkinsc...@googlegroups.com

Ilya Evseenkov thanks for the step-by-step description. It sounds like the problem is incorrect use of the defined git tool in all the cases where it might be used. Since there is a workaround, I'm unlikely to investigate this further.

vurda.gm@gmail.com (JIRA)

unread,
Feb 13, 2019, 9:52:03 AM2/13/19
to jenkinsc...@googlegroups.com

After downgrading plugin "Analysis Model API" from 2.1.2 to 2.1.1 I was able to checkout sources on Mac.

Note that the issue is reproducible only with "Analysis Model API" 2.1.2 and Nodes that have "Launch method" set to "Launch agent agents via SSH"

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 13, 2019, 9:58:04 AM2/13/19
to jenkinsc...@googlegroups.com

Iurii Mark I would be truly amazed if the Analysis Model API plugin downgrade were the root cause of the problem, especially since this problem was reported before Analysis Model API 2.0 was released.

I don't know why the behavior changed when you downgraded. It could be that the downgrade also included a Jenkins restart. It could be that the restart from the downgrade changed some startup ordering or caused the PATH to be initialized differently.

filip.tronnberg@bontouch.com (JIRA)

unread,
Feb 26, 2019, 11:59:02 AM2/26/19
to jenkinsc...@googlegroups.com

We had the exact same problem but were able to get this to work by going to "Manage Jenkins -> Configure System" and adding an environment variable called `PATH+EXTRA` with the value `/usr/local/bin`.

After initially attempting to manually set the path we saw the following output in the console that suggested the use of `PATH+EXTRA`

`Warning: JENKINS-41339 probably bogus PATH=""$PATH":/usr/local/bin":/usr/local/bin; perhaps you meant to use 'PATH+EXTRA=/something/bin'?`

filip.tronnberg@bontouch.com (JIRA)

unread,
Feb 26, 2019, 12:00:02 PM2/26/19
to jenkinsc...@googlegroups.com
Filip Trönnbergq edited a comment on Bug JENKINS-52857

lars.bilke@ufz.de (JIRA)

unread,
Apr 12, 2019, 8:33:02 AM4/12/19
to jenkinsc...@googlegroups.com

I have the exact same issue but none of the workarounds helped.

jmeador@vectorform.com (JIRA)

unread,
Apr 12, 2019, 3:45:06 PM4/12/19
to jenkinsc...@googlegroups.com

I have a different workaround.

Run this command to see where your git programs are stored.

git --exec-path

If you want to be sure it's the correct git binary, use the full path.

/usr/bin/git --exec-path

or

/usr/local/bin/git --exec-path

For me that command outputs the following directory. This is where you want to put git-lfs.

 /usr/local/git/libexec/git-core

You could copy git-lfs into there or create a link so git-lfs can be updated in the original location.

sudo ln -s /usr/local/bin/git-lfs /usr/local/git/libexec/git-core/git-lfs

That's it, it was able to find git-lfs after that.

lars.bilke@ufz.de (JIRA)

unread,
Apr 15, 2019, 5:54:02 AM4/15/19
to jenkinsc...@googlegroups.com

Thanks Jeff, this works for me too! I had set up this symlink (I use Homebrew):

ln -s /usr/local/bin/git-lfs /usr/local/Cellar/git/2.21.0/libexec/git-core/git-lfs

devin@devserver.org (JIRA)

unread,
May 29, 2019, 11:47:02 PM5/29/19
to jenkinsc...@googlegroups.com
Devin Lane edited a comment on Bug JENKINS-52857
I just worked through this issue today and have found a reasonable solution. First, the problem lies in the way that Jenkins makes a connection to the slave; apparently, the "checkout" command uses an environment obtained from a *non-interactive* login, and BEFORE environment from Jenkins node config takes place. The rest of the build uses a "normal" environment so you may find that git lfs commands work there.

One method of fixing this is to alter /etc/ssh/sshd_config (for macOS), set "PermitUserEnvironment" to YES, then restart sshd. Next, edit /Users/Jenkins/.ssh/environment and set PATH="$PATH:/usr/local/bin" etc as required. Finally, have Jenkins reconnect to the slave to pick up the new environment.

  

 More details at [https://support.cloudbees.com/hc/en-us/articles/229724088-How-to-set-up-a-Jenkins-agent-to-have-the-same-path-as-the-user-when-connected-via-Remote-Desktop-]

devin@devserver.org (JIRA)

unread,
May 29, 2019, 11:47:02 PM5/29/19
to jenkinsc...@googlegroups.com

I just worked through this issue today and have found a reasonable solution. First, the problem lies in the way that Jenkins makes a connection to the slave; apparently, the "checkout" command uses an environment obtained from a non-interactive login, and BEFORE environment from Jenkins node config takes place. The rest of the build uses a "normal" environment so you may find that git lfs commands work there. One method of fixing this is to alter /etc/ssh/sshd_config (for macOS), set "PermitUserEnvironment" to YES, then restart sshd. Next, edit /Users/Jenkins/.ssh/environment and set PATH="$PATH:/usr/local/bin" etc as required. Finally, have Jenkins reconnect to the slave to pick up the new environment.

lars.bilke@ufz.de (JIRA)

unread,
Jul 16, 2019, 4:38:02 AM7/16/19
to jenkinsc...@googlegroups.com
Lars Bilke edited a comment on Bug JENKINS-52857
Thanks Jeff, this works for me too! I had set up this symlink (I use Homebrew):
{code:java}
ln -s /usr/local/bin/git-lfs /usr/local/Cellar/git/2.21.0/libexec/git-core/git- lf{code}
And for Macports:
{code:java}
sudo ln -s `which git- lfs ` /opt/local/libexec/git-core/git-lfs
 {code}

r.dungan@chaostheorygames.com (JIRA)

unread,
Dec 1, 2019, 6:47:03 PM12/1/19
to jenkinsc...@googlegroups.com
Rory Dungan updated an issue
 
Change By: Rory Dungan
Attachment: image-2019-12-02-10-46-34-799.png
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

r.dungan@chaostheorygames.com (JIRA)

unread,
Dec 1, 2019, 6:51:03 PM12/1/19
to jenkinsc...@googlegroups.com
Rory Dungan commented on Bug JENKINS-52857
 
Re: Cannot find git-lfs when it is not installed in /usr/bin/

Had the same problem on Mac. Setting environment variables in the node properties didn't seem to help since it looks like they are applied to "Execute shell" job stages but not the checkout stage.

Another workaround I found is to create a shell script that sets the PATH variable and then runs Git:

#!/bin/sh
export PATH=/usr/local/bin:$PATH
git "$@"

Then in the Jenkins node properties screen, set the Git executable to that script:

Feels like a bit of a hack but hopefully someone else finds it useful
Reply all
Reply to author
Forward
0 new messages