[JIRA] (JENKINS-56569) Prevent automatic pulling of lfs files if no GitLFSPull option is set

26 views
Skip to first unread message

rene.scheibe@gmail.com (JIRA)

unread,
Mar 14, 2019, 4:06:02 PM3/14/19
to jenkinsc...@googlegroups.com
René Scheibe created an issue
 
Jenkins / Bug JENKINS-56569
Prevent automatic pulling of lfs files if no GitLFSPull option is set
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-client-plugin
Created: 2019-03-14 20:05
Environment: git-client-plugin 2.7.6
Priority: Major Major
Reporter: René Scheibe

Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

Problems

  1. If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
  2. If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

I therefore suggest to always set the environment variable GIT_LFS_SKIP_SMUDGE=1 even if no "GitLFSPull" option was specified.
This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

flokli@flokli.de (JIRA)

unread,
Jul 25, 2019, 7:26:02 PM7/25/19
to jenkinsc...@googlegroups.com

If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default?

I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified.
This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.

 

This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.

 

flokli@flokli.de (JIRA)

unread,
Jul 25, 2019, 7:48:03 PM7/25/19
to jenkinsc...@googlegroups.com

It looks like this is already implemented in https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a - however, the git lfs pull doesn't seem to properly replace lfs pointers…

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 24, 2019, 12:56:01 PM8/24/19
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
 
Change By: Mark Waite
Assignee: Mark Waite

flokli@flokli.de (JIRA)

unread,
Aug 29, 2019, 7:03:02 AM8/29/19
to jenkinsc...@googlegroups.com
 
Re: Prevent automatic pulling of lfs files if no GitLFSPull option is set

Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.

 

I think this issue can be closed.

 

I opened JENKINS-59139, as general clone performance has improved a lot recently.

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:22:02 PM11/17/19
to jenkinsc...@googlegroups.com

Regarding this point:

René Scheibe

If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

Florian Klink

Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

In the git-client-plugin it's implemented that first a git fetch (see here) and then a git checkout (see here) is performed and in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here).
If credentials are specified by the user they are only provided for git fetch and git lfs pull.
Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:25:02 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-56569
Regarding this point:

René Scheibe
{quote}If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
{quote}
Florian Klink
{quote}Shouldn't then the clone itself fail, too?

Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.
{quote}
In the git-client-plugin it's implemented that first a the following steps are performed
# {{git fetch}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L563]) and then a
# {{git checkout}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2681]) is performed and
# in case of a specified "Git LFS pull after checkout" behaviour also {{git lfs pull}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2693]) .

If credentials are specified by the user they are only provided for {{git fetch}} and {{git lfs pull}}.
Therefore if {{git-lfs filter-process}} is triggered automatically via {{git checkout}} the checkout fails because the credentials are not provided here.

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:29:02 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-56569
Regarding this point:

René Scheibe
{quote}If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
{quote}
Florian Klink
{quote}Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.
{quote}
In the git-client-plugin it's implemented that the following steps are performed
# in case of a specified "Git LFS pull after checkout" behaviour also {{git lfs pull}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2693])

If credentials are specified by the user they are only provided for {{git fetch}} and {{git lfs pull}}.
Therefore if {{git-lfs filter-process}} is triggered automatically via {{git checkout}} the checkout fails because the credentials are not provided here.


If one really want's to go the other direction to never use {{GIT_LFS_SKIP_SMUDGE}} as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. What's your opinion on this [~markewaite]?

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:29:03 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-56569
Regarding this point:

René Scheibe
{quote}If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
{quote}
Florian Klink
{quote}Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.
{quote}
In the git-client-plugin it's implemented that the following steps are performed
# {{git fetch}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L563])
# {{git checkout}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2681])
# in case of a specified "Git LFS pull after checkout" behaviour also {{git lfs pull}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2693])

If credentials are specified by the user they are only provided for {{git fetch}} and {{git lfs pull}}.
Therefore if {{git-lfs filter-process}} is triggered automatically via {{git checkout}} the checkout fails because the credentials are not provided here.

If one really want's wants to go the other direction to never use {{GIT_LFS_SKIP_SMUDGE}} as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. What's your opinion on this [~markewaite]?

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:31:03 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-56569
Regarding this point:

René Scheibe
{quote}If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
{quote}
Florian Klink
{quote}Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.
{quote}
In the git-client-plugin it's implemented that the following steps are performed
# {{git fetch}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L563])
# {{git checkout}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2681])
# in case of a specified "Git LFS pull after checkout" behaviour also {{git lfs pull}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2693])

If credentials are specified by the user they are only provided for {{git fetch}} and {{git lfs pull}}.
Therefore if {{git-lfs filter-process}} is triggered automatically via {{git checkout}} the checkout fails because the credentials are not provided here.

If one really wants to go the other direction to never use {{GIT_LFS_SKIP_SMUDGE}} as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this [~markewaite]?

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:34:03 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
 
Change By: René Scheibe
Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:
{code:java}

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true
{code}

This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

Problems
# If the remote server requires authentication and none is configured - on the system - on a Jenkins agent (e.g. ssh key in \{{~/.ssh/}} ), the checkout fails.
# If lfs files shall not be pulled at all, there is no way to configure this in the plugin.


I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}} even if no "GitLFSPull" option was specified.
This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

rene.scheibe@gmail.com (JIRA)

unread,
Nov 17, 2019, 12:42:02 PM11/17/19
to jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-56569
Regarding this point:

 René Scheibe [~renescheibe]
{quote}If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.
{quote}
Florian Klink [~flokli]
{quote}Shouldn't then the clone itself fail, too?
Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.
{quote}
In the git-client-plugin it's implemented that the following steps are performed
# {{git fetch}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L563])
# {{git checkout}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2681])
# in case of a specified "Git LFS pull after checkout" behaviour also {{git lfs pull}} (see [here|https://github.com/jenkinsci/git-client-plugin/blob/git-client-3.0.0/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2693])

If credentials are specified by the user they are only provided for {{git fetch}} and {{git lfs pull}}.
Therefore if {{git-lfs filter-process}} is triggered automatically via {{git checkout}} the checkout fails because the credentials are not provided here.

If one really wants to go the other direction to never use {{GIT_LFS_SKIP_SMUDGE}} as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this [~markewaite]?

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 17, 2019, 4:11:05 PM11/17/19
to jenkinsc...@googlegroups.com

I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users.

I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job.

However, René Scheibe, wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?

rene.scheibe@gmail.com (JIRA)

unread,
Nov 18, 2019, 4:29:14 PM11/18/19
to jenkinsc...@googlegroups.com

Right, that's a breaking change so to speak.

If GIT_LFS_SKIP_SMUDGE=1 is always set to disable any enabled smudge filter and git lfs pull is only performed when the "Git LFS pull after checkout" behaviour is specified, users that currently have the smudge filter enabled via a system/global/local git config would have to change their job configuration.

rene.scheibe@gmail.com (JIRA)

unread,
Nov 18, 2019, 4:34:06 PM11/18/19
to jenkinsc...@googlegroups.com

Above I also linked JENKINS-45228 which is about git LFS at merge time. (This here is about git LFS at checkout time).

rene.scheibe@gmail.com (JIRA)

unread,
Nov 19, 2019, 6:15:02 PM11/19/19
to jenkinsc...@googlegroups.com

Above I also linked JENKINS-59516 which is about git LFS pull for submodules. In case it's decided here that the smudge filter is always disabled via GIT_LFS_SKIP_SMUDGE=1, then the referenced issue has to be worked on too. Otherwise it would not be possible at all to pull LFS files in submodules (even not with configuration outside of Jenkins).
Reply all
Reply to author
Forward
0 new messages