[JIRA] (JENKINS-45228) Git merge requires authentication in LFS merges, plugin does not authenticate the git merge command

34 views
Skip to first unread message

emil.styrke@gmail.com (JIRA)

unread,
Mar 5, 2018, 4:46:01 AM3/5/18
to jenkinsc...@googlegroups.com
Emil Styrke commented on Bug JENKINS-45228
 
Re: Git merge requires authentication in LFS merges, plugin does not authenticate the git merge command

I am getting bitten by this as well now, in git plugin version 3.8.0: 

Checking out git http://<bitbucket_server>/<repo>.git http://<bitbucket_server>/<repo>.git into /var/lib/jenkins/workspace/<repo>_PR-109-ZFFDADYA3J6CFXJFIAHHACXI6SU7323HOW62DUSZOKSYPGIJTAKA@script to read Jenkinsfile
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from 2 remote Git repositories
 > git config remote.origin.url http://<bitbucket_server>/<repo>.git # timeout=10
Fetching upstream changes from http://<bitbucket_server>/<repo>.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins @ Bitbucket
 > git fetch --tags --progress http://<bitbucket_server>/<repo>.git +refs/pull-requests/109/from:refs/remotes/origin/PR-109
 > git config remote.upstream.url http://<bitbucket_server>/<repo>.git # timeout=10
Fetching upstream changes from http://<bitbucket_server>/<repo>.git
using GIT_ASKPASS to set credentials Jenkins @ Bitbucket
 > git fetch --tags --progress http://<bitbucket_server>/<repo>.git +refs/heads/release/VER_4-17:refs/remotes/upstream/release/VER_4-17
Merging remotes/upstream/release/VER_4-17 commit 9e203652b8fabc0062fd8a6c8f0f6c3d30d96dcf into PR head commit 366c0da12a0856d527cf40b87d61ed9b30662283
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 366c0da12a0856d527cf40b87d61ed9b30662283
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins @ Bitbucket
 > git lfs pull origin
 > git merge 9e203652b8fabc0062fd8a6c8f0f6c3d30d96dcf # timeout=10

And the last command hangs until it times out because "GIT_ASKPASS" is set to "echo". Mark Waite, would it be possible to let the "git lfs pull" behavior implementation add the necessary credentials for merge commands? (or alternatively make sure that all relevant LFS objects are pulled before attempting the merge, I guess)

 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
May 22, 2018, 10:23:02 AM5/22/18
to jenkinsc...@googlegroups.com

After "git merge" hangs like this and I abort the build via Jenkins, the git-lfs process keeps running on the Windows agent. IIRC, it keeps asking for credentials and running "git credential reject". I didn't check the environment variables of the processes but it seems freshEnv.put("GIT_ASKPASS", "echo"); in launchCommandIn, which is intended to prevent the build from hanging, isn't fulfilling its purpose. I wonder if it should do freshEnv.put("GIT_ASKPASS", "false"); instead; that might make the build at least fail cleanly. I should try changing my Jenkinsfiles to set that.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
May 22, 2018, 10:50:01 AM5/22/18
to jenkinsc...@googlegroups.com

I verified that the git-lfs process has GIT_ASKPASS=echo in its environment, even though my Jenkinsfile starts like this:

pipeline {
    environment {
        // Possible workaround for git-merge hanging in Jenkins
        // due to repeated Git LFS credential prompts
        GIT_ASKPASS = 'false'
    }

So, I cannot use that as a workaround.

These repeated credential retries are consuming about 80% CPU time of one hardware thread, too.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
May 22, 2018, 11:10:01 AM5/22/18
to jenkinsc...@googlegroups.com

In my case, git merge was run because of pull request discovery in a multibranch project using Bitbucket Branch Source. Perhaps that is why the environment variable in Jenkinsfile did not affect it. I have now disabled pull request discovery so that I won't have to kill processes on agents.

r.m.graham@gmail.com (JIRA)

unread,
Jun 13, 2018, 12:10:02 PM6/13/18
to jenkinsc...@googlegroups.com

I'm seeing this in my org folders and multi-branch projects (aka, anything using branch-source). It appears that Jenkins is automatically performing the LFS pull even though it hasn't been configured to do so.

The workaround I used is to explicitly enable it (add the Git LFS pull after checkout option) which configures the auth and allows it to work properly. Note that this is before the Jenkinsfile is actually processed, this is happening for me in the checkout that happens on master in order to read it.

vivien.delmon@minmaxmedical.com (JIRA)

unread,
Sep 4, 2018, 4:50:02 AM9/4/18
to jenkinsc...@googlegroups.com

I don't know if this additional information may help solving this issue. But in my case, the problem happens only when I am merging a branch where LFS files were added. I mean, it does not fail when some LFS files were added in the target branch but only when they were added in the source branch.

Wiping out workspace first.
Cloning the remote Git repository
Cloning repository git@GitServer:SomeRepository.git
 > git init C:\dev\workspace\SomeRepository\R # timeout=10
Fetching upstream changes from git@GitServer:SomeRepository.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress git@GitServer:SomeRepository.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url git@GitServer:SomeRepository.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url git@GitServer:SomeRepository.git # timeout=10
Fetching upstream changes from git@GitServer:SomeRepository.git
using GIT_SSH to set credentials 
 > git fetch --tags --progress git@GitServer:SomeRepository.git +refs/heads/*:refs/remotes/origin/* +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*
 > git rev-parse "81301a232895e0d241d185b1182e080bdf0a7236^{commit}" # timeout=10
 > git branch -a -v --no-abbrev --contains 81301a232895e0d241d185b1182e080bdf0a7236 # timeout=10
Merging Revision 81301a232895e0d241d185b1182e080bdf0a7236 (origin/prepareRelease, origin/merge-requests/5) to origin/master, UserMergeOptions{mergeRemote='origin', mergeTarget='master', mergeStrategy='default', fastForwardMode='--ff'}
 > git rev-parse "origin/master^{commit}" # timeout=10
Enabling Git LFS pull
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f origin/master
 > git config --get remote.origin.url # timeout=10
using GIT_SSH to set credentials 
 > git lfs pull origin
 > git merge --ff 81301a232895e0d241d185b1182e080bdf0a7236 # timeout=10
Enabling Git LFS pull
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 81301a232895e0d241d185b1182e080bdf0a7236
 > git config --get remote.origin.url # timeout=10
using GIT_SSH to set credentials 
 > git lfs pull origin
ERROR: Branch not suitable for integration as it does not merge cleanly: Command "git merge --ff 81301a232895e0d241d185b1182e080bdf0a7236" returned status code 128:
stdout: Updating e0c3f35..81301a2stderr: Downloading samples/BigFile (515.04 KB)
Error downloading object: samples/BigFile (cba8e11): Smudge error: Error downloading samples/BigFile (cba8e11334d9db127df2daa04ff942a88191b5785c6abf6142e6e3701cbb68fc): batch request: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).: exit status 255Errors logged to C:\dev\workspace\SomeRepository\R\.git\lfs\objects\logs\20180904T101810.8132111.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: samples/BigFile: smudge filter lfs failed
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

greg.whiteley@gmail.com (JIRA)

unread,
Feb 18, 2019, 11:24:04 PM2/18/19
to jenkinsc...@googlegroups.com

I'm seeing this problem specifically on the master, not on a node when it is trying to access the Jenkinsfile so no workaround is possible as this is pre-pipeline.

I observe the reverse of Vivien Delmon specifically the issue occurs when LFS files have been added to the target branch (eg master), but are not present on the source branch (the PR branch). This makes sense since the operation is checkout PR branch, then git merge master

In my case "Git LFS pull after checkout" is already present. Since this is prior to pipeline I can't see any other possibility for workaround. Any ideas?

oxygenxo@gmail.com (JIRA)

unread,
Feb 28, 2019, 5:47:03 AM2/28/19
to jenkinsc...@googlegroups.com

I have the same situation Vivien Delmon has.

Greg Whiteley it seems that the only workaround is to add direct 'sh' steps with direct 'git' invocation as Alexander Smorkalov suggested here

oxygenxo@gmail.com (JIRA)

unread,
Feb 28, 2019, 5:53:02 AM2/28/19
to jenkinsc...@googlegroups.com
Andrey Babushkin edited a comment on Bug JENKINS-45228
I have the same situation [~viviendelmon] has.

[~whitty] it seems that the only workaround is to add direct 'sh' steps with direct 'git' invocation as [~asmorkalov] suggested here :
{code:java}
sshagent(credentials: [account]) {
   sh "git clone ${repo_url} ."
   sh "git lfs install"
   sh "git checkout ${source_branch}"
   sh "git checkout ${target_branch}"
   sh "git merge --no-ff ${source_branch}"
   sh "git submodule update --init --recursive"
   sh "git lfs pull"
}

{code}

greg.whiteley@gmail.com (JIRA)

unread,
Feb 28, 2019, 10:04:03 PM2/28/19
to jenkinsc...@googlegroups.com

Thanks Andrey Babushkin - for me that can't work because the problem is on the "master" node as it tries to retrieve the Jenkinsfile - I'm using the Multi-branch Pipeline configuration, which (for a pull request) does:

  1. (On master) retrieve Jenkinsfile checkout repo (or update)
    1. checkout PR branch
    2. Merge to target branch
    3. Read Jenkinsfile for pipeline
  2. Execute Jenkinsfile
    1. Allocate a node node()
    2. Run scm functions checkout(scm)
      1. checkout repo (or update)
      2. checkout PR branch
      3. merge to target branch
    3. .. rest of Jenkinsfile ...

Step 1.2 fails (before the Jenkinsfile is available). Changing the Jenkinsfile would allow me to override step 2.2 with no problems, but 1.2 is executed without user/jenkinsfile intervention - it comes directly from the Multi-branch pipeline configuration and is executed on the master (not on a node) without reading the Jenkinsfile (it's a chicken and egg issue)

geoffroy.jabouley@gmail.com (JIRA)

unread,
Jun 11, 2019, 5:47:03 AM6/11/19
to jenkinsc...@googlegroups.com

Hello

also facing this issue in some cases where files where modified/added into source branch of PR.

Any idea when/if LFS credentials support will be added to the merge portion of the git plugin? Maybe at leat providing some implementation guidance for a future contribution

BR

geoffroy.jabouley@gmail.com (JIRA)

unread,
Jun 11, 2019, 11:24:05 AM6/11/19
to jenkinsc...@googlegroups.com
Geoffroy Jabouley edited a comment on Bug JENKINS-45228
Hello

also facing this issue in some cases where files where modified/added into source branch of PR.

Any idea when/if LFS credentials support will be added to the merge portion of the git plugin? Maybe at leat least providing some implementation guidance for a future contribution

BR

geoffroy.jabouley@gmail.com (JIRA)

unread,
Jun 18, 2019, 6:25:04 AM6/18/19
to jenkinsc...@googlegroups.com
Geoffroy Jabouley edited a comment on Bug JENKINS-45228
Hello

also facing this issue in some cases where files where modified/added into source branch of PR.

Any idea when/if LFS credentials support will be added to the merge portion of the git plugin? Maybe at least providing some implementation guidance for a future contribution

BR

 

Edit: [https://github.com/jenkinsci/git-client-plugin/pull/433]

jonas.hanschke@de.bosch.com (JIRA)

unread,
Aug 27, 2019, 5:30:02 AM8/27/19
to jenkinsc...@googlegroups.com

A way to work around this problem is to rebase the PR's source branch onto the target branch. The problem occurs only when there are new LFS files in the target branch, which will not be the case anymore after a rebase (they will already be present in the source branch, then).

jonas.hanschke@de.bosch.com (JIRA)

unread,
Aug 27, 2019, 5:31:03 AM8/27/19
to jenkinsc...@googlegroups.com
Jonas Hanschke edited a comment on Bug JENKINS-45228
A way to work around this problem is to rebase the PR's source branch onto the target branch. The problem occurs only when there are new LFS files in the target branch, which will not be the case anymore after a rebase (they will already be present in the source branch, then).


Still, this is a very annoying bug, which blocks our workflow (that usually doesn't involve rebasing). Any heads-up on when a fix can be expected? Thanks!

geoffroy.jabouley@gmail.com (JIRA)

unread,
Aug 27, 2019, 5:41:04 AM8/27/19
to jenkinsc...@googlegroups.com

Jonas Hanschke, you can check my ongoing PR (https://github.com/jenkinsci/git-client-plugin/pull/433, branch is based on v2.8.1), and recompile/install the plugin while change is being reviewed and integrated by Mark Waite

rene.scheibe@gmail.com (JIRA)

unread,
Dec 15, 2019, 4:00:12 PM12/15/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
 
Jenkins / Bug JENKINS-45228
Change By: René Scheibe
In case of changes in LFS data git merge requires authentication, but authentication data is not provided.

Jenkins version: 2.46.3

Git plugin version: 3.3.0

Pipeline step and log:
{code:java}
def mr_checkout_with_merge(url, account)
{
checkout changelog: true, poll: true, scm: [
$class: 'GitSCM',
branches: [[name: '${gitlabSourceBranch}']],
doGenerateSubmoduleConfigurations: false,
extensions: [
[$class: 'RelativeTargetDirectory', relativeTargetDir: 'src'],
[$class: 'UserIdentity', email: 'jenkins', name: 'jenkins'],
[$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false],
[$class: 'PreBuildMerge', options: [fastForwardMode: 'NO_FF', mergeRemote: 'origin', mergeStrategy: 'default', mergeTarget: '${gitlabTargetBranch}']],
[$class: 'CloneOption', depth: 0, noTags: false, reference: '', shallow: false, timeout: 60],
[$class: 'GitLFSPull']
],
submoduleCfg: [],
userRemoteConfigs: [[credentialsId: account, url: url]]
]
}{code}
 
ERROR: Branch not suitable for integration as it does not merge cleanly: Command "git merge --no-ff 6af4f83cc9f2fc96d68df589e9822acf2fbf7fee" returned status code 128:
stdout:
stderr: Downloading data/test_data.tar.gz (106.24 MB)
Error downloading object: data/test_data.tar.gz (1b3d845): Smudge error: Error downloading data/test_data.tar.gz (1b3d845ad0f78327a16db63e03a7bc60dd2f7134bbaeeb117c3602d037893eef): batch request: Permission denied (publickey).: exit status 255

Errors logged to /home/jenkins/workspace/Precommits/Ubuntu16/src/.git/lfs/objects/logs/20170626T055343.268201821.log

Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- – %f failed 2
error: external filter git-lfs smudge -- – %f failed
fatal: data/test_data.tar.gz: smudge filter lfs failed
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

rene.scheibe@gmail.com (JIRA)

unread,
Dec 26, 2019, 8:21:05 AM12/26/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
Change By: René Scheibe
Component/s: git-client-plugin

rene.scheibe@gmail.com (JIRA)

unread,
Dec 26, 2019, 8:22:09 AM12/26/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
In case of changes in LFS data git merge requires authentication, but authentication data is not provided.

Jenkins version: 2.46.3

Git plugin version: 3.3.0

Pipeline step and log
:

{code:java}
def mr_checkout_with_merge(url, account) {
  checkout changelog: true, poll: true, scm: [
    $class: 'GitSCM',
    branches: [[name: '${gitlabSourceBranch}']],
    doGenerateSubmoduleConfigurations: false,
    extensions: [
      [$class: 'RelativeTargetDirectory', relativeTargetDir: 'src'],
      [$class: 'UserIdentity', email: 'jenkins', name: 'jenkins'],
      [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false],
      [$class: 'PreBuildMerge', options: [fastForwardMode: 'NO_FF', mergeRemote: 'origin', mergeStrategy: 'default', mergeTarget: '${gitlabTargetBranch}']],
      [$class: 'CloneOption', depth: 0, noTags: false, reference: '', shallow: false, timeout: 60],
      [$class: 'GitLFSPull']
    ],
    submoduleCfg: [],
    userRemoteConfigs: [[credentialsId: account, url: url]]
  ]
}{code}
  Log output:
{code:java}
ERROR: Branch not suitable for integration as it does not merge cleanly: Command "git merge --no-ff 6af4f83cc9f2fc96d68df589e9822acf2fbf7fee" returned status code 128:
stdout:
stderr: Downloading data/test_data.tar.gz (106.24 MB)
Error downloading object: data/test_data.tar.gz (1b3d845): Smudge error: Error downloading data/test_data.tar.gz (1b3d845ad0f78327a16db63e03a7bc60dd2f7134bbaeeb117c3602d037893eef): batch request: Permission denied (publickey).: exit status 255
Errors logged to /home/jenkins/workspace/Precommits/Ubuntu16/src/.git/lfs/objects/logs/20170626T055343.268201821.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge – %f failed 2
error: external filter git-lfs smudge – %f failed
fatal: data/test_data.tar.gz: smudge filter lfs failed {code}

rene.scheibe@gmail.com (JIRA)

unread,
Dec 26, 2019, 8:22:09 AM12/26/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
Change By: René Scheibe
Environment:
Jenkins version: 2.46.3
Git plugin version: 3.3.0
In case of changes in LFS data git merge requires authentication, but authentication data is not provided.

 Jenkins version: 2.46.3  

Git plugin version: 3.3.0
Pipeline step:

rene.scheibe@gmail.com (JIRA)

unread,
Dec 26, 2019, 8:23:03 AM12/26/19
to jenkinsc...@googlegroups.com
René Scheibe updated an issue
In case of changes in LFS data git merge requires authentication, but authentication data is not provided.

  

rene.scheibe@gmail.com (JIRA)

unread,
Jan 3, 2020, 1:07:04 PM1/3/20
to jenkinsc...@googlegroups.com
Change By: René Scheibe
Status: Open Fixed but Unreleased
Resolution: Fixed

rene.scheibe@gmail.com (JIRA)

unread,
Jan 3, 2020, 1:10:02 PM1/3/20
to jenkinsc...@googlegroups.com

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Jan 7, 2020, 2:15:03 AM1/7/20
to jenkinsc...@googlegroups.com

Thanks, I see the git-client 3.1.0-beta binary is available from the experimental update center.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Jan 7, 2020, 2:44:05 AM1/7/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Bug JENKINS-45228
Thanks, I see the git-client 3.1.0-beta binary is available from the [experimental update center|https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/].

 It seems to be working OK so far, in merges that involve some LFS files, but I have not yet verified whether the previous version of the git-client plugin would fail in these specific merges.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Jan 7, 2020, 2:57:04 AM1/7/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Bug JENKINS-45228
Thanks, I see the git-client 3.1.0-beta binary is available from the [experimental update center|https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/].

It seems to be working OK so far, when merging a pull request to a branch in merges that involve some which LFS files , but I have not yet been modified. I also verified whether that attempting the previous version of the same merges with git-client plugin would fail in these specific merges 3 . 0.0 causes the build to hang because of repetitive authentication errors.

I should perhaps file a separate issue about the GIT_ASKPASS=echo environment variable.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Jan 7, 2020, 3:05:05 AM1/7/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Bug JENKINS-45228
Thanks, I see the git-client 3.1.0-beta binary is available from the [experimental update center|https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/].

It seems to be working OK so far, when a multibranch pipeline with Bitbucket Branch Source was merging a pull request to a branch in which LFS files have been modified. I also verified that attempting the same merges with git-client 3.0.0 causes the build to hang because of repetitive authentication errors.

I have not tested running {{checkout}} as a pipeline step from within Jenkinsfile, with either version of git-client.

I should perhaps file a separate issue about the GIT_ASKPASS=echo environment variable.

Kalle.Niemitalo@procomp.fi (JIRA)

unread,
Jan 7, 2020, 3:08:03 AM1/7/20
to jenkinsc...@googlegroups.com
Kalle Niemitalo edited a comment on Bug JENKINS-45228
Thanks, I see the git-client 3.1.0-beta binary is available from the [experimental update center|https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/].

It seems to be working OK so far, when a multibranch pipeline with Bitbucket Branch Source was merging merges a pull request to a branch in which LFS files have been modified. I also verified that attempting the same merges with git-client 3.0.0 causes the build to hang because of repetitive authentication errors. (To verify this, I had to delete the workspace first, as Git LFS would otherwise have reused the LFS files that it cached when git-client 3.1.0-beta provided the credential.)

I have not tested running {{checkout}} as a pipeline step from within Jenkinsfile, with either version of git-client.

I should perhaps file a separate issue about the GIT_ASKPASS=echo environment variable.

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 18, 2020, 1:33:05 PM1/18/20
to jenkinsc...@googlegroups.com
Mark Waite updated Bug JENKINS-45228
 

Fixed in git client plugin 3.1.0 release Jan 18, 2020.
Change By: Mark Waite
Status: Fixed but Unreleased Closed
Released As: git client plugin 3.1.0
Reply all
Reply to author
Forward
0 new messages