[JIRA] [git-plugin] (JENKINS-32258) Excessive Git --Rev-Parse invocation when using Prune

5 views
Skip to first unread message

dev@baltrinic.com (JIRA)

unread,
Dec 30, 2015, 11:27:04 PM12/30/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic created an issue
 
Jenkins / Bug JENKINS-32258
Excessive Git --Rev-Parse invocation when using Prune
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-plugin
Created: 31/Dec/15 4:26 AM
Environment: Introduced in Git Plugin 2.4.1
Labels: git git-plugin
Priority: Major Major
Reporter: Kenneth Baltrinic

History

This was originally discussed in the comments of

JENKINS-26587 as it was first identified as a regression when testing that issue. As Git Plugin 2.4.1 has how been release w/o resolving this regression, it has escaped and I am opening a separate ticket.

Description of Problem

Previously in (and correctly?) for when using the "Prune stale remote-tracking branches" behavior, every build, after the git checkout -f but before the git reset --hard there was a git rev-parse --verify HEAD command. Now in place of the single rev-parse command there is a git rev-parse /remotes/origin/... command for every remote branch that exists.

Steps to reproduce.

  1. Install a baseline Jenkins Master, which will include the Git Plugin v 2.4.1. (We used the 1.625.3 LTS version.)
  2. Setup credentials so that it can authenticate to a remote repo. (We use Atlasssian Stas.h.
  3. Create a freestyle job to pull from a remote git repo that has multiple (preferably many) branches.
  • branch specifier origin/master
  • Additional Behaviors: Prune stale remote-tracking branches.

Run the job and observe the output. It will be something similar to the following:

Building on master in workspace /var/lib/jenkins/workspace/git-test
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url ssh://git@stash:7999/TEST/test.git # timeout=10
Pruning obsolete local branches
Fetching upstream changes from ssh://git@stash:7999/TEST/test.git
 > git --version # timeout=10
using GIT_SSH to set credentials Credential ID: jenkins-stash-ssh
 > git fetch --tags --progress ssh://git@stash:7999/TEST/test.git +refs/heads/*:refs/remotes/origin/* --prune
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 38b70706154d37132bb67b82af7317e2ed85f9aa (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 38b70706154d37132bb67b82af7317e2ed85f9aa
First time build. Skipping changelog.
 > git branch -a # timeout=10
 > git rev-parse remotes/origin/CMOBL-4832-create-separate-onarrival-api^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5545^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5802^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5803^{commit} # timeout=10

... several thousand ref-parses later ...

 > git rev-parse remotes/origin/webformtomvc-part1^{commit} # timeout=10
 > git rev-parse remotes/origin/wip-csn-6200^{commit} # timeout=10
 > git rev-parse remotes/origin/wip-prod-20347^{commit} # timeout=10
Finished: SUCCESS

With 17K+ remote branches in our repository, this added fifteen minutes to builds that previously took three to eight minutes total. Obviously we could stand to clean up some old branches but this behavior is not acceptable.

Workarounds

  1. Remove the prune stale branches behavior.
  2. Downgrade to Git Plugin 2.4.0
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

dev@baltrinic.com (JIRA)

unread,
Dec 30, 2015, 11:28:01 PM12/30/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic updated an issue
Change By: Kenneth Baltrinic
h2. History

This was originally discussed in the comments of JENKINS-26587 as it was first identified as a regression when testing that issue.  As Git Plugin 2.4.1 has how been release w/o resolving this regression, it has escaped and I am opening a separate ticket.

h2. Description of Problem
Previously in (and correctly?)
 for  when using the "Prune stale remote-tracking branches" behavior,  in  every build, after the {{git checkout -f}} but before the {{git reset --hard}} there was a {{git rev-parse --verify HEAD}} command. Now in place of the single rev-parse command there is a {{git rev-parse /remotes/origin/...}} command _for every remote branch that exists_. 

h2. Steps to reproduce.  
# Install a baseline Jenkins Master, which will include the Git Plugin v 2.4.1. (We used the 1.625.3 LTS version.)
# Setup credentials so that it can authenticate to a remote repo. (We use Atlasssian Stas.h.
# Create a freestyle job to pull from a remote git repo that has multiple (preferably many) branches.
  * branch specifier origin/master
  * Additional Behaviors: Prune stale remote-tracking branches.


Run the job and observe the output.  It will be something similar to the following:

{code}Building on master in workspace /var/lib/jenkins/workspace/git-test

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url ssh://git@stash:7999/TEST/test.git # timeout=10
Pruning obsolete local branches
Fetching upstream changes from ssh://git@stash:7999/TEST/test.git
 > git --version # timeout=10
using GIT_SSH to set credentials Credential ID: jenkins-stash-ssh
 > git fetch --tags --progress ssh://git@stash:7999/TEST/test.git +refs/heads/*:refs/remotes/origin/* --prune
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 38b70706154d37132bb67b82af7317e2ed85f9aa (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 38b70706154d37132bb67b82af7317e2ed85f9aa
First time build. Skipping changelog.
 > git branch -a # timeout=10
 > git rev-parse remotes/origin/CMOBL-4832-create-separate-onarrival-api^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5545^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5802^{commit} # timeout=10
 > git rev-parse remotes/origin/CROSSTAB-5803^{commit} # timeout=10

... several thousand ref-parses later ...

 > git rev-parse remotes/origin/webformtomvc-part1^{commit} # timeout=10
 > git rev-parse remotes/origin/wip-csn-6200^{commit} # timeout=10
 > git rev-parse remotes/origin/wip-prod-20347^{commit} # timeout=10
Finished: SUCCESS{code}


With 17K+ remote branches  in our repository, this added fifteen minutes to builds that previously took three to eight minutes total. Obviously we could stand to clean up some old branches but this behavior is not acceptable.

h2. Workarounds
# Remove the prune stale branches behavior.
# Downgrade to Git Plugin 2.4.0

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 2, 2016, 5:42:02 PM1/2/16
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-32258
 
Re: Excessive Git --Rev-Parse invocation when using Prune

I've written a test which detects the failure and used `git bisect` to isolate the break to a BuildData related change from Nicolas De Loof.

I hope to investigate alternatives for a fix soon, or you can use that test and the commit information to experiment with possible fixes.

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 4, 2016, 12:38:03 AM1/4/16
to jenkinsc...@googlegroups.com

I've submitted a pull request to the git client plugin which I believe resolves this bug report without altering the algorithm that was added to reduce the size of BuildData. Until build expires, you can download git-client.hpi from the CloudBees Jenkins pull request builder.

The pull request may also be worth reviewing, in case you see some problem that I missed. I'm most likely the one who will merge the pull request, but I have requested a review from others in case there is some mistake in the code.

dev@baltrinic.com (JIRA)

unread,
Jan 8, 2016, 5:10:02 PM1/8/16
to jenkinsc...@googlegroups.com

I did a preliminary test of this on a new Jenkins box and the issue appears to be resolved. Has anyone else reviewed your pull request? I would like some more eyes on this before we put it on our production box. Any chance of getting a Git Client Plugin release soon?

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 8, 2016, 6:30:02 PM1/8/16
to jenkinsc...@googlegroups.com

There's been no feedback on the pull request either from Nicolas De Loof or from the larger request I made to the Cloudbees team and the recently created jenkins reviewers team.

Since the request for code review has been unanswered for 5 days, it may be time for me to merge the change to the master branch and release a new version of the plugin.

I can release a new version of the git client plugin which would only contain that fix, and am confident it will resolve the issue. The pull request describes some of the techniques I used to verify the fix.

scm_issue_link@java.net (JIRA)

unread,
Jan 9, 2016, 9:57:02 AM1/9/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Mark Waite
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
http://jenkins-ci.org/commit/git-client-plugin/1a450a5ee993209cc9f27f2a4bfb31bc08a33257
Log:
[Fix JENKINS-32258] remove excess rev-parse calls

The BuildData cleanup change to remove dead branches from BuildData
needs the list of current branches. Generating the list of current
branches called rev-parse for each branch. With small branch counts,
that isn't a huge problem. Large branch counts make it painfully slow.

One of the problem repositories has 17000 branches.

Since the BuildData cleanup step is introducing extra work, this may
still be slower than earlier versions, since larger sized BuildDData
was accepted prior to git plugin 2.4.1 and git client 1.19.1.

scm_issue_link@java.net (JIRA)

unread,
Jan 9, 2016, 9:57:04 AM1/9/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Mark Waite
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/89ab085182b7a4da6ae575fa4e154a3f3bf0add0
Log:
Merge pull request #196 from MarkEWaite/master-JENKINS-32258-many-rev-parse-calls

Fix JENKINS-32258 - many rev-parse calls pruning remote tracking branches

Compare: https://github.com/jenkinsci/git-client-plugin/compare/3139a94ab5b8...89ab085182b7

scm_issue_link@java.net (JIRA)

unread,
Jan 9, 2016, 9:57:04 AM1/9/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Mark Waite
Path:

src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/3a8d219569652fbe0531f6ffefa441818acaca06
Log:
Test for JENKINS-32258

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 11, 2016, 9:44:02 AM1/11/16
to jenkinsc...@googlegroups.com
Mark Waite started work on Bug JENKINS-32258
 
Change By: Mark Waite
Status: Open In Progress

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 19, 2016, 9:25:01 PM1/19/16
to jenkinsc...@googlegroups.com
Mark Waite resolved as Fixed
 

Fixed in git client plugin 1.19.2. Release to artifactory failed due to credentials error.

Change By: Mark Waite
Status: In Progress Resolved
Resolution: Fixed

dev@baltrinic.com (JIRA)

unread,
Feb 2, 2016, 8:55:04 AM2/2/16
to jenkinsc...@googlegroups.com
Kenneth Baltrinic edited a comment on Bug JENKINS-32258
 
Re: Excessive Git --Rev-Parse invocation when using Prune
Confirmed 1.19.2 fixes the issue for us.   Thank You!

dev@baltrinic.com (JIRA)

unread,
Feb 2, 2016, 8:55:04 AM2/2/16
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 22, 2019, 9:31:54 PM10/22/19
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Fixed
 
Change By: Mark Waite
Status: Resolved Closed
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages