[JIRA] [git-plugin] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

324 views
Skip to first unread message

scott.hebert@ericsson.com (JIRA)

unread,
May 28, 2015, 9:23:02 AM5/28/15
to jenkinsc...@googlegroups.com
Scott Hebert commented on Bug JENKINS-19022
 
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

I migrated the script to https://wiki.jenkins-ci.org/display/JENKINS/Remove+Git+Plugin+BuildsByBranch+BuildData

The latest update includes a fix to better support SCM polling.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

nthiebaud@gmail.com (JIRA)

unread,
Jan 26, 2016, 6:20:02 PM1/26/16
to jenkinsc...@googlegroups.com

Thanks Scott for making the script available.
Got it hard by this bug.... to a point were the instance was unusable
I finally found this bug report and your script.. it saved the day, thanks again.

nthiebaud@gmail.com (JIRA)

unread,
Jan 26, 2016, 6:20:05 PM1/26/16
to jenkinsc...@googlegroups.com
Norbert Thiebaud edited a comment on Bug JENKINS-19022
Thanks Scott for making the script available.
Got  it  hit  hard by this bug.... to a point were the instance was unusable

I finally found this bug report and your script.. it saved the day, thanks again.

jglick@cloudbees.com (JIRA)

unread,
Mar 10, 2016, 3:04:01 PM3/10/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
 
Jenkins / Bug JENKINS-19022
Change By: Jesse Glick
Labels: branch build.xml builddata git  performance

jendap@gmail.com (JIRA)

unread,
May 31, 2016, 5:42:01 PM5/31/16
to jenkinsc...@googlegroups.com
Jan Prach commented on Bug JENKINS-19022
 
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

This has quadratic memory complexity for modern workflows!

Nicolas De Loof any progress fixing this upstream? Can I help?

Thank you for the workaround Scott Hebert!

This is very bad. It renders jenkins unusable. In our case jenkins lazy.BuildReference took 95% heap. But actually 91% heap is the git.util.BuildData. Those builds are by default hold as soft references. It sounds good. Problem is once those deserialized build.xml files take more than a heap it will deserialize gigabytes of data on every job view. It will not OOM because it is freeing those softreferences within a single request. But it can be extremely slow. It can read GB of data on every request. It is not ok. It cause timeouts and get progressively worse with every branch and/or every record in job history.

I would be surprised if some (many) people would not leave jenkins for this issue. It is not easy to say for most people why page load takes 20 seconds or more.

jendap@gmail.com (JIRA)

unread,
Jun 1, 2016, 6:56:02 PM6/1/16
to jenkinsc...@googlegroups.com

We have run the script. I was suspicious if keeping only getLastBuiltRevision() is the right thing to do. It is not. At least not in combination with multijob plugin. It has triggered to rebuild all the branches again. Thousands of builds. Luckily I did it no 30 seconds sanity job. If I would have done it on all our jobs it would trigger about 20k 30 minutes long builds.

The only safe option is to remove the git branches and keep build history short. The script is not good enough.

thefriendlycoder@gmail.com (JIRA)

unread,
Jul 27, 2016, 12:09:02 PM7/27/16
to jenkinsc...@googlegroups.com

Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. Also, in such cases, when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases it can actually cause the dashboard to become unresponsive and time out.

This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process.
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

thefriendlycoder@gmail.com (JIRA)

unread,
Jul 27, 2016, 12:10:02 PM7/27/16
to jenkinsc...@googlegroups.com
Kevin Phillips edited a comment on Bug JENKINS-19022
Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. Also, in such cases, when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases (i.e.: with hundreds or thousands of builds in the history) it can actually cause the entire dashboard to become unresponsive and time out.


This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process.

thefriendlycoder@gmail.com (JIRA)

unread,
Jul 27, 2016, 12:19:04 PM7/27/16
to jenkinsc...@googlegroups.com
Kevin Phillips edited a comment on Bug JENKINS-19022
Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job (recursively) of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. In our test cases approximately 90% of that data is consumed by the Git BuildData sections.

Also, in such cases , when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases (i.e.: with hundreds or thousands of builds in the history) it can actually cause the entire dashboard to become unresponsive and time out.


This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process.

lwalkera+jenkins@ieee.org (JIRA)

unread,
Jul 27, 2016, 1:07:03 PM7/27/16
to jenkinsc...@googlegroups.com

+1 To what Kevin said. We have a MultiJob that 3-levels deep with other MultiJobs and MatrixJobs, and new runs typically have 4-5 MB build.xml files with most of that being GitBuildData. We're using the StashBuildTrigger plugin so wiping the build data is ok for us, and we saw the disk usage go from 3GB to 187MB after purging the Git BuildData with a modified version of the script above.

import hudson.matrix.*
import hudson.model.*
import com.tikal.jenkins.plugins.multijob.*

hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.getAllItems(AbstractProject.class);

// Iterate over all jobs and find the ones that have a hudson.plugins.git.util.BuildData
// as an action.
//
// We then clean it by removing the useless array action.buildsByBranchName
//
def numJobs = 0;
def runcounter = 0;
def cleanGit;
cleanGit = { build ->
  gitActions = build.getActions(hudson.plugins.git.util.BuildData.class);
  if (gitActions != null) {
    for (action in gitActions) {
      action.buildsByBranchName = new HashMap<String, Build>();
      hudson.plugins.git.Revision r = action.getLastBuiltRevision();
      if (r != null) {
        for (branch in r.getBranches()) {
         action.buildsByBranchName.put(branch.getName(), action.lastBuild)
       }
      }
      build.actions.remove(action)
      build.actions.add(action)
      build.save();
      runcounter++;
    }
  }
};

for (job in allItems) {
  numJobs++;
  def counter = 0;
  for (locbuild in job.getBuilds()) {
    // It is possible for a build to have multiple BuildData actions
    // since we can use the Mulitple SCM plugin.
    def gitActions = locbuild.getActions(hudson.plugins.git.util.BuildData.class)
    if (gitActions != null) {
      for (action in gitActions) {
        counter++;
      }
    }
    if (job instanceof MatrixProject) {
      runcounter = 0;
      for (run in locbuild.getRuns()) {
        cleanGit(run);
      }
      if (runcounter > 0) {
        println(" -->> cleaned: " + runcounter + " runs");
      }
    }

    if (job instanceof MultiJobProject) {
      runcounter = 0;
      cleanGit(locbuild);
      def recurseSubBuild;
      recurseSubBuild = { sb ->
          for(bld in sb)
          {
            if(bld.build != null)
            {
              cleanGit(bld.build);
              if(bld.build instanceof MultiJobBuild)
              {
                if(bld.build.getSubBuilds().size() != 0) {
                  recurseSubBuild(bld.build.getSubBuilds());
                }
              }
            }
          }
      };
      recurseSubBuild(locbuild.getSubBuilds());
      println("***************");
      if (runcounter > 0) {
        println(" -->> cleaned: " + runcounter + " runs");
      }
    }
    if (counter > 0) {
      println("-- cleaned: " + counter + " builds");
    }
  }
}

lwalkera+jenkins@ieee.org (JIRA)

unread,
Jul 27, 2016, 1:09:04 PM7/27/16
to jenkinsc...@googlegroups.com
Laine Walker-Avina edited a comment on Bug JENKINS-19022
+1 To what Kevin said. We have a MultiJob that 3-levels deep with other MultiJobs and MatrixJobs, and new runs typically have 4-5 MB build.xml files with most of that being GitBuildData. We're using the StashBuildTrigger plugin so wiping the build data is ok for us, and we saw the disk usage go from 3GB to 187MB after purging the Git BuildData with a modified version of the script above.
As a workaround we have to have the jenkns master on a server with 32 GB of RAM and hope that everything fits into the page cache or heap.

{code:java}
{code}

lwalkera+jenkins@ieee.org (JIRA)

unread,
Jul 27, 2016, 1:11:02 PM7/27/16
to jenkinsc...@googlegroups.com
Laine Walker-Avina updated an issue
 
Change By: Laine Walker-Avina
Labels: branch build.xml builddata git multi-job performance

lwalkera+jenkins@ieee.org (JIRA)

unread,
Jul 27, 2016, 1:11:03 PM7/27/16
to jenkinsc...@googlegroups.com
Laine Walker-Avina updated an issue
Change By: Laine Walker-Avina
Labels: branch build.xml builddata disk git multi-job performance

aheritier@apache.org (JIRA)

unread,
Aug 2, 2016, 8:41:01 AM8/2/16
to jenkinsc...@googlegroups.com
Arnaud Héritier assigned an issue to Unassigned
Change By: Arnaud Héritier
Assignee: Nicolas De Loof

thefriendlycoder@gmail.com (JIRA)

unread,
Aug 3, 2016, 1:07:01 PM8/3/16
to jenkinsc...@googlegroups.com
Kevin Phillips commented on Bug JENKINS-19022
 
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

I have successfully tested a variation of the groovy scripts mentioned previously to remove the 90% bloat in some of our build logs. One modification I did make was to only process builds other than the last build. This appears to prevent the premature build triggers mentioned earlier since the plugin appears to cache all of the relevant build history for every branch built previously in the last build. So by excluding that one build from the script we prevent jobs from rebuilding every branch immediately afterwards. I believe this solution is preferable to preserving the last built revision being done above.

thefriendlycoder@gmail.com (JIRA)

unread,
Aug 3, 2016, 1:08:01 PM8/3/16
to jenkinsc...@googlegroups.com

Question: does anyone know what other plugins / behaviours might be affected by the purging of this Git build data from the build logs? I am trying to figure out what other subsystems might be affected so I can better test the impact of running this script before rolling it out into a production environment.

aheritier@apache.org (JIRA)

unread,
Oct 18, 2016, 3:14:02 AM10/18/16
to jenkinsc...@googlegroups.com
Arnaud Héritier resolved as Fixed
 

Was fixed by Nicolas De Loof in Git plugin >= 2.4.0
https://github.com/jenkinsci/git-plugin/pull/312
Change By: Arnaud Héritier
Status: Open Resolved
Assignee: Nicolas De Loof
Resolution: Fixed

jglick@cloudbees.com (JIRA)

unread,
Oct 20, 2016, 3:28:04 PM10/20/16
to jenkinsc...@googlegroups.com
Jesse Glick reopened an issue
 

No, the fix was reverted.
Change By: Jesse Glick
Resolution: Fixed
Status: Resolved Reopened
Assignee: Nicolas De Loof

aheritier@apache.org (JIRA)

unread,
Oct 20, 2016, 6:37:02 PM10/20/16
to jenkinsc...@googlegroups.com

scm_issue_link@java.net (JIRA)

unread,
Feb 10, 2017, 12:49:02 PM2/10/17
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/5bb7eed6a9231af15e7c0f5a964f3044381a979d
Log:
JENKINS-19022 Print a warning to the build log when the job seems to be in trouble due to buildsByBranchName bloat.

scm_issue_link@java.net (JIRA)

unread,
Feb 10, 2017, 12:50:01 PM2/10/17
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Mark Waite
Path:
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/cae9fb61ffcd2144278aec2255cef897cca569d0
Log:
Merge pull request #472 from jglick/buildsByBranchName-JENKINS-19022

JENKINS-19022 Print a warning to the build log when the job seems to be in trouble due to buildsByBranchName bloat

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 4, 2017, 12:25:02 PM3/4/17
to jenkinsc...@googlegroups.com

A warning is reported to the log by git plugin 3.1.0 released 4 Mar 2017.
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

tsunanet@gmail.com (JIRA)

unread,
May 11, 2018, 2:49:03 AM5/11/18
to jenkinsc...@googlegroups.com

There is a bit of movement on the pull request on GitHub, which gives me hope, but for those stuck with the groovy script to do periodic cleanups, I've measured that the cost is about 65ms per build on a typical dedicated server with 2 spinning drives in a RAID1.  This means that the periodic cleanups quickly get increasingly expensive once thousands of builds start to accumulate.

mark.earl.waite@gmail.com (JIRA)

unread,
May 11, 2018, 7:38:02 AM5/11/18
to jenkinsc...@googlegroups.com

Benoit Sigoure have you tried the build from the pull request in your environment? It has been running well for Jacob Keller and would be good to have additional users report their results.

jacob.keller@gmail.com (JIRA)

unread,
May 11, 2018, 11:28:03 AM5/11/18
to jenkinsc...@googlegroups.com

Yes, please. Additional testing would be a huge benefit.

You can even use the one compiled by the pull request build tester found at https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-plugin/detail/PR-578/11/artifacts

Specifically https://ci.jenkins.io/job/Plugins/job/git-plugin/job/PR-578/11/artifact/target/git.hpi

tsunanet@gmail.com (JIRA)

unread,
May 11, 2018, 4:33:04 PM5/11/18
to jenkinsc...@googlegroups.com

Thanks for the link, I've installed the plugin [version: 3.9.0-SNAPSHOT (private-69892a21-jenkins)] and disabled the daily job that runs the groovy script.  Anything special I should look out for?  The issue with the bloat is going to take time to manifest itself again as usually it requires thousands of build.xml files to accumulate.

mark.earl.waite@gmail.com (JIRA)

unread,
May 11, 2018, 4:55:02 PM5/11/18
to jenkinsc...@googlegroups.com

The biggest concern (for me) is that you enable the extension and watch for regressions in other areas of the git plugin. Does change history still display as expected? Are builds triggered as expected? Are there places where you detect the absence of the build data?

jacob.keller@gmail.com (JIRA)

unread,
May 11, 2018, 6:45:02 PM5/11/18
to jenkinsc...@googlegroups.com

Make sure that commits aren't rebuilt when they've already been covered by a build. Make sure that job start time isn't significantly worse.

tsunanet@gmail.com (JIRA)

unread,
Jun 29, 2018, 5:27:04 PM6/29/18
to jenkinsc...@googlegroups.com

Things have been working fine for me, although we recently inadvertently upgraded to 3.9.1 and lost the fix in the process.  Is there a newer version of this fix, based on 3.9.1, we could try?  Any idea on the approximate timeline to merge this fix?
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

mark.earl.waite@gmail.com (JIRA)

unread,
Jun 29, 2018, 6:08:02 PM6/29/18
to jenkinsc...@googlegroups.com

Benoit Sigoure I've made my attempt to resolve a merge conflict. The resulting artifacts should include a git.hpi that you can use.

tsunanet@gmail.com (JIRA)

unread,
Jul 6, 2018, 2:38:02 AM7/6/18
to jenkinsc...@googlegroups.com

Looks like a mistake was made in the initial rebase and the latest attempt didn't pass the build..?

mark.earl.waite@gmail.com (JIRA)

unread,
Jul 6, 2018, 7:46:03 AM7/6/18
to jenkinsc...@googlegroups.com

Benoit Sigoure unlikely that there was any mistake in the rebase. The build definitely ran but there appears to have been an infrastructure error on ci.jenkins.io. I've made a trivial change to one of the files in the pull request. The build has started again.

The "resulting artifacts" link referenced in my previous comment still includes a git.hpi which you could use for testing.

tsunanet@gmail.com (JIRA)

unread,
Jul 6, 2018, 10:51:28 AM7/6/18
to jenkinsc...@googlegroups.com

Sorry I was going by the comment on GitHub that the wrong side of the merge had been initially picked to resolve a conflict. I'll try the new artifact produced by that latest build, thanks!

tsunanet@gmail.com (JIRA)

unread,
Jul 6, 2018, 10:52:02 AM7/6/18
to jenkinsc...@googlegroups.com

tsunanet@gmail.com (JIRA)

unread,
Jul 6, 2018, 11:22:08 AM7/6/18
to jenkinsc...@googlegroups.com
Benoit Sigoure edited a comment on Bug JENKINS-19022
Hmm, the new plugin won't load because:
{code:java}
SEVERE: Failed Loading plugin Jenkins Git plugin v4.0.0-rc1685.659d6dcce0e8 (git)
java.io.IOException: Jenkins Git plugin v4.0.0-rc1685.659d6dcce0e8 failed to load.
- Jenkins Git client plugin v2.7.2 is older than required. To fix, install v3.0.0-beta3 or later.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:655)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:515)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) {code}
I downloaded the latest {{.hpi}} from [here| [ https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-client-plugin/detail/master/210/artifacts] ] .  I hope that's all that was needed :)

tsunanet@gmail.com (JIRA)

unread,
Jul 6, 2018, 11:22:08 AM7/6/18
to jenkinsc...@googlegroups.com

Hmm, the new plugin won't load because:

SEVERE: Failed Loading plugin Jenkins Git plugin v4.0.0-rc1685.659d6dcce0e8 (git)
java.io.IOException: Jenkins Git plugin v4.0.0-rc1685.659d6dcce0e8 failed to load.
 - Jenkins Git client plugin v2.7.2 is older than required. To fix, install v3.0.0-beta3 or later.
        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:655)
        at hudson.PluginManager$2$1$1.run(PluginManager.java:515)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I downloaded the latest .hpi from [here|https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-client-plugin/detail/master/210/artifacts].  I hope that's all that was needed

mark.earl.waite@gmail.com (JIRA)

unread,
Jul 6, 2018, 11:26:02 AM7/6/18
to jenkinsc...@googlegroups.com

It would be enough to install git client plugin 3.0.0-beta3, the latest git client plugin from the experimental update center, but you're also welcome to use one from the CI build. That is slightly newer than 3.0.0-beta3 and should work just as well as the CI build you used.

mark.earl.waite@gmail.com (JIRA)

unread,
Dec 4, 2018, 7:05:03 AM12/4/18
to jenkinsc...@googlegroups.com

I have completed my testing of the pull request from Jacob Keller. I need to write some documentation of the change for the release notes, then it will be ready to merge.

A week or less after that merge is completed, I intend to deliver the git client plugin 3.0.0-rc and the git plugin 4.0.0-rc builds.
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

stefan.hengelein@gmail.com (JIRA)

unread,
Feb 6, 2019, 12:33:03 PM2/6/19
to jenkinsc...@googlegroups.com

>> Kevin Phillips added a comment - 2016-08-03 17:07


> Question: does anyone know what other plugins / behaviours might be affected by the purging of this Git build data from the build logs? I am trying to figure out what other subsystems > might be affected so I can better test the impact of running this script before rolling it out into a production environment.

 

This "fix" broke SCM polling for our use case:

  • We use an GIT SCM build trigger with a pattern, so all branches that match "FOOBAR-*" are built. With this change now our jenkins tries to rebuild ALL OF OUR BRANCHES from the last years. As you can imagine that is quite the joke and would take weeks until all of our builds ran through. Do you have a solution how to fix this? I'm not so keen on having to stay on the old plugin versions forever.
  • Probably not related but another issue: "ERROR: [GitHub Commit Status Setter] - Cannot retrieve Git metadata for the build, setting build result to FAILURE"

jglick@cloudbees.com (JIRA)

unread,
Feb 6, 2019, 1:12:12 PM2/6/19
to jenkinsc...@googlegroups.com

does anyone know what other plugins / behaviours might be affected

I would start with this, though I would not expect that to be an exhaustive list of affected functionality.

stefan.hengelein@gmail.com (JIRA)

unread,
Feb 7, 2019, 11:56:03 AM2/7/19
to jenkinsc...@googlegroups.com

Jesse Glick What? I've just cited what Kevin Phillips wrote and later expanded on issues we're seeing.

I'm looking for a strategy to migrate to the new version of the plugin that doesn't involve our CI running for weeks to catch up the mess that was created here.

connor.tumbleson@gmail.com (JIRA)

unread,
Feb 7, 2019, 12:38:03 PM2/7/19
to jenkinsc...@googlegroups.com

Tagging onto this ticket, because something here caught my eye that Stefan Hengelein posted

> With this change now our jenkins tries to rebuild ALL OF OUR BRANCHES from the last years.

With the upgrade of (git-plugin) 4.0.0-rc, all our branches are being rebuilt (years old branches) after a commit to the project. This is blowing up reports, status reports, slack messages and more. As you can guess, years of builds rerunning is creating the quite the panic. I'm not even sure if the issue is resolved once this backlog of builds is done, because it might take days to get there.

What can I do to help? My first instinct is to downgrade back to 3.9.2 (git-plugin) but not sure if that will be worth it.

mark.earl.waite@gmail.com (JIRA)

unread,
Feb 7, 2019, 2:49:03 PM2/7/19
to jenkinsc...@googlegroups.com

You should downgrade to git plugin 3.9.2 then install git plugin 3.9.3 which includes a fix for an agent / tool interaction issue that was first introduced in git plugin 3.9.2.

The git plugin 4.0.0-rc is only a release candidate, not a production ready release. I made a mistake when I chose the version number. I assumed that the "-rc" suffix would deliver the plugin only from the experimental update center and not from the general update center. I was wrong. I apologize sincerely for my mistake.

The issues detected with git plugin 4.0.0-rc indicate that it was not as close to release as my testing indicated. More testing and more fixes will be needed before git plugin 4.0.0 will be released for general availability.

You can help assure that the problems you've found are fixed in git plugin 4.0.0-rc by providing a bug report which contains numbered steps that describe how someone else can see the same bug that you are seeing. The descriptions in the comments of this specific bug report seem to indicate at least two different bugs which are outside of this bug report and should be reported separately so that they can be tracked separately to resolution.

connor.tumbleson@gmail.com (JIRA)

unread,
Feb 7, 2019, 4:46:03 PM2/7/19
to jenkinsc...@googlegroups.com

Thanks Mark Waite - we reverted 3.9.2, then re-upgraded to 3.9.3. Unfortunately our builds were in quite a messed up shape. New commits to a branch were triggering all branches that still lived on the remote. We tried deleting the repos from the filesystem of Jenkins, hoping a new clone would resolve that - It did not. For other googlers who stumble upon this page, we had to restore a server backup and lose about 2 days of history on builds. We lost minor configurations, but it was way better than having our system bloated with thousands of builds and building nearly everything on each commit.

Mistakes happen mate, no worries and I'll try and file a bug to keep these bugs organized.

tzafrir11@gmail.com (JIRA)

unread,
Feb 8, 2019, 5:14:03 AM2/8/19
to jenkinsc...@googlegroups.com
tzafrir commented on Bug JENKINS-19022

The buildData is also required for gitlab plugin.
gitlab plugin pipeline step "gitlabCommitStatus" isn't working anymore with 4.0.0-rc

 It does not find any BuildData object.  Had to downgrade to 3.9.3 where all was working fine.

stefan.hengelein@gmail.com (JIRA)

unread,
Feb 8, 2019, 1:02:09 PM2/8/19
to jenkinsc...@googlegroups.com

Thanks for your quick responses!

Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit.

Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue!

 

< /td>

jacob.keller@gmail.com (JIRA)

unread,
Feb 9, 2019, 9:04:03 PM2/9/19
to jenkinsc...@googlegroups.com

tzafrir, the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData.

As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted.

If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again....

It may be worth more seriously considering the XmlFile approach.

jdoyle@iso-ne.com (JIRA)

unread,
Jun 25, 2019, 4:44:03 PM6/25/19
to jenkinsc...@googlegroups.com
Jim D commented on Bug JENKINS-19022

Mark Waite, I'm glad I found the comments in this issue, thank you!  When the final 4.0.0 is released with the changes for this issue, will it make sure to handle both BuildDetails and BuildData for previous build runs?  We updated to 4.0.0-rc some time back, and I just came across this thread recently trying to resolve a BuildData issue, and rolled back to 3.9.3.  At this point, Git plugin is working and creating BuildData for each build run, but all those old builds done with 4.0.0-rc don't have any kind of "Git Build Data" in the Jenkins UI anymore, and no BuildData or BuildDetails when retrieved programmatically from WorkflowRun.getAllActions.  I think it must be that 3.9.3 isn't able to read the new BuildDetails object info.  Will 4.0.0 be able to read both, from both 3.9.X builds and 4.0.0 builds?  Thanks again.

 

mark.earl.waite@gmail.com (JIRA)

unread,
Jun 25, 2019, 5:50:08 PM6/25/19
to jenkinsc...@googlegroups.com

Jim D this issue won't be fixed in 4.0.0. The incompatibilities from the BuildDetails change were too great for the community. The accidental release of git plugin 4.0.0-rc to the production update centers showed incompatibilities that I had missed in my testing and that others had missed in their testing.

BuildData will be the same bloated memory user in 4.0.0 that it is in 3.x.

jdoyle@iso-ne.com (JIRA)

unread,
Jun 26, 2019, 8:28:05 AM6/26/19
to jenkinsc...@googlegroups.com
Jim D commented on Bug JENKINS-19022

Thanks for the update!

zongyao8752593@163.com (JIRA)

unread,
Sep 27, 2019, 2:54:03 AM9/27/19
to jenkinsc...@googlegroups.com
zy zhang commented on Bug JENKINS-19022

Hi, you can use the below groovy script to delete git revisions for the current build.

import jenkins.model.*

jenkinsInstance = jenkins.model.Jenkins.get()

def job = jenkinsInstance.getItemByFullName(JOB_NAME);
def build = job.getBuild(BUILD_NUMBER)
def prj = build.project
def gitActions = build.getActions(hudson.plugins.git.util.BuildData.class)

if (gitActions != null) {
for (action in gitActions)

{ build.actions.remove(action) //build.actions.add(action) build.save() }

}
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages