[JIRA] (JENKINS-61233) Do not activate the build discarder during version updates

22 views
Skip to first unread message

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Feb 26, 2020, 6:49:03 AM2/26/20
to jenkinsc...@googlegroups.com
Andreas Nusser created an issue
 
Jenkins / Improvement JENKINS-61233
Do not activate the build discarder during version updates
Issue Type: Improvement Improvement
Assignee: Unassigned
Components: core
Created: 2020-02-26 11:48
Environment: Jenkins 2.222
OpenJdk 1.8
CentOS Linux release 7.7.1908 (Core)
Installed via YUM
Plugins:
org.jenkins-ci.main:jenkins-war:2.222
org.jenkins-ci:crypto-util:1.1
commons-httpclient:commons-httpclient:3.1-jenkins-1
aopalliance:aopalliance:1.0
com.google.code.findbugs:annotations:3.0.0
commons-beanutils:commons-beanutils:1.9.3
com.google.inject:guice:4.0
org.springframework:spring-dao:1.2.9
org.kohsuke.stapler:stapler-jrebel:1.258
org.codehaus.groovy:groovy-all:2.4.12
org.jenkins-ci:constant-pool-scanner:1.2
org.jenkins-ci.modules:windows-slave-installer:1.12
org.connectbot.jbcrypt:jbcrypt:1.0.0
org.ow2.asm:asm-commons:5.0.3
org.jenkins-ci:symbol-annotation:1.1
org.slf4j:slf4j-api:1.7.26
commons-digester:commons-digester:2.1
org.kohsuke:libpam4j:1.11
com.github.jnr:jnr-posix:3.0.45
org.jvnet.winp:winp:1.28
org.apache.commons:commons-compress:1.19
org.jenkins-ci.modules:instance-identity:2.2
org.kohsuke:asm6:6.2
net.sf.kxml:kxml2:2.3.0
org.kohsuke:libzfs:0.8
org.jenkins-ci.main:remoting:4.2
org.jenkins-ci.modules:sshd:2.6
org.kohsuke:access-modifier-annotation:1.16
org.jenkins-ci.modules:slave-installer:1.7
org.kohsuke.stapler:json-lib:2.4-jenkins-2
org.jfree:jfreechart:1.0.19
org.slf4j:slf4j-jdk14:1.7.26
org.jvnet.robust-http-client:robust-http-client:1.2
org.ow2.asm:asm:5.0.3
com.github.jnr:jnr-ffi:2.1.8
com.github.jnr:jnr-constants:0.9.9
org.kohsuke.stapler:stapler-adjunct-timeline:1.5
org.jvnet.hudson:commons-jelly-tags-define:1.0.1-hudson-20071021
commons-lang:commons-lang:2.6
org.springframework:spring-jdbc:1.2.9
org.codehaus.woodstox:wstx-asl:3.2.9
org.springframework:spring-core:2.5.6.SEC03
org.springframework:spring-aop:2.5.6.SEC03
org.jfree:jcommon:1.0.23
org.samba.jcifs:jcifs:1.3.17-kohsuke-1
net.java.dev.jna:jna:5.3.1
net.i2p.crypto:eddsa:0.3.0
org.jenkins-ci:winstone:5.8
com.sun.solaris:embedded_su4j:1.1
com.github.jnr:jffi:1.2.17
javax.inject:javax.inject:1
org.jenkins-ci.modules:upstart-slave-installer:1.1
args4j:args4j:2.33
org.fusesource.jansi:jansi:1.11
org.springframework:spring-beans:2.5.6.SEC03
net.java.sezpoz:sezpoz:1.13
javax.xml.stream:stax-api:1.0-2
org.jvnet.hudson:activation:1.1.1-hudson-1
commons-jelly:commons-jelly-tags-fmt:1.0
org.slf4j:log4j-over-slf4j:1.7.26
oro:oro:2.0.8
org.jenkins-ci:commons-jexl:1.1-jenkins-20111212
org.jenkins-ci.plugins.icon-shim:icon-set:1.0.5
org.apache.ant:ant-launcher:1.9.14
stax:stax-api:1.0.1
com.google.code.findbugs:jsr305:2.0.1
org.kohsuke:windows-package-checker:1.2
org.acegisecurity:acegi-security:1.0.7
org.slf4j:jcl-over-slf4j:1.7.26
commons-fileupload:commons-fileupload:1.3.1-jenkins-2
org.jenkins-ci.modules:launchd-slave-installer:1.2
org.jenkins-ci:annotation-indexer:1.12
jline:jline:2.12
commons-codec:commons-codec:1.12
org.jenkins-ci:task-reactor:1.5
org.kohsuke.stapler:stapler-adjunct-zeroclipboard:1.3.5-1
commons-io:commons-io:2.6
org.kohsuke.stapler:stapler-adjunct-codemirror:1.3
org.ow2.asm:asm-util:5.0.3
org.jenkins-ci:bytecode-compatibility-transformer:2.0-beta-2
org.apache.sshd:sshd-core:1.7.0
org.kohsuke.stapler:stapler-jelly:1.258
org.kohsuke:akuma:1.10
javax.mail:mail:1.4.4
org.hamcrest:hamcrest-core:1.3
org.springframework:spring-context-support:2.5.6.SEC03
com.google.guava:guava:11.0.1
org.jenkins-ci.modules:ssh-cli-auth:1.7
org.jvnet.hudson:jtidy:4aug2000r7-dev-hudson-1
org.jenkins-ci:version-number:1.6
org.jenkins-ci:commons-jelly:1.1-jenkins-20120928
org.jenkins-ci.ui:handlebars:1.1.1
org.springframework:spring-context:2.5.6.SEC03
org.jvnet.localizer:localizer:1.26
org.jenkins-ci.ui:jquery-detached:1.2.1
org.ow2.asm:asm-analysis:5.0.3
io.github.stephenc.crypto:self-signed-cert-generator:1.0.0
javax.servlet.jsp.jstl:javax.servlet.jsp.jstl-api:1.2.1
commons-discovery:commons-discovery:0.4
com.github.jnr:jffi:1.2.16
org.jenkins-ci:memory-monitor:1.9
org.jenkins-ci.modules:systemd-slave-installer:1.1
org.jvnet.hudson:xstream:1.4.7-jenkins-1
org.jvnet:tiger-types:2.2
com.sun.xml.txw2:txw2:20110809
org.springframework:spring-web:2.5.6.SEC03
org.kohsuke.jinterop:j-interop:2.0.6-kohsuke-1
io.jenkins.stapler:jenkins-stapler-support:1.1
org.jenkins-ci.main:jenkins-core:2.222
org.jruby.ext.posix:jna-posix:1.0.3-jenkins-1
org.kohsuke.stapler:stapler-groovy:1.258
javax.annotation:javax.annotation-api:1.2
org.kohsuke.jinterop:j-interopdeps:2.0.6-kohsuke-1
com.infradna.tool:bridge-method-annotation:1.13
org.ow2.asm:asm-tree:5.0.3
org.kohsuke:asm5:5.0.1
org.jenkins-ci.main:cli:2.222
antlr:antlr:2.7.6
relaxngDatatype:relaxngDatatype:20020414
com.jcraft:jzlib:1.1.3-kohsuke-1
org.jenkins-ci.ui:bootstrap:1.3.2
commons-collections:commons-collections:3.2.2
junit:junit:4.12
net.sf.ezmorph:ezmorph:1.0.6
org.kohsuke.stapler:stapler:1.258
org.springframework:spring-webmvc:2.5.6.SEC03
com.github.jnr:jnr-x86asm:1.0.2
xpp3:xpp3:1.1.4c
jaxen:jaxen:1.1-beta-11
org.apache.ant:ant:1.9.14
org.dom4j:dom4j:2.1.1
commons-jelly:commons-jelly-tags-xml:1.1
Priority: Minor Minor
Reporter: Andreas Nusser

After I upgraded Jenkins from 2.220 to 2.222, the build discarder started to delete a ton of old builds. This had the effect that Jenkins somehow lost track if a branch of a multi-branch-pipeline job was already built. Which resulted in all of these branches being triggered and subsequently causing a huge spike in load. Additionally, this triggerd a bug in the Gitlab plugin (https://github.com/jenkinsci/gitlab-plugin/issues/1030) which caused almost all of these builds to fail.

 

Please don't do this. Don't introduce global features (build discarder) which mess with projects/jobs. I believe that this feature is perfect for newly setup Jenkins servers but not for Jenkins servers which are already running for a long time.

 

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

dbeck@cloudbees.com (JIRA)

unread,
Feb 29, 2020, 8:38:04 AM2/29/20
to jenkinsc...@googlegroups.com

I believe that this feature is perfect for newly setup Jenkins servers but not for Jenkins servers which are already running for a long time.

Unfortunately next to impossible to make this distinction.

I am surprised that so many jobs are affected though. The default global build discarder simply applies jobs' already configured build discarder, and would only delete builds that would be deleted on the next run anyway. Is there something about your setup that would make the configuration much more aggressive since their last run?

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Mar 3, 2020, 4:05:03 PM3/3/20
to jenkinsc...@googlegroups.com
Andreas Nusser updated an issue
 
Change By: Andreas Nusser
After I upgraded Jenkins from 2.220 to 2.222, the build discarder started to delete a ton of old builds. This had the effect that Jenkins somehow lost track if a branch of a multi-branch-pipeline job was already built. Which resulted in all of these branches being triggered and subsequently causing a huge spike in load. Additionally, this triggerd a bug in the [ Gitlab plugin ( Plugin| [https://github.com/jenkinsci/gitlab-plugin/issues/1030 ) ] ] which caused almost all of these builds to fail.

 

Please don't do this. Don't introduce global features (build discarder) which mess with projects/jobs. I believe that this feature is perfect for newly setup Jenkins servers but not for Jenkins servers which are already running for a long time.

 

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Mar 3, 2020, 4:06:02 PM3/3/20
to jenkinsc...@googlegroups.com
Andreas Nusser commented on Improvement JENKINS-61233
 
Re: Do not activate the build discarder during version updates

I don't know how special our setup is, but it is quiet customized. If necessary I can provide you with more details later but the 1000m view is:

  • (Almost) all our Jobs are defined by Seed-Jobs via the [Job DSL Plugin|https://jenkinsci.github.io/job-dsl-plugin/]
  • (Almost) all Jobs are multi branch pipeline jobs
  • I'm right now not 100% but I believe that these jobs use the "orphanedItemStrategy". But I would need to double check when I'm back at the office.
  • The branches tend to get rather old. Because of reasons ... branches are created for a specific date and after this date left untouched for most of the time. This means that there are multi branch jobs with branches last changed and built 2018. Before the "incident" these jobs would still hold a build from 2018.

Do you need/want more information?

dbeck@cloudbees.com (JIRA)

unread,
Mar 3, 2020, 4:39:03 PM3/3/20
to jenkinsc...@googlegroups.com
Daniel Beck edited a comment on Improvement JENKINS-61233
[~jenkinsjenkins] Could you share how these pipelines are configured, specifically the value for the {{logRotator}}? Or perhaps {{buildDiscarder}}? Nothing you describe inherently would cause this problem.

Alternatively, run the following in the Script Console:
{noformat}
Jenkins.get().allItems(Job.class).each { j ->
  println j.buildDiscarder
  if (j.buildDiscarder instanceof hudson.tasks.LogRotator) {
    def lr = j.buildDiscarder
    println "${lr.daysToKeep} ${lr.numToKeep}"
  }
}
return {noformat}

dbeck@cloudbees.com (JIRA)

unread,
Mar 3, 2020, 4:39:03 PM3/3/20
to jenkinsc...@googlegroups.com

Andreas Nusser Could you share how these pipelines are configured, specifically the value for the logRotator? Or perhaps buildDiscarder? Nothing you describe inherently would cause this problem.

Alternatively, run the following in the Script Console:

Jenkins.get().allItems(Job.class).each { j ->
  if (j.buildDiscarder instanceof hudson.tasks.LogRotator) {
    def lr = j.buildDiscarder
    println "${lr.daysToKeep} ${lr.numToKeep}"
  }
}
return 
Add Comment Add Comment
 

dbeck@cloudbees.com (JIRA)

unread,
Mar 3, 2020, 4:55:02 PM3/3/20
to jenkinsc...@googlegroups.com

In fact, what's confusing is that the built-in build discarder, which is the most likely one to be configured on your projects, will never delete the most recent successful build. There may be some wonkiness if configured to retain 0 builds, and the latest build is failing. But that would be a really weird setup.

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Mar 4, 2020, 2:58:02 AM3/4/20
to jenkinsc...@googlegroups.com
Andreas Nusser updated an issue
Change By: Andreas Nusser
Attachment: jenkins-script-output.txt

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Mar 4, 2020, 3:01:02 AM3/4/20
to jenkinsc...@googlegroups.com
Andreas Nusser commented on Improvement JENKINS-61233
 
Re: Do not activate the build discarder during version updates

The output of your script is attached as file. jenkins-script-output.txt

The seed jobs looke similar to:

 

def createBuildPipeline(folderPath, projectName, pipelineName = BUILD_DIR) {
    def repoGit = Constants.getSCMRepo(projectName)
    def repoHttp = Constants.getHTTPRepo(projectName)    folder(folderPath) {
        description("Bla bla")
    }    // Build Jobs
    multibranchPipelineJob(folderPath + pipelineName) {
        description("Bla bla")
        branchSources {
            def uniqueId = "${folderPath}-${projectName}-${pipelineName}"
            git {
                id(uniqueId)
                remote(repoGit)
                credentialsId(JENKINS_CREDENTIALS_ID)
            }
        }
        triggers {
            periodicFolderTrigger{
                interval(SCAN_REPO_AFTER_MINUTES)
            }
        }        orphanedItemStrategy {
            // Trims dead items by the number of days or the number of items.
            discardOldItems {
                // Sets the number of days to keep old items.
                daysToKeep(DELETED_BRANCHES_KEEP_DAYS)
                // Sets the number of old items to keep.
                numToKeep(DELETED_BRANCHES_KEEP_NUMBER)
            }
        }        configure { node ->
            node / 'sources' / 'data' / 'jenkins.branch.BranchSource' {
                source(class: 'jenkins.plugins.git.GitSCMSource') {
                    remote(repoGit)
                    credentialsId(JENKINS_CREDENTIALS_ID)
                    includes('*')
                    excludes()
                    ignoreOnPushNotifications('false')
                }                // this property prevents scanning the pipeline from triggering builds of all branches
                strategy(class: 'jenkins.branch.DefaultBranchPropertyStrategy') {
                    properties(class: 'java.util.Arrays$ArrayList') {
                        a(class: 'jenkins.branch.BranchProperty-array') {
                            'jenkins.branch.NoTriggerBranchProperty'()
                        }
                    }
                }
            }        }
    }
}

I mean, there is the possibility that this was not the fault of the build discarder but some other feature (Gitlab cough). Still the fact remains that after the upgrade, where I upgraded Jenkins and the plugins, I had to rebuild all branches because the old builds where gone. Maybe there was even a bug in the old discarding handling and the fix caused this. 

 

dbeck@cloudbees.com (JIRA)

unread,
Mar 4, 2020, 3:43:03 AM3/4/20
to jenkinsc...@googlegroups.com

TBH my best guess is that Gitlab misbehaved.

Here's an idea: Look at build numbers. If they're very low, like build #1, that would indicate that the SCM deleted the branch and recreated it. If they're where you expect them to be, then something else is going on.

dbeck@cloudbees.com (JIRA)

unread,
Mar 4, 2020, 3:43:03 AM3/4/20
to jenkinsc...@googlegroups.com
Daniel Beck edited a comment on Improvement JENKINS-61233
TBH my best guess is that Gitlab misbehaved.

Here's an idea: Look at build numbers. If they're very low, like build #1, that would indicate that the SCM deleted the branch job and recreated it. If they're where you expect them to be, then something else is going on.

andreas.nusser+jenkins@gmail.com (JIRA)

unread,
Mar 4, 2020, 4:11:03 AM3/4/20
to jenkinsc...@googlegroups.com

They have number like #22 or #27

If you like to close the issue, it is fine with me. I just wanted to raise awareness for a potential problem.

dbeck@cloudbees.com (JIRA)

unread,
Mar 4, 2020, 4:15:04 AM3/4/20
to jenkinsc...@googlegroups.com

I don't want to close this, but understand what happened. And all the straightforward explanations are out by this point I think

So far there's a single report of data loss, so the likelihood is that it's something specific about your setup – but even if we decide nothing needs fixing here, it's still good to understand the circumstances.

dbeck@cloudbees.com (JIRA)

unread,
Mar 4, 2020, 4:21:03 AM3/4/20
to jenkinsc...@googlegroups.com

Oleg Nenashev PTAL. Are we inviting huge problems in the upcoming LTS here, or just a freak outlier? Can you think of other causes for this problem?

dbeck@cloudbees.com (JIRA)

unread,
Apr 18, 2020, 3:42:02 PM4/18/20
to jenkinsc...@googlegroups.com
Daniel Beck closed an issue as Won't Fix
 

As there have been no further reports of this behavior despite the feature having been in LTS for several weeks, I expect the issue that caused log deletion to be unrelated to global build discarders.

Change By: Daniel Beck
Status: Open Closed
Resolution: Won't Fix
Reply all
Reply to author
Forward
0 new messages