(I hypothesize that each build runs Project Build Discarder twice in parallel and those race against each other
I don't remember how far along I was in the investigation when I posted the comment, but IIRC this is correct. Specifically, the "background" build discarders are also run after a build finishes, in addition to the regular build discarder. The idea being, why should it only run periodically, when it could as well run when a build is finished, and not accumulate a lot of builds for its periodic run? https://github.com/jenkinsci/jenkins/blob/da90af311587f6c3d37ec4e9c4b4637763924743/core/src/main/java/jenkins/model/GlobalBuildDiscarderListener.java#L46-L50 Now, if you configure the "job specific build discarder" as part of global/background build discarders (the default), there'll be two build discarder instantiations running after a build is finished, both configured to delete the exact same builds. So this configuration basically creates the circumstances under which the (AFAICT) long-standing concurrency bug occurs. (Strictly speaking, any global build discarder could trigger this, but if the configurations are different, it's likely that only one build discarder will ever find builds to delete.) A workaround for this specific case only could be to skip the "project specific build discarder" when we're triggered by a build finishing (i.e. only run it when triggered by the periodic run), but that seems like a hack, but doesn't consider fixed global build discarders that match the job configuration exactly. A more reasonable workaround, don't run multiple build discarders on a job in parallel. There's currently no synchronization there.
Ideally however, we'd fix the concurrency issues in build deletion, then neither builds in quick succession, nor background build discarders will be a problem. |
