[JIRA] (JENKINS-43353) Ability to abort all previous running builds

0 views
Skip to first unread message

svanoort@cloudbees.com (JIRA)

unread,
Jun 8, 2018, 8:59:02 AM6/8/18
to jenkinsc...@googlegroups.com
Sam Van Oort commented on New Feature JENKINS-43353
 
Re: Ability to abort all previous running builds

Jesse Glick This is confusingly worded – could it be rephrased as "abort previous running builds even after they've passed the milestone"?
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Jun 8, 2018, 9:22:02 PM6/8/18
to jenkinsc...@googlegroups.com

Not necessarily. The milestone step is simply a tool which could be used to implement this. The use case is to abort all previous running builds. See my first comment.

xistence@0x58.com (JIRA)

unread,
Aug 23, 2018, 1:39:03 PM8/23/18
to jenkinsc...@googlegroups.com

When pushing new changes to a PR, it would be awesome if the multi branch had an option to automatically kill all previous running builds/queued builds for that branch so that only the latest gets queued/built. This way if a developer commits and pushes and then does the same again immediately afterwards we don't have two builds, especially if the builds take 20+ minutes each, and we don't care about the intermediate result.
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

jglick@cloudbees.com (JIRA)

unread,
Aug 23, 2018, 2:59:02 PM8/23/18
to jenkinsc...@googlegroups.com

Yes that is exactly the use case listed in this issue’s description.

(Pipeline builds do not generally sit in the queue, pending JENKINS-39180.)

nickbroon@gmail.com (JIRA)

unread,
Mar 14, 2019, 7:42:03 AM3/14/19
to jenkinsc...@googlegroups.com
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

bsquizzato@gmail.com (JIRA)

unread,
Apr 23, 2019, 3:39:03 PM4/23/19
to jenkinsc...@googlegroups.com

Jesse Glick's solution worked well for me until recently, when passing milestones started to take awhile. Passing the milestones used to happen very quickly but now for whatever reason it takes a second or so to pass each milestone. When your build number is 100+, obviously this can cause a big delay trying to pass each milestone created in the loop.

So I've started to go with an approach like this:

def buildNumber = env.BUILD_NUMBER as int
milestone(buildNumber - 1)
milestone(buildNumber)

The result here would be:
Build 1 runs and creates milestone 0, and milestone 1
While build 1 is running, build 2 fires. It has milestone 1 and milestone 2

Since build 2 passes milestone 1, it causes build #1 to abort.

 

bsquizzato@gmail.com (JIRA)

unread,
Apr 23, 2019, 6:39:02 PM4/23/19
to jenkinsc...@googlegroups.com
Brandon Squizzato edited a comment on New Feature JENKINS-43353
[~jglick]'s solution worked well for me until recently, when passing milestones started to take awhile. Passing the milestones used to happen very quickly but now for whatever reason it takes a second or so to pass each milestone. When your build number is 100+, obviously this can cause a big delay trying to pass each milestone created in the loop.


So I've started to go with an approach like this:

{{def buildNumber = env.BUILD_NUMBER as int}}
{{ if (buildNumber > 1) milestone(buildNumber - 1)}}

{{milestone(buildNumber)}}

The result here would be:
Build 1 runs and creates milestone
0, and milestone 1

While build 1 is running, build 2 fires. It has milestone 1 and milestone 2

Since build 2 passes milestone 1, it causes build #1 to abort.

jglick@cloudbees.com (JIRA)

unread,
Apr 24, 2019, 10:20:03 AM4/24/19
to jenkinsc...@googlegroups.com

Yes, this step is not going to be efficient when there are a lot of milestones. I filed PR 19 to record your suggestion.

xistence@0x58.com (JIRA)

unread,
Apr 24, 2019, 4:33:02 PM4/24/19
to jenkinsc...@googlegroups.com

Brandon Squizzato thanks for that solution, we are up to build number 900 and passing all of the milestones while pretty fast is not the fastest, this should help

mark.han@willowtreeapps.com (JIRA)

unread,
Oct 1, 2019, 11:47:03 AM10/1/19
to jenkinsc...@googlegroups.com
Mark Han commented on New Feature JENKINS-43353

We should make sure this change will allow for QUEUED builds to cancel RUNNING builds. We are running into a scenario where the amount of executors is limited, and the build server is being super congested by running out-of-date PR builds that should ideally be cancelled by the incoming queued PR builds. 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Oct 1, 2019, 11:57:03 AM10/1/19
to jenkinsc...@googlegroups.com

Pipeline builds are flyweight and normally exit the queue immediately; it is node blocks which would wait in queue. So just put the milestone dance at the top of the Jenkinsfile and you are fine. Working in jenkinsci/bom as far as I know.

mark.han@willowtreeapps.com (JIRA)

unread,
Oct 2, 2019, 11:40:04 AM10/2/19
to jenkinsc...@googlegroups.com
Mark Han commented on New Feature JENKINS-43353

Jesse Glick TIL about flyweight! This is helpful. In our recent past we have had issues in the past with milestones.

 

Scenario: We have 7 PRs ready using Bitbucket, and 7 PR builds are queued up using the Multibranch Pipeline. After the first one of those gets merged, the other 6 have to rebuild because Develop was changed. However, a few of the previous runs are still trying to execute so a queue forms that looks something like this (Assuming there are multiple builds surrounding these that look similarly): 

QUEUED JOBS:

**part of Android-Build-Name >> PR-2715 #8

part of Android-Build-Name >> PR-2715 #7

RUNNING JOBS:
part of Android-Build-Name >> PR-2715 #6

So, ideally we want to kill this running job #6 and the queued job #7 if a newer job #8 is queued up.

Would something similar to line number 1 in the jenkinsfile for `jenkinsci/bom` solve this?

mark.han@willowtreeapps.com (JIRA)

unread,
Oct 2, 2019, 11:40:07 AM10/2/19
to jenkinsc...@googlegroups.com
Mark Han edited a comment on New Feature JENKINS-43353
[~jglick] TIL about flyweight! This is helpful. In our recent past we have had issues in the past with milestones.

 

*Scenario*: We have 7 PRs ready using Bitbucket, and 7 PR builds are queued up using the Multibranch Pipeline. After the first one of those gets merged, the other 6 have to rebuild because Develop was changed. However, a few of the previous runs are still trying to execute so a queue forms that looks something like this (Assuming there are multiple builds surrounding these that look similarly): 

*QUEUED JOBS:*

 ** part of Android-Build-Name >> PR-2715 #8


part of Android-Build-Name >> PR-2715 #7


 *RUNNING JOBS:*

part of Android-Build-Name >> PR-2715 #6

So, ideally we want to kill this running job #6 and the queued job #7 if a newer job #8 is queued up.




 Would something similar to line number 1 in the jenkinsfile for `jenkinsci/bom` solve this?

jglick@cloudbees.com (JIRA)

unread,
Oct 2, 2019, 11:45:03 AM10/2/19
to jenkinsc...@googlegroups.com

Yes, it should. When build 8 starts, it should abort builds 6 and 7. When a build is aborted, in turn, it should abort any sh steps already running inside any node blocks, and also cancel any queue items corresponding to node blocks which have not yet started.

mark.han@willowtreeapps.com (JIRA)

unread,
Oct 2, 2019, 2:16:03 PM10/2/19
to jenkinsc...@googlegroups.com
Mark Han commented on New Feature JENKINS-43353

Ended up not using Milestone because it doesn't communicate the build # that was stopped. Instead we're trying to use 

def cancelPreviousBuilds() {
 // Check for other instances of this particular build, cancel any that are older than the current one
 def jobName = env.JOB_NAME
 def currentBuildNumber = env.BUILD_NUMBER.toInteger()
 def currentJob = Jenkins.instance.getItemByFullName(jobName)

 // Loop through all instances of this particular job/branch
 for (def build : currentJob.builds) {
 if (build.isBuilding() && (build.number.toInteger() < currentBuildNumber)) {
 echo "Older build still queued. Sending kill signal to build number: ${build.number}"
 build.doStop()
 }
 }
}

If we put this outside of our node blocks, things are working well. We previously had this in a node{} which was horrible, and resolved the issue after you informing me of the heavyweight vs flyweight executors.

KMOmarovich@gmail.com (JIRA)

unread,
Oct 21, 2019, 2:29:03 PM10/21/19
to jenkinsc...@googlegroups.com

Brandon Squizzato Not sure how your solution is supposed to work after the first build. Each subsequent build will create two milestones and pass them immediately. First build is the only one that will be aborted.

bsquizzato@gmail.com (JIRA)

unread,
Oct 21, 2019, 5:32:02 PM10/21/19
to jenkinsc...@googlegroups.com

Kamil Magomedov

Build #1 would define milestone(1)
Build #2 would define milestone(1) and milestone(2)
Build #3 would define milestone(2) and milestone(3)
Build #4 would define milestone(3) and milestone(4)

and so on ...

Every time a build runs, it is passing the highest milestone of the previous build. This causes the previous build to abort. From https://jenkins.io/doc/pipeline/steps/pipeline-milestone-step/ –

> The milestone step forces all builds to go through in order, so an older build will never be allowed pass a milestone (it is aborted) if a newer build already passed it. 

KMOmarovich@gmail.com (JIRA)

unread,
Oct 22, 2019, 2:39:06 AM10/22/19
to jenkinsc...@googlegroups.com

Brandon Squizzato ah, I apologize, I had the wrong idea about the way milestone works. I thought that for the older builds to be aborted the newest build has to pass at least one more milestone step. So three milestone steps for Build #3 and so on. 

KMOmarovich@gmail.com (JIRA)

unread,
Oct 22, 2019, 2:41:02 AM10/22/19
to jenkinsc...@googlegroups.com
Kamil Magomedov edited a comment on New Feature JENKINS-43353
[~bsquizz] ah, I apologize, I had the wrong idea about the way milestone works. I thought that for the older builds to be aborted the newest build has to pass at least one more milestone step. So three milestone steps for Build #3 and so on.   That's how I always used it. Good to know that there's the other way, thank you!

paul@nkey.com.br (JIRA)

unread,
Jan 23, 2020, 5:31:05 PM1/23/20
to jenkinsc...@googlegroups.com

How to use the milestone solution in a declarative pipeline ? Brandon Squizzato Do you have a sample Jenkinsfile doing this?
Reply all
Reply to author
Forward
0 new messages