[JIRA] (JENKINS-46354) Retry in Declarative options skips retried stages due to earlier failure

1 view
Skip to first unread message

vivek.pandey@gmail.com (JIRA)

unread,
Feb 27, 2018, 12:20:02 AM2/27/18
to jenkinsc...@googlegroups.com
Vivek Pandey updated an issue
 
Jenkins / Bug JENKINS-46354
Retry in Declarative options skips retried stages due to earlier failure
Change By: Vivek Pandey
Sprint: Declarative - 1.2, Pipeline - December March 2018
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

delgod@delgod.com (JIRA)

unread,
Jun 11, 2018, 10:20:02 AM6/11/18
to jenkinsc...@googlegroups.com

bcampolo@bantamcity.com (JIRA)

unread,
Oct 11, 2018, 8:43:02 AM10/11/18
to jenkinsc...@googlegroups.com

Andrew Bayer - can you provide an updated status on this issue?  The priority is marked as Major and it has been open for over a year.  The only PR looks like it was closed.  This bug completely breaks the pipeline retry option for declarative pipelines.  I've tried various workarounds including setting the currentBuild.result without success.  Any help would be appreciated.
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

bcampolo@bantamcity.com (JIRA)

unread,
Oct 11, 2018, 8:45:03 AM10/11/18
to jenkinsc...@googlegroups.com
Bruno Campolo edited a comment on Bug JENKINS-46354
[~abayer] - can you provide an updated status on this issue?  The priority is marked as Major and it has been open for over a year.  The only PR looks like it was closed.  This bug completely breaks the pipeline retry option for declarative pipelines.  I've tried various workarounds including setting the currentBuild.result without success.  Any help would be appreciated. [~jglick]

andrew.bayer@gmail.com (JIRA)

unread,
Oct 11, 2018, 8:46:03 AM10/11/18
to jenkinsc...@googlegroups.com

I'll try to make another run at this soon.

jglick@cloudbees.com (JIRA)

unread,
Oct 11, 2018, 10:03:02 AM10/11/18
to jenkinsc...@googlegroups.com

Not sure I understand the original use case

to make sure the stage graph is consistent

huh? But it seems to me that if you use script blocks (especially to muck around with the “declarative” logic in such a fundamental way), all bets are off.

bcampolo@bantamcity.com (JIRA)

unread,
Oct 12, 2018, 8:18:01 AM10/12/18
to jenkinsc...@googlegroups.com

Jesse Glick - I'm not sure what "to make sure the stage graph is consistent" means either, but the heart of this issue is that when adding the retry option to a declarative pipeline options block, after the first pass fails, it correctly starts back at the top of the pipeline, but then incorrectly skips every job "due to earlier failure(s)", which means options { retry(3) } in the declarative pipeline is unusable.

jglick@cloudbees.com (JIRA)

unread,
Oct 12, 2018, 11:09:01 AM10/12/18
to jenkinsc...@googlegroups.com

So…is there a test case that does not involve script? That is really all I was getting at.

bcampolo@bantamcity.com (JIRA)

unread,
Oct 12, 2018, 2:44:03 PM10/12/18
to jenkinsc...@googlegroups.com
Bruno Campolo updated an issue
 
Change By: Bruno Campolo
Comment: {code:java}
// code placeholder
{code}
import jenkins.model.*pipeline \{ agent any options { retry(3) } stages \{ stage('Stage1') { steps { echo "Starting Stage1" } } stage('Stage2') \{ steps { echo "Starting Stage2" error "This will cause the pipeline to retry" } } } post \{ always { echo 'This will always run' } success \{ echo 'This will run only if successful' } failure \{ echo 'This will run upon failure' } unstable \{ echo 'This will run only if the run was marked as unstable' } changed \{ echo 'This will run only if the state of the Pipeline has changed' } } }

bcampolo@bantamcity.com (JIRA)

unread,
Oct 12, 2018, 2:44:03 PM10/12/18
to jenkinsc...@googlegroups.com

import jenkins.model.*pipeline { agent any options

{ retry(3) }

stages { stage('Stage1') { steps

{ echo "Starting Stage1" }

} stage('Stage2') { steps

{ echo "Starting Stage2" error "This will cause the pipeline to retry" }

} } post { always

{ echo 'This will always run' }

success { echo 'This will run only if successful' } failure { echo 'This will run upon failure' } unstable { echo 'This will run only if the run was marked as unstable' } changed { echo 'This will run only if the state of the Pipeline has changed' } } }

bcampolo@bantamcity.com (JIRA)

unread,
Oct 12, 2018, 2:45:02 PM10/12/18
to jenkinsc...@googlegroups.com
import jenkins.model.*

pipeline {
    agent any
	options {
		retry(3)
	}
    stages {
        stage('Stage1') {
            steps {
                echo "Starting Stage1"

bcampolo@bantamcity.com (JIRA)

unread,
Oct 12, 2018, 5:28:02 PM10/12/18
to jenkinsc...@googlegroups.com
Bruno Campolo edited a comment on Bug JENKINS-46354
{code:java}

import jenkins.model.*

pipeline {
    agent any
options {
  retry(3)
}
    stages {
        stage('Stage1') {
            steps {
                echo "Starting Stage1"
            }
        }
        stage('Stage2') {
            steps {
                echo "Starting Stage2"
                error "This will cause the pipeline to retry"
            }
        }
    }
    post {
        always {
            echo 'This will always run'
        }
        success {
            echo 'This will run only if successful'
        }
        failure {
            echo 'This will run upon failure'
        }
        unstable {
            echo 'This will run only if the run was marked as unstable'
        }
        changed {
            echo 'This will run only if the state of the Pipeline has changed'
        }
    }
}
{code}

andrew.bayer@gmail.com (JIRA)

unread,
Oct 15, 2018, 5:58:03 AM10/15/18
to jenkinsc...@googlegroups.com
Andrew Bayer started work on Bug JENKINS-46354
 
Change By: Andrew Bayer
Status: Open In Progress

andrew.bayer@gmail.com (JIRA)

unread,
Oct 15, 2018, 6:17:01 AM10/15/18
to jenkinsc...@googlegroups.com

I've started playing around with this again, and it's more complicated than I'd like. Ensuring that the stages are executed after retry is actually pretty easy, but the problem is that if a stage errors out the first time through, the build as a whole will still be marked as a failure, because we set the build to failed as soon as we hit an error, and there's no way to undo that once we've set it.

I have some ideas on how I can unravel this, but they'll require looking into everything that could be using currentBuild.result within Declarative and figuring out how we can cheat around that, or writing our own Declarative-specific retry logic that can avoid setting currentBuild.result when we're in a retry until we've hit the max attempts. So...yeah. More thought needed.

atharva.inamdar07@gmail.com (JIRA)

unread,
Feb 11, 2019, 6:18:01 AM2/11/19
to jenkinsc...@googlegroups.com

Is there any update on this issue? Or a workaround until this is fixed?

michal.matyjek@gmail.com (JIRA)

unread,
Feb 15, 2019, 12:53:02 AM2/15/19
to jenkinsc...@googlegroups.com

If I read it correctly - then this essentially means retry doesn't really work? If it does not retry the failed stages?

michal.matyjek@gmail.com (JIRA)

unread,
Feb 15, 2019, 1:02:04 AM2/15/19
to jenkinsc...@googlegroups.com

I have this pipeline:

pipeline {
    agent any

    options {
        timestamps()
        buildDiscarder(logRotator(daysToKeepStr:'7'))
        disableConcurrentBuilds()
        retry(3)
    }

    stages {
        stage('reset demo') {
            steps {
                sh 'exit 0'
            }
        }
        stage('Run tests') {
            steps {
                sh 'if (( RANDOM % 2 )); then exit 0; else exit 1; fi;'
            }
        }
    }

    post {
        failure {
            slackSend color: "danger", channel: "slack", message: "FAILED"
        }
        success {
            slackSend color: "good", channel: "slack", message: "PASS"
            cleanWs()
        }
        unstable {
            slackSend color: "warning", channel: "slack", message: "UNSTABLE"
        }
        unsuccessful {
            cleanWs()
        }
    }
}

and seeing this 2 times (hence my comment above that retry doesn't work):
ERROR: script returned exit code 1
Retrying
[Pipeline] {
[Pipeline] stage
[Pipeline]

{ (reset demo) Stage "reset demo" skipped due to earlier failure(s) [Pipeline] }

[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run tests)
Stage "Run tests" skipped due to earlier failure(s)

My understanding of retry is that both stages should be rerun....

Also a side note - I am getting post-failure step ran on each retry... which is unexpected - I would expect it to run once only when entire pipeline (including retries) is done....

atharva.inamdar07@gmail.com (JIRA)

unread,
Mar 25, 2019, 4:31:03 AM3/25/19
to jenkinsc...@googlegroups.com

Andrew, thanks for working on this. Is there a workaround available until this issue is fixed? Currently, we have to manually restart the build which isn't great.

andrew.bayer@gmail.com (JIRA)

unread,
Apr 12, 2019, 2:29:04 PM4/12/19
to jenkinsc...@googlegroups.com
Change By: Andrew Bayer
Status: In Progress Review

andrew.bayer@gmail.com (JIRA)

unread,
Apr 12, 2019, 2:30:02 PM4/12/19
to jenkinsc...@googlegroups.com

andrew.bayer@gmail.com (JIRA)

unread,
Apr 15, 2019, 11:17:03 AM4/15/19
to jenkinsc...@googlegroups.com
 

Releasing as Declarative 1.3.8.
Change By: Andrew Bayer
Status: In Review Resolved
Resolution: Fixed

bitwiseman@gmail.com (JIRA)

unread,
Oct 22, 2019, 11:25:46 PM10/22/19
to jenkinsc...@googlegroups.com
Liam Newman closed an issue as Fixed
 

Bulk closing resolved issues.

Change By: Liam Newman
Status: Resolved Closed
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages