[JIRA] (JENKINS-51454) Pipeline retry operation doesn't retry when there is a timeout inside of it

120 views
Skip to first unread message

me@basilcrow.com (JIRA)

unread,
May 29, 2019, 8:21:02 PM5/29/19
to jenkinsc...@googlegroups.com
Basil Crow updated an issue
 
Jenkins / Bug JENKINS-51454
Pipeline retry operation doesn't retry when there is a timeout inside of it
Change By: Basil Crow
Summary: Declarative pipeline Pipeline retry operation doesn't retry when there is a timeout inside of it
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

me@basilcrow.com (JIRA)

unread,
May 29, 2019, 8:21:03 PM5/29/19
to jenkinsc...@googlegroups.com
Basil Crow updated an issue
When a declarative {{timeout}} call is fired inside of a {{retry}}, *retry is not being triggered* and *job execution is aborted*, the only way to make it work is by surrounding the {{timeout}} operation with a {{try/catch}}.

h3. without try/catch
Log output
{code}
Cancelling nested steps due to timeout
{code}
Execution result
{code}
Timeout has been exceeded
Finished: ABORTED
{code}

h3. with try/catch
Log output
{code}
Timeout set to expire after 2 sec without activity
Sleeping for 2 sec
Cancelling nested steps due to timeout
ERROR: catched timeout! org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
Retrying
{code}
Execution result
{code}
Finished: SUCCESS
{code}
 
h2. Examples to reproduce the issue
h3. Failing example
{code:java}
node {
    def timeoutSeconds = 3
    stage('Preparation') { // for display purposes
        retry(3){
            timeout(activity: true, time: 2, unit: 'SECONDS') {
                sleep(timeoutSeconds)
            }
            timeoutSeconds--
        }
   }
}
{code}

h3. Working example
{code:java}
node {
    def timeoutSeconds = 3
    stage('Preparation') { // for display purposes
        retry(3){
            try{
                timeout(activity: true, time: 2, unit: 'SECONDS') {
                    sleep(timeoutSeconds)
                }
            }catch(err){
                timeoutSeconds--
                script{
                  def user = err.getCauses()[0].getUser()
                  error "[${user}] catched timeout! $err"
                }
            }
        }
   }
}
{code}

me@basilcrow.com (JIRA)

unread,
May 29, 2019, 8:22:02 PM5/29/19
to jenkinsc...@googlegroups.com
Basil Crow updated an issue
Change By: Basil Crow
Component/s: workflow-basic-steps-plugin
Component/s: pipeline-model-definition-plugin

me@basilcrow.com (JIRA)

unread,
May 29, 2019, 8:22:03 PM5/29/19
to jenkinsc...@googlegroups.com
Basil Crow assigned an issue to Unassigned
Change By: Basil Crow
Assignee: Andrew Bayer

me@basilcrow.com (JIRA)

unread,
May 29, 2019, 8:24:01 PM5/29/19
to jenkinsc...@googlegroups.com
Basil Crow commented on Bug JENKINS-51454
 
Re: Pipeline retry operation doesn't retry when there is a timeout inside of it

As a workaround, I was able to catch FlowInterruptedException and rethrow a more generic exception:

import org.jenkinsci.plugins.workflow.steps.FlowInterruptedException

retry(3) {
  try {
    timeout(time: 10, unit: 'MINUTES') {
      [..]
    }
  } catch (FlowInterruptedException e) {
   // Work around https://issues.jenkins-ci.org/browse/JENKINS-51454
    error 'Timeout has been exceeded'
  }
}

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 9, 2019, 2:40:04 PM12/9/19
to jenkinsc...@googlegroups.com

I think there are two desired behaviors depending on the placement of timeout and retry relative to each other.

First case (this ticket):

retry(3) {
  timeout(time: 5, unit: 'MINUTES') {
    // Something that can fail
  }
}

In this case, if the timeout triggers, I think the desired behavior is for retry to run its body again. This is not the current behavior.

Second case (not this ticket):

timeout(time: 5, unit: 'MINUTES') {
  retry(3) {
    // Something that can fail
  }
}

In this case, if the timeout triggers, I think the desired behavior is for retry to be aborted without retrying anything. This is the current behavior, and as far as I understand, is working as-designed after JENKINS-44379.

Switching timeout to use a different kind of exception would fix the first case, but break the second case. To support both use cases, something more complex would be needed (see PR 81 for a possible approach, although retry would need to be updated as well)). Basil Crow Noted that a fix for JENKINS-60354 might overlap a bit with this issue.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

pixman20@gmail.com (JIRA)

unread,
Mar 27, 2020, 6:03:07 PM3/27/20
to jenkinsc...@googlegroups.com
pixman20 commented on Bug JENKINS-51454

FWIW: I did a slight modification to the initial workaround and added it to our global shared libraries as retryEvenOnTimeout.
 
It's slightly more robust:

def call(Map retryOptions, Closure closure) {
    retry(retryOptions) {
        try {
            closure()
        } catch(org.jenkinsci.plugins.workflow.steps.FlowInterruptedException ex){
            def causes = ex.getCauses()
            if(causes.find { ! (it instanceof org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ExceededTimeout) }) {
                throw ex
            }
            // Wrap the real exception and throw a RuntimeException that'll trigger the retry
            throw new RuntimeException(ex)
        }
    }
}

Other potential solution options are to merge retry and timeout functionality to be able to allow the user to have the option of which functionality they need.
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

jrogers@socialserve.com (JIRA)

unread,
Apr 29, 2020, 7:53:04 AM4/29/20
to jenkinsc...@googlegroups.com

pixman20 Your retryEvenOnTimeout didn't quite work for me, but it did when I replaced "Map retryOptions" with "int retryCount":

def call(int retryCount, Closure closure) {
    retry(retryCount) {
        
try {
            closure()
        } catch(org.jenkinsci.plugins.workflow.steps.FlowInterruptedException ex){
            def causes = ex.getCauses()
            if(causes.find { ! (it instanceof org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ExceededTimeout) }) {
                throw ex
            }
            // Wrap the real exception and throw a RuntimeException that'll trigger the retry
            throw new RuntimeException(ex)
        }
    }
}
Reply all
Reply to author
Forward
0 new messages