Possibility to retry tests until FAILED (not success status)

22 views

Skip to first unread message

Katya L

unread,

Aug 30, 2017, 3:35:33 PM8/30/17

to testng-dev

We need to have a possibility to run tests until test will be failed (because we have flaky tests and want to run a test until it failed and then investigate: read logs etc) Invocation count is not a workaround.

1st problem: hardcoded FAILURE test status in condition for retry:

boolean willRetry = retryAnalyzer != null && status == ITestResult.FAILURE && failure.instances != null && retryAnalyzer.retry(testResult);

In old TestNG versions we had a chance to avoid it by the following way:

class RetryUntilFailed : IRetryAnalyzer {

    companion object {
        private var retryCount = 0
        private var maxRetryCount = 1000    
    }
    
    override fun retry(result: ITestResult?): Boolean {
        if (result!!.getAttribute("RETRY") == "NO")
            return false
        else if (result.getAttribute("RETRY") == "YES" && retryCount < maxRetryCount) {
            retryCount++
            return true
        }
        return false
    }
}

class RetryUntilFailedListener : IInvokedMethodListener {
    override fun beforeInvocation(method: IInvokedMethod?, testResult: ITestResult?) {

    }

    override fun afterInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
        if (method!!.isTestMethod) {
            if (testResult!!.status == ITestResult.SUCCESS) {
                println("SUCCESS. NEED TO RETRY")
                testResult.status = ITestResult.FAILURE //because retry works only if (testResult.getStatus() == ITestResult.FAILURE)
                testResult.setAttribute("RETRY", "YES")
            } else if (testResult.status == ITestResult.FAILURE)
                testResult.setAttribute("RETRY", "NO")
        }
    }

}

But in the latest TestNG it doesn't work because handleInvocationResults() is invoked before afterInvocation() and no chance to override test status before retrying logic

finally {
  ...
  handleInvocationResults(tm, results, expectedExceptionClasses, failureContext);
  ...
  // Run invokedMethodListeners after updating TestResult
  runInvokedMethodListeners(AFTER_INVOCATION, invokedMethod, testResult);
  ...
}

Any ideas?

Thanks in advance

Julien Herr

unread,

Aug 30, 2017, 3:55:33 PM8/30/17

to testng-dev

Hi,

I'm not sure the regression is so recent. Could you tell us which is the last version working as you expect?