Removing failed tests from within an IRetryAnalyzer

[jpi...@live.com.au]

Hi. What I am trying to do is to create a RetryAnalyser class that will retry failed test methods two more times. However, if one of the attempts is successful, I want the test suite to pass. The only way I can think of to achieve this is to remove the failed test attempts from the TestContext.

My RetryAnalyser is below:

import org.testng.IRetryAnalyzer;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.Reporter;

public class RetryAnalyser
implements IRetryAnalyzer
{
    private static final int MAX_COUNT = 2;
    private int myCount = 0;

    @Override
    public boolean retry( final ITestResult result )
    {
        if ( !result.isSuccess() )
        {
            if ( myCount < MAX_COUNT )
            {
                ++myCount;

                result.getTestContext().getFailedTests().removeResult( result.getMethod() );

                final String message =
                    Thread.currentThread().getName() + ": Error in " +
                        result.getName() + " with status " + result.getStatus() +
                        " Retrying " + myCount + " times";

                System.out.println( message );
                Reporter.log( message );

                return true;
            }
        }

        return false;
    }
}

The problem I seem to be having is that after I remove the method from the failed tests on the last attempt it is being reinserted. i.e. If the test method fails 3 times, I expect the suite to only report one failure (the last attempt) but instead it is reporting 2 failures? Can somebody please explain this, or show me a better way to achieve what I'm trying to do?

Cheers,
Jake.

[Sabaat Ahmad]

I am using TestNG 6.8.7 which has a method

result.getTestContext().getFailedTests().removeResult( result );

This successfully remove the previous result if I invoke the retry() method once. But if the test fails a second time, the result.getTestContext().getFailedTests().size() is somehow 2 (not 1). In addition to this I always get a failed + pass result in the report. If I retry only once, I get only a pass result.

This seems odd. Has anybody else faced this issue?

[Sabaat Ahmad]

the reason I don't want to remove duplicates in onFinish() method of the listener is that if the test have dependencies, then the dependent methods always get skipped as for them the test had failed. Has someone found a perfect solution to removing duplicates?

[Raju]

Hi Ahmad,

Please try the below code:

public void onFinish(ITestContext context) {
    Set<ITestResult> failedTests = context.getFailedTests().getAllResults();
    for (ITestResult temp : failedTests) {
        ITestNGMethod method = temp.getMethod();
        if (context.getFailedTests().getResults(method).size() > 1) {
            failedTests.remove(temp);
        } else {
            if (context.getPassedTests().getResults(method).size() > 0) {
                failedTests.remove(temp);
            }
        }
    }

}

For more details try to check in TestNG Selenium Tutorials

On Thursday, October 17, 2013 5:46:58 AM UTC+5:30, jpi...@live.com.au wrote:

[Manas Gupta]

Hi Sabaat,

I'm facing the same problem but not able to resolve it yet even after trying that solution to remove failedTest in onFinishMethod(). Do you have any solution for this. Thanks in advance.

[Scott Babcock]

When you "remove" a retried test from the results collection, it doesn't actually get removed from the collection. It's still there - its result just gets marked as SKIPPED. You can differentiate these "removed" failures from explicitly skipped tests by looking at the "thrown" property of the test result. In the case of explicitly skipped tests, the "thrown" property contain SkipException; for "removed" results, the "thrown" property contains the exception that caused the failure.