Data provider inconsistency when applying retry testng 7.8

[Nithin Gopalakrishnan]

Hi Team.

I have a project where I have an annotation transformer, retry listener and test listener using testng 7.8, listeners are set via surefire properties.

It seems that for data provider returning custom object array, retry is not invoked when data provider is set to run parallel. In the console output, only 1 test is skipped (and retried) while the other two directly failed and went to the on failed listener event

Sample Code:

AnnotationTransformer

public class AnnotationTransformer implements IAnnotationTransformer {
@Override
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
if (Config.RETRY_FAILED_TESTS) {
System.out.println("Setting retry analyzer for "+testMethod.getName());
annotation.setRetryAnalyzer(RetryAnalyzer.class);
}
}

Retry Analyzer

public class RetryAnalyzer implements IRetryAnalyzer {
int counter = 0;
int retryLimit = Integer.parseInt(PropertiesFinder.getProperty("failedRetryCount"));

@Override
public boolean retry(ITestResult iTestResult) {
if (counter < retryLimit) {
counter++;
System.out.println("Retrying "+iTestResult.getName());
return true;
}
return false;
}
}

Test and Data Provider

@DataProvider(name = "testData",parallel = true)
public Object[][] getData(){
return new Object[][]{
{new SearchData("Test1", 1 ,"Test One")},
{new SearchData("Test2", 2, "Test Two")},
{new SearchData("Test3", 3, "Test Three")}
};
}

@Test(description = "To test data provider", dataProvider = "testData", groups = {"dataProviderTest","RegressionTest"})
public void API_dummyTest(SearchData searchData){
System.out.println("This is for "+searchData.name+" "+searchData.age);
int number= RandomUtils.nextInt(0,3);
Assert.verify("Verifying "+number,number).isEqualTo(searchData.age);
}

Test Listener

public class CoreListener implements ITestListener {
@Override
public void onTestFailure(ITestResult result) {
System.out.println("Inside test failure for "+result.getName());
}

@Override
public void onTestSkipped(ITestResult result) {
System.out.println("Inside test skipped for "+result.getName()); }

@Override
public void onTestStart(ITestResult result) {
System.out.println("Inside test start for "+result.getName()); }

@Override
public void onFinish(ITestContext context) {
System.out.println("Finished all");
}
}

Console Log

INFO] Running com.tester.tests.Tester
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite parallel="methods" data-provider-thread-count="5" name="tester Test Run">
  <test thread-count="5" parallel="methods" name="Default XmlTest name be91acef-0211-4a5a-bebe-1383d6b5ad4d">
    <groups>
      <run>
        <include name="dataProviderTest"/>
        <exclude name="ignore"/>
        <exclude name="wip"/>
      </run>
    </groups>
    <packages>
      <package name="com.tester.tests"/>
    </packages>
  </test> <!-- Default XmlTest name be91acef-0211-4a5a-bebe-1383d6b5ad4d -->
</suite> <!-- tester Test Run -->

Setting retry analyzer for testCartNavigation
Setting retry analyzer for testSearchNavigation
Setting retry analyzer for API_dummyTest
Inside test start for API_dummyTest
Inside test start for API_dummyTest
Inside test start for API_dummyTest
This is for Test1 1
This is for Test2 2
This is for Test3 3
Retrying API_dummyTest
Inside test skipped for API_dummyTest
Inside test start for API_dummyTest
This is for Test1 1
Inside test failure for API_dummyTest
Inside test failure for API_dummyTest
Finished all
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 1, Time elapsed: 1.919 s <<< FAILURE! -- in com.tester.tests.Tester
[ERROR] com.tester.tests.Tester.API_dummyTest[com.tester.tests.SearchData@2685c1b4](3) -- Time elapsed: 0.001 s <<< FAILURE!
java.lang.AssertionError:

Verification Failed:
Reason: Actual value '2' is not equal to comparison value '3'
Description: Verifying 2
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:28)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)

at com.tester.flow.assertions.stackbuilder.ErrorStack.runHardAssertion(ErrorStack.java:76)
at com.tester.flow.assertions.stackbuilder.ErrorStack.access$500(ErrorStack.java:12)
at com.tester.flow.assertions.stackbuilder.ErrorStack$Builder.build(ErrorStack.java:174)
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:30)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:664)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:227)
at org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:80)
at org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:17)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

[ERROR] com.tester.tests.Tester.API_dummyTest[com.tester.tests.SearchData@514e4c80](4) -- Time elapsed: 0.005 s <<< FAILURE!
java.lang.AssertionError:

Verification Failed:
Reason: Actual value '0' is not equal to comparison value '1'
Description: Verifying 0
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:28)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)

at com.tester.flow.assertions.stackbuilder.ErrorStack.runHardAssertion(ErrorStack.java:76)
at com.tester.flow.assertions.stackbuilder.ErrorStack.access$500(ErrorStack.java:12)
at com.tester.flow.assertions.stackbuilder.ErrorStack$Builder.build(ErrorStack.java:174)
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:30)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:664)
at org.testng.internal.invokers.TestInvoker.retryFailed(TestInvoker.java:248)
at org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:103)
at org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:17)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   Tester.API_dummyTest:50
Verification Failed:
Reason: Actual value '2' is not equal to comparison value '3'
Description: Verifying 2
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:28)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)

[ERROR]   Tester.API_dummyTest:50
Verification Failed:
Reason: Actual value '0' is not equal to comparison value '1'
Description: Verifying 0
at com.tester.flow.assertions.BaseAssertion.isEqualTo(BaseAssertion.java:28)
at com.tester.tests.Tester.API_dummyTest(Tester.java:50)

[INFO]
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 1
[INFO]
[ERROR] There are test failures.

Could someone please tell me if there are any errors in my implementation or its an issue 

Thanks,

Nithin

[⇜Krishnan Mahadevan⇝]

This looks like a bug. I have filed an issue on your behalf with a more succinct test case, that can be used to reproduce the problem.

https://github.com/testng-team/testng/issues/2934

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/220f6e10-5e33-430d-86f6-b1023352c180n%40googlegroups.com.

[Nithin Gopalakrishnan]

Thanks Krishnan Mahadevan.