New data is not passed from dataProvider to test on test retry

[Ekateryna Shyshkina]

TestNG version: 7.6.0, 7.7.1

Facing such issue and don't know how to get around it.

I am using dataProvider for passing objects as test parameters and Retry class (that extedds RetryAnalyzerCount) to retry failed tests.

All is working fine except: if some of object properties were changed during test before it fails, dataProvider method is running on retry but don't pass new object to test, so in test we have old set data from 1st run.

Example:

    @DataProvider(name = "userData")
    public static Object[] getUserData() {
        System.out.println("DataProvider is started...");
        TestUser testUser = new TestUser("John");
        System.out.println("Username from dataprovider: " + testUser.getName());
        return new Object[]{testUser};
    }

    @Test(retryAnalyzer = Retry.class, dataProvider = "userData")
    public void testRetry(TestUser user) {
        System.out.println("Test is started...");
        System.out.println("Username from test: " + user.getName());
        user.setName("New Name");
        Assert.fail();
    }

Output:

DataProvider is started...
Username from dataprovider: John
Test is started...
Username from test: John

Retry test...

DataProvider is started...
Username from dataprovider: John
Test is started...
Username from test: New Name

[⇜Krishnan Mahadevan⇝]

Thanks for uncovering a discrepancy with TestNG when it comes to data providers.

It looks like TestNG ends up invoking a data provider for a failed test that is about to be retried even though TestNG does not do anything with the data returned by the latest invocation of the data provider.

Here's a cleaned up sample of what you shared, which re-iterates my point. For the sake of completeness, I have logged a defect for this https://github.com/cbeust/testng/issues/2884 which hopefully should be available in the upcoming TestNG release (7.8.0 perhaps?)

Java

import com.rationaleemotions.DemoTest.Commentary;
import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(Commentary.class)
public class DemoTest {

  public static final AtomicInteger dataProviderInvocationCount = new AtomicInteger(0);

  @Test(dataProvider = "dp", retryAnalyzer = TryAgain.class)
  public void testMethod(Pojo user) {
    System.out.println("Test is started for data " + user.toString() + " with name " + user.getName());
    user.setName("Dragon-Warrior");
    Assert.fail();
  }

  @DataProvider(name = "dp")
  public static Object[][] getTestData() {
    System.err.println(
        "Data Provider invocation count #" + dataProviderInvocationCount.incrementAndGet());
    return new Object[][]{
        {new Pojo().setName("John")}
    };
  }

  public static class TryAgain implements IRetryAnalyzer {

    private int counter = 1;

    @Override
    public boolean retry(ITestResult result) {
      return counter++ != 3;
    }
  }

  public static class Pojo {

    private String name;

    public Pojo() {
      System.err.println("Instantiated " + this);
    }

    public String getName() {
      return name;
    }

    public Pojo setName(String name) {
      this.name = name;
      return this;
    }
  }

  public static class Commentary implements IExecutionListener {

    @Override
    public void onExecutionStart() {}

    @Override
    public void onExecutionFinish() {
      System.err.println(
          "Data Provider was invoked " + dataProviderInvocationCount.get() + " time(s).");
    }
  }
}

Here's the execution output. Notice the following :

• the print statement in the data provider is getting invoked multiple times, re-iterating the point that the data provider is being called multiple times (whereas it should not have been the case)
• The instance Pojo@2bfc268b that was created the first time the data provider was called, is what is being passed to the test method when a retry is happening and so the name that was changed by me in the test method is what is coming back when the test is getting retried (which is why John became Dragon-Warrior)
  

Bash

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Data Provider invocation count #1
Instantiated com.rationaleemotions.DemoTest$Pojo@2bfc268b
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name John

Test ignored.
Data Provider invocation count #2
Instantiated com.rationaleemotions.DemoTest$Pojo@2b91004a
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name Dragon-Warrior

Test ignored.
Data Provider invocation count #3
Instantiated com.rationaleemotions.DemoTest$Pojo@169e6180
Test is started for data com.rationaleemotions.DemoTest$Pojo@2bfc268b with name Dragon-Warrior

java.lang.AssertionError: null

    at org.testng.Assert.fail(Assert.java:110)
    at org.testng.Assert.fail(Assert.java:115)
    at com.rationaleemotions.DemoTest.testMethod(DemoTest.java:22)
    at java.base/

===============================================
Default Suite
Total tests run: 3, Passes: 0, Failures: 1, Skips: 0, Retries: 2
===============================================

Data Provider was invoked 3 time(s).

Process finished with exit code 0

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/dc1fdb47-e3fa-43a9-ac5a-e99f01df10dcn%40googlegroups.com.

[Ekateryna Shyshkina]

thank you for raising defect

пятница, 17 марта 2023 г. в 12:29:36 UTC+2, Krishnan Mahadevan: