DataProvider not ran if class annotated @Test at class level

[Javier Ortiz Bultron]

I'm having a weird issue I can't figure out. It's driving me insane.

I have a Maven project for which I'm running the same command both locally and on Jenkins. The environment is the same as far as I can tell:

• Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)

• JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)

• TestNG 7.8.0

Hopefully, someone has an idea of what to do or what else I can do to try to figure out what's going on.

https://github.com/testng-team/testng/issues/2924

One of the tests that have the issue (only tests with DataProviders) show this pattern in the logs. Enabled verbose to level 10 in hopes that something would show up.

Locally:

[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest] [2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this [2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest

...

[2023-06-06 16:55:47.316] [INFO] ===== Test class com.XXX.framework.utils.TimeUtilsTest [2023-06-06 16:55:47.316] [INFO] @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@283eb984] [2023-06-06 16:55:47.316] [INFO] ======

...

[TestNG] INVOKING: "Surefire test" - com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s): NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))

... Test executes and results, etc.

On Jenkins:

[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest] [2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this [2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest

...

[2023-06-06 21:02:29.529] [INFO] ===== Test class com.XXX.framework.utils.TimeUtilsTest [2023-06-06 21:02:29.529] [INFO] @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620] [2023-06-06 21:02:29.529] [INFO] ======

I see no other references about the test class in the rest of the trace. Basicaly all tests with a DataProvider don't run.

Maybe someone has a suggestion on what else to look for.

I've tried the following:

• Making sure the right Test annotation is used (TestNG vs. JUnit

• Remove test groups or any other listener that could be interfering.

• Downgrade TestNG version.

• Pray.

[⇜Krishnan Mahadevan⇝]

Javier,

I have added a comment to the issue asking for more information.

While you create a sample project that can be used to reproduce the problem and attach it to the defect, please also do the following and include it as well.

1. Create a listener that looks like below.
2. Add the listener either in your suite file or via the service loader mechanism.
3. Sift through its output to see if anything stands out in that.

Java

import org.testng.IDataProviderListener;
import org.testng.IDataProviderMethod;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;

public class RunningCommentary implements IDataProviderListener, IInvokedMethodListener {

  @Override
  public void beforeDataProviderExecution(IDataProviderMethod dataProviderMethod,
      ITestNGMethod method, ITestContext iTestContext) {
    String methodName = dataProviderMethod.getMethod().getDeclaringClass().getName() +
        "." + dataProviderMethod.getMethod().getName();
    System.err.println(
        "Commencing to execute the data provider " + methodName
            + " for the test method " + method.getQualifiedName());
  }

  @Override
  public void afterDataProviderExecution(IDataProviderMethod dataProviderMethod,
      ITestNGMethod method, ITestContext iTestContext) {
    String methodName = dataProviderMethod.getMethod().getDeclaringClass().getName() +
        "." + dataProviderMethod.getMethod().getName();
    System.err.println(
        "Completed executing the data provider " + methodName
            + " for the test method " + method.getQualifiedName());
  }

  @Override
  public void onDataProviderFailure(ITestNGMethod method, ITestContext ctx, RuntimeException t) {
    String methodName = method.getDataProviderMethod().getMethod().getDeclaringClass().getName() +
        "." + method.getDataProviderMethod().getMethod().getName();

    System.err.println(methodName +
        " failed when trying to run it for " + method.getQualifiedName());
    System.err.println("Exception was ");
    t.printStackTrace();
  }

  @Override
  public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    String prefix = "Commencing executing the ";
    if (method.isTestMethod()) {
      prefix = " test method ";
    }
    if (method.isConfigurationMethod()) {
      prefix += " configuration method ";
    }
    System.err.println(prefix + method.getTestMethod().getQualifiedName());
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    String prefix = "Completed executing the ";
    if (method.isTestMethod()) {
      prefix = " test method";
    }
    if (method.isConfigurationMethod()) {
      prefix += " configuration method ";
    }
    System.err.println(prefix + method.getTestMethod().getQualifiedName());
    if (!testResult.isSuccess()) {
      System.err.println("Exception was ");
      testResult.getThrowable().printStackTrace();
    }
  }
}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-dev/69179bd1-ba36-45ca-ba42-0c6ade484170n%40googlegroups.com.