Can I have TestNG fail/skip if the DataProvider provides no test cases?

[Nick Flanders]

If my TestNG DataProvider has some logic in it, but then it results in an empty Object[][], I would like TestNG to count this as a failed test.

I would prefer to not have to put logic in the DataProvider which checks that the Object[][].length > 0 since my package has many of these types of DataProviders. 

Is it possible to have TestNG mark myTest as Failed or at least Skipped?

Ex. 

@DataProvider(name = "emptyDataProvider")
public Object[][] emptyDataProvider() {
    // Some misc logic...
    return new Object[][] { };
} 

@Test(groups = {"beta"}, dataProvider = "emptyDataProvider")
public void myTest(final String param1) {  

    // some assertions
}

[⇜Krishnan Mahadevan⇝]

Nick,

This should work fine as per expectations if you are trying using the latest released version of TestNG viz., 7.1.0.

You should see an exception that looks like below

org.testng.internal.reflect.MethodMatcherException: 

[public void com.rationaleemotions.TestNGDataProviderExample.testMethod(int)] has no parameters defined but was found to be using a data provider (either explicitly specified or inherited from class level annotation).

Data provider mismatch

Method: testMethod([Parameter{index=0, type=int, declaredAnnotations=[]}])

Arguments: []

And causes the test method to be skipped.

Alternatively you can also build a listener that implements org.testng.IDataProviderListener and within its afterDataProviderExecution inspect the org.testng.IDataProviderMethod parameter and throw exceptions accordingly as well

Since its a listener, you now have the option of auto wiring it via a META-INF/services/org.testng.ITestNGListener file and thus have it get invoked automatically without needing any additional changes.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/75f02409-dbba-4d69-ad79-1546c1d6ae6c%40googlegroups.com.

[Nick Flanders]

Thank you for the quick response Krishnan,

It seems that the Exception that you mentioned is only being thrown when there is a mismatch in the number of parameters provided to a test case by a DataProvider. I would like a test to fail if the DataProvider provides no test cases.

I have tried to provide a pastebin demonstrating the test cases that I am looking at. https://pastebin.com/cBULgXXz

I would like the "myTest_emptyDataProvider" test case to fail if possible. 

I looked into the possibility of using the DataProvider listener and interceptor (https://pastebin.com/tzQLA377) but this is just causing the test to be skipped rather rather than fail.

Thanks again for your help so far!

Nick

[Krishnan Mahadevan]

Nick,

 

You can try doing something like this ( The test method would still be marked as skipped only, because the data provider which is like the pre-requisite for the test method threw an exception )

 

import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(DPListener.class)
public class DPGames {

  @Test(dataProvider = "emptyDataProvider")
  public void myTest_emptyDataProvider(final String param1) {
        assertThat(param1, is(param1));
  }

  @DataProvider(name = "emptyDataProvider")
  public Object[][] emptyDataProvider() {
    return new Object[][] {
    };
  }
}

 

The listener would look like below:

 

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.testng.IDataProviderInterceptor;
import org.testng.IDataProviderMethod;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;

public class DPListener implements IDataProviderInterceptor, ITestListener {
  private List<String> emptyDataMethods = new ArrayList<>();

  @Override
  public void onTestSkipped(ITestResult result) {
    if (emptyDataMethods.contains(result.getMethod().getQualifiedName())) {
      ITestNGMethod m = result.getMethod();
      System.err.println(
          m.getQualifiedName()
              + " was skipped because the data provider "
              + getQualifiedName(m.getDataProviderMethod())
              + " returned no values");
    }
  }

  private String getQualifiedName(IDataProviderMethod m) {
    return m.getMethod().getDeclaringClass().getName() + "." + m.getMethod().getName();
  }

  @Override
  public Iterator<Object[]> intercept(
      Iterator<Object[]> original,
      IDataProviderMethod dataProviderMethod,
      ITestNGMethod method,
      ITestContext iTestContext) {
    if (original.hasNext()) {
      return original;
    }
    emptyDataMethods.add(method.getQualifiedName());
    throw new IllegalStateException("No test data provided by " + dataProviderMethod.getName());
  }
}

 

 

The execution output looks like below:

 

[Utils] [ERROR] [Error] java.lang.IllegalStateException: No test data provided by emptyDataProvider

    at com.rationaleemotions.googleforums.dataprovider.DPListener.intercept(DPListener.java:42)

    at org.testng.internal.Parameters.handleParameters(Parameters.java:858)

    at org.testng.internal.Parameters.handleParameters(Parameters.java:740)

    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:59)

    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:38)

    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:788)

    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)

    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)

    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)

    at java.util.ArrayList.forEach(ArrayList.java:1257)

    at org.testng.TestRunner.privateRun(TestRunner.java:767)

    at org.testng.TestRunner.run(TestRunner.java:588)

    at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)

    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)

    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)

    at org.testng.SuiteRunner.run(SuiteRunner.java:286)

    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)

    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)

    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1214)

    at org.testng.TestNG.runSuitesLocally(TestNG.java:1136)

    at org.testng.TestNG.runSuites(TestNG.java:1066)

    at org.testng.TestNG.run(TestNG.java:1034)

    at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)

    at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)

com.rationaleemotions.googleforums.dataprovider.DPGames.myTest_emptyDataProvider was skipped because the data provider com.rationaleemotions.googleforums.dataprovider.DPGames.emptyDataProvider returned no values

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.com/

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/6c240429-b8c6-4b36-bf00-73f13d32805d%40googlegroups.com.