Handling of failing data providers

[Tobias Riemenschneider]

Hi,

I am currently doing some research on the issue that Gradle silently ignores failing data providers when using TestNG (TestNG skips the tests in this case and skipped tests are silently ignored by Gradle). When looking for the place where this issue should (and could) be fixed I stumbled upon an inconsistency in the handling on data provider failures in TestNG (at least from a let's call it philosophical standpoint).

0. When a test method throws an exception that is not expected the test fails.
1. When a test method's data provider throws an exception the complete test method is skipped. (If tests are failing or being skipped in this case is a design decision and I am perfectly fine with both posibilites.)
2. When a data provider provides a set of parameters that does not fit the test method's signature the concrete invocation of the test method is failing due to an org.testng.internal.reflect.MethodMatcherException. That's a little bit strange since the test is actually not failing at all. It just can't be invoked with the given set of parameters. At least from my point of view this case is closer related to case 1 than to case 0 and should be handled similar.
3. When a data provider is not configured correctly, e.g., a data provider of that name cannot be found, the complete test method is also failing due to an org.testng.TestNGException. Again, from my point of view this is closer related to case 1 than to case 0 and should be handled similar.

What do you think about that?

Regards, Toby

[Tobias Riemenschneider]

I forgot to mention that I am using TestNG 6.9.13.6 (the latest and greatest, at least to my knowledge). I just did the same tests with 6.9.10 and realized that case 3 was handled by skipping the whole test method...

[Tobias Riemenschneider]

According to the project's CHANGES list, the changed behavior between 6.9.10 and 6.9.13.6 might be a side effect of fixing GITHUB-1181.

[Julien Herr]

TestNGException or sub-exception are expected when a user made an error (case2: wrong param match or case3: data provider missing) and he is supposed to fix it.

In the case of TestNGException, the run is not supposed to have results.

When the data provider run fails, then we cannot determine the result of the test(s) and that's why they are skipped.

[Tobias Riemenschneider]

OK, that's the way it is currently implemented. But what about the design. From my point of view a test either can be executed (data provider is there and provides valid parameters) and will then succeed or fail (case 1) or it cannot be executed and is therefore skipped (case 2). The current implementation splits case 2 into:
a. the concrete reason for not being able to execute the test is known and the test fails, or
b. the concrete reason is not known and the test is marked as being skipped.
In both cases the test was not run at all (it was skipped) but this fact is shown either as test failure or as skipped test (cases 1 and 2b are intermixed).

[Julien Herr]

Do you have any runnable samples for 2a and 2b? It will be more concrete.

And I think the best way to discuss will be with a pull-request where you'll show the behavior you expect in dedicated testng test(s) ;)