TestNG behaviour when no methods are found

[Eran]

Hi,

Today, if TestNG is invoked and it doesn't find any tests to run, it marks the test run as a success (0/0/0).
Obviously, if you open the reports you could see that no test was executed but if you run TestNG automatically by Ant and rely on the failureProperty to perform some actions, that can lead to a problem.
As it's common to make mistakes like misspelling a group, people might get the false impression that all tests pass when in fact no test was executed. Therefore, I believe that such an "empty run" should be considered as a failure.

What does everybody think?

Regards,
Eran

[Alexandru Popescu]

IMO a failing test is completely different from an empty test class.
What we can probably do, but I am not yet convinced is possible, is to
signal somehow that nothing was run.

./alex
--
.w( the_mindstorm )p.

[Hani Suleiman]

An exception or error message should be shown, explaining what was
searched and why tests weren't found, for example:

WARN: No tests found in groups 'foo', 'bar', 'poo'

[Eran]

Printing an warning/error message is definitely a must but unfortunately it requires a human to read it...
Perhaps we can set the failure property to a special value?

Regards,
Eran

[Cédric Beust ♔]

Hi Eran,

This is a good point and I've been bitten by it a few times as well.  I think it makes sense to make an exception for this case and have TestNG exit with an error message and an error code.

--
Cedric

--
Cédric