Today, if TestNG is invoked and it doesn't find any tests to run, it marks the test run as a success (0/0/0). Obviously, if you open the reports you could see that no test was executed but if you run TestNG automatically by Ant and rely on the failureProperty to perform some actions, that can lead to a problem. As it's common to make mistakes like misspelling a group, people might get the false impression that all tests pass when in fact no test was executed. Therefore, I believe that such an "empty run" should be considered as a failure.
IMO a failing test is completely different from an empty test class. What we can probably do, but I am not yet convinced is possible, is to signal somehow that nothing was run.
> Today, if TestNG is invoked and it doesn't find any tests to run, it marks > the test run as a success (0/0/0). > Obviously, if you open the reports you could see that no test was executed > but if you run TestNG automatically by Ant and rely on the failureProperty > to perform some actions, that can lead to a problem. > As it's common to make mistakes like misspelling a group, people might get > the false impression that all tests pass when in fact no test was executed. > Therefore, I believe that such an "empty run" should be considered as a > failure.
> IMO a failing test is completely different from an empty test class. > What we can probably do, but I am not yet convinced is possible, is to > signal somehow that nothing was run.
> ./alex > -- > .w( the_mindstorm )p.
> On 6/20/06, Eran <eran...@gmail.com> wrote: >> Hi,
>> Today, if TestNG is invoked and it doesn't find any tests to run, >> it marks >> the test run as a success (0/0/0). >> Obviously, if you open the reports you could see that no test was >> executed >> but if you run TestNG automatically by Ant and rely on the >> failureProperty >> to perform some actions, that can lead to a problem. >> As it's common to make mistakes like misspelling a group, people >> might get >> the false impression that all tests pass when in fact no test was >> executed. >> Therefore, I believe that such an "empty run" should be considered >> as a >> failure.
Printing an warning/error message is definitely a must but unfortunately it requires a human to read it... Perhaps we can set the failure property to a special value?
Regards, Eran
On 6/20/06, Hani Suleiman <h...@formicary.net> wrote:
> An exception or error message should be shown, explaining what was > searched and why tests weren't found, for example:
> WARN: No tests found in groups 'foo', 'bar', 'poo'
> On Jun 20, 2006, at 2:46 PM, Alexandru Popescu wrote:
> > IMO a failing test is completely different from an empty test class. > > What we can probably do, but I am not yet convinced is possible, is to > > signal somehow that nothing was run.
> > ./alex > > -- > > .w( the_mindstorm )p.
> > On 6/20/06, Eran <eran...@gmail.com> wrote: > >> Hi,
> >> Today, if TestNG is invoked and it doesn't find any tests to run, > >> it marks > >> the test run as a success (0/0/0). > >> Obviously, if you open the reports you could see that no test was > >> executed > >> but if you run TestNG automatically by Ant and rely on the > >> failureProperty > >> to perform some actions, that can lead to a problem. > >> As it's common to make mistakes like misspelling a group, people > >> might get > >> the false impression that all tests pass when in fact no test was > >> executed. > >> Therefore, I believe that such an "empty run" should be considered > >> as a > >> failure.
This is a good point and I've been bitten by it a few times as well. I think it makes sense to make an exception for this case and have TestNG exit with an error message and an error code.
-- Cedric
On 6/20/06, Alexandru Popescu <the.mindstorm.mailingl...@gmail.com> wrote:
> IMO a failing test is completely different from an empty test class. > What we can probably do, but I am not yet convinced is possible, is to > signal somehow that nothing was run.
> ./alex > -- > .w( the_mindstorm )p.
> On 6/20/06, Eran <eran...@gmail.com> wrote: > > Hi,
> > Today, if TestNG is invoked and it doesn't find any tests to run, it > marks > > the test run as a success (0/0/0). > > Obviously, if you open the reports you could see that no test was > executed > > but if you run TestNG automatically by Ant and rely on the > failureProperty > > to perform some actions, that can lead to a problem. > > As it's common to make mistakes like misspelling a group, people might > get > > the false impression that all tests pass when in fact no test was > executed. > > Therefore, I believe that such an "empty run" should be considered as a > > failure.
|
