counting tests before they run

[glen]

I'm integrating TestNG with a legacy test framework that was primarily
designed to test C and C++ code, but we're branching out into testing
java code now.

The legacy test framework counts the number of tests that are going to
be run, *before* the tests are actually executed. This is so that we
can get an accurate count of the number of tests that *should* have
run even if the C/C++ code core dumps on us 1/2 way through the test
run.

Is there any way to count the number of testng tests without actually
executing them?
Could we do a dry run of a testng suite somehow, maybe using a
listener to somehow skip over a test if we're in counting mode?

I looked at using an IMethodInterceptor to do this, however most of
are tests fall into the un-ordered category, so the interceptor
doesn't get executed for those tests (and I don't want to introduce an
artificial dependancy just for this feature)

[Thomas Fišer]

The idea I can think of off the top of my head is to run the tests
ordered, run them through a Method interceptor that counts them and
randomizes the order.

Hope that helps

-Doc

[Cédric Beust ♔]

Hi Glen,

Your interpretation is correct, there is currently no way of doing this. I agree it would be nice, especially for tools, such as the Eclipse plug-in. You will notice that when you run the tests with Eclipse, the total number of methods keeps increasing as TestNG discovers them (TestNG *does* know the number of <test> tags upfront, though, so that's a start).

I'm planning to add support for this at some point.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Jek]

Hello,

In one of our test systems, we use an IMethodSelector to do a 'fake'
run before we do a real one. Our IMethodSelector implementation
gathers all of the test class names along with the groups that appear
in the class. It tells TestNG not to run any of the methods. After
gathering classes and groups, this particular test system runs one
test class at a time for a certain group. We can then report how far
along we are in the number of classes we expect to run. You can also
use the same technique and gather test methods (instead of classes),
however you will not get the true number of methods if any of your
tests have data providers.

@Cedric: This would be a nice feature to have, however data providers
present a problem. I can imagine some cases where you will not want
to iterate through data providers to determine the number of tests it
will run. You could introduce a way for a DP to predict the number of
tests it will run. That should work for data providers that are
determinist.

-John

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

[Cédric Beust ♔]

John,

Absolutely, if I add support for this feature, I would definitely not invoke data providers since these might be one use only or they might have side effects that could interfere with the next run.

The number of methods would therefore be approximate, but it would definitely still be useful.

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Cédric Beust ♔]

Hi Glen,

No, this will work, but it will only give you the number of <test> tags, not the number of methods.

-- 

Cédric

On Tue, Nov 16, 2010 at 5:04 AM, glen <lockha...@gmail.com> wrote:

We've got pretty simple use cases for our tests, so we probably won't
be using too many advanced features.
All of out tests will be executed via an xml test suite so for now
this is what I've done to get this working.

I created a new class TestNGRunner that derives from TestNG

public class TestNGRunner extends TestNG {
   public int getTestCount() {
       int count = 0;
       for(XmlSuite suite : m_suites) {
           count += suite.getTests().size();
       }
       return count;
   }
}

Then in my framework I've got the following

public void execute (boolean counting) {
   TestNGRunner testng = new TestNGRunner();
   //....addListeners
   //....setTestSuites

   if(counting) {
       testng.initializeSuitesAndJarFile();
       return testng.getTestCount();
   }

   testng.run();
   return 0;
}


This probably has more holes in than swiss cheese, but for our usage
so far this seems to work.

Are there any obvious problems that I've overlooked here?

/Glen

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > --
> > > Cédric
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
> Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[glen]

Ok, so I've got a solution that works for me, others mileage may vary.

I derive a runner class from the TestNG class. To this I add a
getTestCount method.

You use TestNGRunner exactly as you would the TestNG class, so some
sample code would be as follows

public int execute(boolean counting) {

TestNGRunner testng = new TestNGRunner();

//addListeners(...);
//setXmlSuite(...);

if(counting) {

return testng.getTestCount();
}

testng.run();
return 0;
}

The implementation of TestNGRunner.getTestCount is as follows.

public class TestNGRunner extends TestNG {

public int getTestCount() {

initializeSuitesAndJarFile();
int count = 0;
for(XmlSuite xnlSuite : m_suites) {
for(XmlTest xmlTest : xnlSuite.getTests()) {
for(XmlClass xmlClass : xmlTest.getXmlClasses()) {
try {
@SuppressWarnings("rawtypes")
Class testClass =
getClass().getClassLoader().loadClass(xmlClass.getName());
for(Method testClassMethods :
testClass.getMethods()) {
for(Annotation annotation :
testClassMethods.getAnnotations()) {

if(annotation.annotationType().getCanonicalName().equals(org.testng.annotations.Test.class.getCanonicalName()))
{
count++;
}
}
}
} catch (ClassNotFoundException e) {
System.out.println(e);
//log the exception and swallow it, the actual
run of the test will expose this as an error.
//our counts will be off, but there's not much
we can do
}
}
}
}
return count;
}
}


This works for any of our existing test cases.
I'll probably need to refine this further to deal with disabled tests,
etc.

Have I missed any edge cases here that mean this won't work long term?



cheers,
Glen

On Nov 15, 7:05 pm, Cédric Beust ♔ <ced...@beust.com> wrote:

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

[Cédric Beust ♔]

Hi Glen,

It's a good first approximation. It will miss invocationCount (easy to fix by just doing a bit more introspection) and, as we discussed earlier, methods with data providers and tests created by @Factory, values which you probably should not try to get before the test actually runs.

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric