Programmatic group creation?

[davetron5000]

I'm currently running tests that are defined by very simple XML
files. My @Test class takes a parameter filled by a @DataProvider,
which picks up all these XML files. The result is what I want, which
is one "test" per file.

However, I'd like to be able to group these files, possibly
dynamically, so that I can more easily run a subset of them while
debugging. This is being done through ant.

I can see a few options:

1 - Create an ant task to dynamically create testng.xml, with groups
as needed.
2 - run TestNG programmatically (this would be done with
suiteRunnerClass?)
3 - Ditch my xml file idea for something better

Before I go down any of these paths, just wondering what makes the
most sense, or if I'm going against the grain too much? My goal is to
drop in XML files for tests without having to write java code and
recompile to add new tests.

[Mark Derricutt]

Do you want to create dynamic groups as in TestNG test groups, or just dynamic groupings of tests?

A while ago as an experiment I wrote an EL requirements processor for dynamic test execution:

http://www.talios.com/writing_fine_grained_test_requirements_with_expressions.htm

which used an AnnotationProcessor to alter the group name of a test in order to fail it.  I suspect this is a far cry from what you want, but may give you some ideas for things to try...

Mark

--
"It is easier to optimize correct code than to correct optimized code." -- Bill Harlan

[Cédric Beust ♔]

On Tue, May 20, 2008 at 2:49 PM, davetron5000 <davetr...@gmail.com> wrote:

I'm currently running tests that are defined by very simple XML
files.  My @Test class takes a parameter filled by a @DataProvider,
which picks up all these XML files.  The result is what I want, which
is one "test" per file.

However, I'd like to be able to group these files, possibly
dynamically, so that I can more easily run a subset of them while
debugging.  This is being done through ant.

I can see a few options:

1 - Create an ant task to dynamically create testng.xml, with groups
as needed.

You can also create Xml objects directly instead of going through a generated XML file (look up the Javadoc for XmlSuite).

 

2 - run TestNG programmatically (this would be done with
suiteRunnerClass?)

No, just new TestNG(), set whatever you need on it, add a listener and run().

 

3 - Ditch my xml file idea for something better

Before I go down any of these paths, just wondering what makes the
most sense, or if I'm going against the grain too much?  My goal is to
drop in XML files for tests without having to write java code and
recompile to add new tests.

I'm not 100% clear what you are trying to do, can you give an example?

--
Cédric

[davetron5000]

What I'm doing:

Basically creating a REST API and the tests needed to make sure it's
working. Each test is very simple, basically:

URI
Parameters/Query String
Request Method
Expected Status Code
Expected Content Type
Expected Content

I have a class, RestTest, that can be deserialized from a simple XML
format containing the above information. I have another class which
basically has one @Test method taking a RestTest, with a data provider
pointing to another method that reads in whatever XML files it finds
in a specific directory, deserializes them into RestTest objects, and
sends them out as the Object[][] array.

This is all run from ant via the <testng> task.

Both options seem viable in terms of dynamically creating groups, so
my question is how to link up this programmatic means to my ant file.
It sounds like if I create an XmlSuite object, or instantiate TestNG
and configure it, my ant file will just kick off a Java program that
accomplishes this. That would be OK, but I'd ideally to use the ant
task, so that it can better accommodate future tests (tests that may
have nothing to do with my REST stuff). That's why I was curious
about suiteRunnerClass.

On May 20, 6:23 pm, Cédric Beust ♔ <cbe...@google.com> wrote:
> On Tue, May 20, 2008 at 2:49 PM, davetron5000 <davetron5...@gmail.com>

[Cédric Beust ♔]

The SuiteRunner class is not part of the public API, so I wouldn't recommend relying on it (as opposed to the TestNG class).

Based on your scenario, it looks like generating an XML file would be the best way to solve your problem since it will leave a trace on your hard drive if you ever need to investigate a test failure and that you will be able to use the regular TestNG ant task to run it.

The best way to generate this file is probably to create an XmlSuite object (and its children) and then call toXml() on it.

Hope this helps.

--
Cédric

[davetron5000]

Ok, so I'm finally looking into this. XMLSuite doesn't have a
setTests method, but has a getTests method. Is the idea for me to
subclass it to return whatever XMLTest objects are relevant for what
I'm doing? Further, it's not clear from the XMLTest class (or from
the DTD) how to set the group of a test.

Basically I have one test class that takes a file as a parameter. So,
say I decide I want the file name to determine the group, so if I have

foo_test1.xml
foo_test2.xml
bar_test3.xml
bar_test4.xml
baz_test5.xml

I want to have 5 total tests, with three groups: "foo", "bar", and
"baz".

To execute those tests, without groups, I would generate something
like:

<suite>
<test name="foo_test1">
<parameter name="file" value="foo_test1.xml" />
<classes><class name="package.for.my.TestClass" /></classes>
</test>
<!-- repeat for each of the other four files -->
</suite>

So, what woudl the testng.xml look like with the groups, and the, how
do I use the XmlTest to accomplish that?

Dave

On May 21, 10:55 am, Cédric Beust ♔ <cbe...@google.com> wrote:
> The SuiteRunner class is not part of the public API, so I wouldn't recommend
> relying on it (as opposed to the TestNG class).
>
> Based on your scenario, it looks like generating an XML file would be the
> best way to solve your problem since it will leave a trace on your hard
> drive if you ever need to investigate a test failure and that you will be
> able to use the regular TestNG ant task to run it.
>
> The best way to generate this file is probably to create an XmlSuite object
> (and its children) and then call toXml() on it.
>
> Hope this helps.
>
> --
> Cédric
>

> On Wed, May 21, 2008 at 7:36 AM, davetron5000 <davetron5...@gmail.com>

[Cédric Beust ♔]

On Tue, Jun 3, 2008 at 8:07 AM, davetron5000 <davetr...@gmail.com> wrote:

Ok, so I'm finally looking into this.  XMLSuite doesn't have a
setTests method, but has a getTests method.  Is the idea for me to
subclass it to return whatever XMLTest objects are relevant for what
I'm doing?

No, just create an XmlTest and pass the parent XmlSuite in the constructor.

 

 Further, it's not clear from the XMLTest class (or from
the DTD) how to set the group of a test.

XmlTest#setIncludedGroups.

Let me know if you have need more information.

--
 Cedric

 

--
Cédric

[davetron5000]

Question on setIncludedGroups.

Will that basically pt the XmlTest on which it's called into that
group, even if the test class makes no mention of it. e.g.

<suite>
<test>
<groups><run><include name="foo"></run></groups>
<classes><class name="my.TestClass" /></classes>
</test>
<test>
<groups><run><include name="bar"></run></groups>
<classes><class name="my.TestClass2" /></classes>
</test>
</suite>

public class TestClass {
public void runTest() {
// do the tests
}
}

public class TestClass2 {
public void runOtherTest() {
// do some other tests
}
}

If I tell ant to only run tests "bar", would it skip the first test
(TestClass) and only run TestClass2?


On Jun 3, 11:27 am, Cédric Beust ♔ <cbe...@google.com> wrote:

[Cédric Beust ♔]

I'm a bit confused by your message, but what will be run depends on the groups that the methods in class TestClass2 belong to.

--
Cedric

--
Cédric

[davetron5000]

OK, I guess what I want to do is to assign those methods to groups
dynamically, at test run time. In other words, when run with
parameter "foo", I want it to be in group "blah", but when run with
parameter "bar", I want it to be in group "quux". Essentially, I want
to be able to say "run group foo" and have it run the test with only
the parameters in the "foo" group, based on my programming logic.

It sounds like the only way to assign a test to a group is via the
annotations/doclets?

Dave

On Jun 3, 11:51 am, Cédric Beust ♔ <cbe...@google.com> wrote:
> I'm a bit confused by your message, but what will be run depends on the
> groups that the methods in class TestClass2 belong to.
>
> --
> Cedric
>

[Cédric Beust ♔]

On Tue, Jun 3, 2008 at 9:38 AM, davetron5000 <davetr...@gmail.com> wrote:

OK, I guess what I want to do is to assign those methods to groups
dynamically, at test run time.  In other words, when run with
parameter "foo", I want it to be in group "blah", but when run with
parameter "bar", I want it to be in group "quux".  Essentially, I want
to be able to say "run group foo" and have it run the test with only
the parameters in the "foo" group, based on my programming logic.

It sounds like the only way to assign a test to a group is via the
annotations/doclets?

Correct.  If you want to change these dynamically, take a look at IAnnotationTransformer.

--
Cédric