Test Suites, Ant, Surefire, and JunitReport
Dan Fabulich
I spent some time solving a thorny problem over the last couple of days;
I'm not 100% sure I like the solution to it, so I wanted to write a bit
about it to see what other people think.
The problem occurs in cases where one test suite incorporates tests from
multiple test classes. This is the normal use case for a JUnit TestSuite.
You'll typically write a JUnit suite like this:
Example 1
public class MyTestSuite extends TestSuite {
public static Test suite() {
MyTestSuite suite = new MyTestSuite();
suite.addTestSuite(FooTestCase.class);
suite.addTest(BarTestSuite.suite());
return suite;
}
}
(In TestNG, suites are defined using XML files, and are considerably more
common in TestNG-land.)
The question I'm trying to answer is: how should the results of running a
TestSuite be represented in XML, and how should that XML be represented in
HTML? You might think this would be a solved problem, but it doesn't
appear to be. :-( Bugs SUREFIRE-433 and TESTNG-204 all relate to this, as
well as Ant bug 24106.
http://jira.codehaus.org/browse/SUREFIRE-433
http://jira.opensymphony.com/browse/TESTNG-204
http://issues.apache.org/bugzilla/show_bug.cgi?id=24106
First, it seems like most people don't even use test suites, preferring to
simply run every test in their src/test/java directory. If you do this,
you get a bunch of XML files, one file per TestCase class, like this:
Example 2
TEST-com.mycompany.Test1.xml
TEST-com.mycompany.Test2.xml
TEST-com.mycompany.Test3.xml
The individual XML files look like this:
Example 3
<testsuite name="com.mycompany.Test1" failures="1" errors="0" tests="2" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0">
<failure type="java.lang.AssertionError" message="I don't like this movie!">java.lang.AssertionError: I don't like this movie!
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.fail(Assert.java:91)
at com.mycompany.Test1.test1b(Test1.java:14)
</failure>
</testcase>
</testsuite>
That's all fine, but what if you want to use a test suite? When you do
that in Ant or in Maven today, you get just one XML file:
Example 4
TEST-com.mycompany.MyTestSuite.xml
In Surefire 2.4, the XML unfortunately looks like this:
Example 5
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0" />
<testcase name="test2a" time="0.0" />
<testcase name="test2b" time="0.0" />
<testcase name="test3a" time="0.0" />
<testcase name="test3b" time="0.0" />
</testsuite>
Note that information has been lost here, and the information that *is*
provided is misleading. Specifically, we lost the fact that test1a was a
method of com.mycompany.Test1; all we know is its method name.
Furthermore, the report misleadingly seems to suggest that test1a is a
method of MyTestSuite, whereas in fact MyTestSuite just has a suite()
method and no test methods of its own.
Most of the JUnit reporting tools, including Ant's JunitReport task and
Maven's surefire-report plugin (but also including Hudson and others)
incorrectly assume that all of those testcases are methods of MyTestSuite.
As a result, it makes it look like all of your tests are methods of one
class in one package. If you've got two "testOutput" methods, they'll be
indistinguishable (except by stacktrace, if one is present).
The same thing happens when you run TestNG tests: it always generates all
of the test results in one XML file. (Though it contains more
information, see below.)
More generally, both Ant's JunitReport and surefire-report incorrectly
assume a one-to-one relationship between TestCase classes, XML files, and
"suites" of tests. They assume that every XML file contains one suite of
tests, which is the same thing as a TestCase class, and all of the tests
in the suite are just methods of the TestCase class. This assumption is
wrong.
Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
classes to include the class name of the class on every <testcase>
element:
Example 7
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
<testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
</testsuite>
This is good, because it allows you to figure out which class the method
really belongs to. This is what TestNG outputs when it generates
JUnit-like results. Unfortunately, nobody honors the "classname"
attribute, including Ant 1.7's JunitReport task!
Even if we did try to honor that information and report on it in the HTML,
how would we do it? Right now, the reports are organized in terms of
packages, and within the packages you'll get a list of classes (assumed
incorrectly to be the same thing as "suites"), and within the list of
classes you'll get a list of test method names. There's no room for
"suites" (as distinct from classes) in these reports at all.
And arguably there shouldn't be a place for this information, because
apparently 80-90% of the time suites aren't being used; adding a "suites"
section would be redundant in those cases.
We could throw away the "suite" wrapper and pretend as if the tests were
just classes and methods, but note something else: the sum of the "time"
attribute on the <testcase> elements is normally different from the "time"
attribute on the <testsuite>. That's because the timer has a 16ms
resolution on most operating systems. By starting the timer at the
beginning of the entire suite, we can capture overall time that we lose
when we turn the timer on and off. (And what if the tests had been run in
parallel?)
In bug SUREFIRE-433, it was argued that we should generate separate JUnit
XML files for every class. You could even imagine converting the results
from Example 7 into results that looked like Example 2, throwing away all
of the suite-level information.
I certainly want to be conservative in what I emit and liberal in what I
accept, but I *really* don't want to throw away information, especially at
the XML level. With that said, as we're trying to make the report look
something like the way it looks today, we're going to lose suite-level
information in the HTML.
What I propose is that, in order to avoid destroying information, Surefire
should generate XML that looks like Example 7 (all-in-one-file), and not
try to fake it to look like Example 2 (one-file-per-class). (TestNG's
junit-like output also generates files like Example 7.) However, when it
comes time to generate an HTML report, surefire-reports will discard
suite-level information, and treat large suites like Example 7 as if they
had been presented in separate files like Example 2. I'd argue that all
of the other JunitReport-like tools (including Ant) should probably follow
the same lead.
My proposal is not obviously right, because, again, most other JUnit
report tools also have this reporting bug; when they try to format TestNG
output or Surefire output, they'll incorrectly report all methods to be
members of the suite class. Maybe since we expect EVERYONE to have this
bug, we should be even more conservative in what we emit and generate
multi-file output, just so the other tools will know how to interpret it
correctly. That would make it more likely that the HTML output would
contain complete information.
What do other people think? Agree, disagree?
-Dan
Mike
file generated by the Surefire plugin, but without the classes that
the tests came from, the information in that xml file is pretty
meaningless. Your solution would work for me. Have you implemented the
change yet anywhere or are you just proposing something? I really need
a fix to this issue.
On Jan 24, 2:12 pm, Dan Fabulich <d...@fabulich.com> wrote:
> I spent some time solving a thorny problem over the last couple of days;
> I'm not 100% sure I like the solution to it, so I wanted to write a bit
> about it to see what other people think.
>
> The problem occurs in cases where one test suite incorporates tests from
> multiple test classes. This is the normal use case for a JUnit TestSuite.
> You'll typically write a JUnit suite like this:
>
> Example 1
> public class MyTestSuite extends TestSuite {
> public static Test suite() {
> MyTestSuite suite = new MyTestSuite();
> suite.addTestSuite(FooTestCase.class);
> suite.addTest(BarTestSuite.suite());
> return suite;
> }
> }
>
> (In TestNG, suites are defined using XML files, and are considerably more
> common in TestNG-land.)
>
> The question I'm trying to answer is: how should the results of running a
> TestSuite be represented in XML, and how should that XML be represented in
> HTML? You might think this would be a solved problem, but it doesn't
> appear to be. :-( Bugs SUREFIRE-433 and TESTNG-204 all relate to this, as
> well as Ant bug 24106.
>
Dan Fabulich
> Have you implemented the change yet anywhere or are you just proposing
> something? I really need a fix to this issue.
I'm coding it now, but I took a break to explain myself. :-)
-Dan
Steve Loughran
>
> Most of the JUnit reporting tools, including Ant's JunitReport task and
> Maven's surefire-report plugin (but also including Hudson and others)
> incorrectly assume that all of those testcases are methods of MyTestSuite.
> As a result, it makes it look like all of your tests are methods of one
> class in one package. If you've got two "testOutput" methods, they'll be
> indistinguishable (except by stacktrace, if one is present).
>
> The same thing happens when you run TestNG tests: it always generates all
> of the test results in one XML file. (Though it contains more
> information, see below.)
>
> More generally, both Ant's JunitReport and surefire-report incorrectly
> assume a one-to-one relationship between TestCase classes, XML files, and
> "suites" of tests. They assume that every XML file contains one suite of
> tests, which is the same thing as a TestCase class, and all of the tests
> in the suite are just methods of the TestCase class. This assumption is
> wrong.
I guess the reason for the assumption is that nobody normally writes a
test suite for JUnit, as it is needless coding that a pattern like
**/*Test.class could do at the build file level, where you can also
turn different tests on and off. But as you point out, it doesnt hold
for other test systems.
> Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
> classes to include the class name of the class on every <testcase>
> element:
>
> Example 7
> <testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4" time="0.031">
> <properties><!-- ... --></properties
> <testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
> <testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
> <testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
> <testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
> <testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
> <testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
> </testsuite>
I forget when, but take your word for it.
> This is good, because it allows you to figure out which class the method
> really belongs to. This is what TestNG outputs when it generates
> JUnit-like results. Unfortunately, nobody honors the "classname"
> attribute, including Ant 1.7's JunitReport task!
You contributions to the XSL file are welcome :)
>
> Even if we did try to honor that information and report on it in the HTML,
> how would we do it? Right now, the reports are organized in terms of
> packages, and within the packages you'll get a list of classes (assumed
> incorrectly to be the same thing as "suites"), and within the list of
> classes you'll get a list of test method names. There's no room for
> "suites" (as distinct from classes) in these reports at all.
>
> And arguably there shouldn't be a place for this information, because
> apparently 80-90% of the time suites aren't being used; adding a "suites"
> section would be redundant in those cases.
>
> We could throw away the "suite" wrapper and pretend as if the tests were
> just classes and methods, but note something else: the sum of the "time"
> attribute on the <testcase> elements is normally different from the "time"
> attribute on the <testsuite>. That's because the timer has a 16ms
> resolution on most operating systems.
>
> In bug SUREFIRE-433, it was argued that we should generate separate JUnit
> XML files for every class. You could even imagine converting the results
> from Example 7 into results that looked like Example 2, throwing away all
> of the suite-level information.
>
> I certainly want to be conservative in what I emit and liberal in what I
> accept, but I *really* don't want to throw away information, especially at
> the XML level. With that said, as we're trying to make the report look
> something like the way it looks today, we're going to lose suite-level
> information in the HTML.
>
> What I propose is that, in order to avoid destroying information, Surefire
> should generate XML that looks like Example 7 (all-in-one-file), and not
> try to fake it to look like Example 2 (one-file-per-class). (TestNG's
> junit-like output also generates files like Example 7.) However, when it
> comes time to generate an HTML report, surefire-reports will discard
> suite-level information, and treat large suites like Example 7 as if they
> had been presented in separate files like Example 2. I'd argue that all
> of the other JunitReport-like tools (including Ant) should probably follow
> the same lead.
I think that if everyone else has a bug, its hard to call it a bug.
More a presentation choice :)
> My proposal is not obviously right, because, again, most other JUnit
> report tools also have this reporting bug; when they try to format TestNG
> output or Surefire output, they'll incorrectly report all methods to be
> members of the suite class. Maybe since we expect EVERYONE to have this
> bug, we should be even more conservative in what we emit and generate
> multi-file output, just so the other tools will know how to interpret it
> correctly. That would make it more likely that the HTML output would
> contain complete information.
>
> What do other people think? Agree, disagree?
I do need to allocate time to fix reporting in Ant, both what we
generate (assuming a junit4 task), and what the tool processes. Me,
I'm more concerned with the problem of trying to merge log results
from a server and a client, as the current reports only pick the
stdout and stderr from one side of the party,
-steve
Dan Fabulich
>> What I propose is that, in order to avoid destroying information, Surefire
>> should generate XML that looks like Example 7 (all-in-one-file), and not
>> try to fake it to look like Example 2 (one-file-per-class). (TestNG's
>> junit-like output also generates files like Example 7.) However, when it
>> comes time to generate an HTML report, surefire-reports will discard
>> suite-level information, and treat large suites like Example 7 as if they
>> had been presented in separate files like Example 2. I'd argue that all
>> of the other JunitReport-like tools (including Ant) should probably follow
>> the same lead.
>
> I think that if everyone else has a bug, its hard to call it a bug. More
> a presentation choice :)
I'd thought somebody might say that! :-) Still, do you think TestNG and
other tools should therefore generate multiple XML files, to be compatible
with the other (arguably broken) reporting tools?
-Dan
Steve Loughran
No,
* junit report will take other sets of XSL sheets than those built in
to Ant's own JAR. This allows anyone to fix the XSLs without waiting
for ant's (fairly mature) release cycle.
* CI tools are a separate issue. Don't know there
Looking at the reports -and trying to imagine a time where I ever get
to stop making meta-RPM-RPMs (don't ask), I do want to push the test
formats forward in a way everyone is happy. I really dont like the way
the current stuff sticks summary info as toplevel attributes (time,
results), as it stops us streaming out the XML as we go. I'm starting
to wonder if we couldnt move to a new format, and have the option of
XSL generation of the legacy stuff, purely for those tools that havent
been upgraded in sync.
-steve