Problems with surefire reports, missing information

[Henrik Lynggaard]

Hi

I am testing testng with maven2 and I have run into a problem.

In the surefire report there is missing some information, both the
package list and class list is wrong.

The setup is ass following. I have created small project with two test
packages, each with one test class, which in turn each has 2 test
methods one that fails and one that succeeds.

The in the surefire report the package list contains 1 row with blank
package name claiming 4 run tests and 2 failures.

The Class list also only contains 1 row, class name TestSuite and
claiming 4 run tests and 2 failures

Both lists should contain 2 rows with 2 runs and 1 failure in each.

plugin versions:
testng 5.7
surefire-report: 2.4
[DEBUG]Adding managed dependencies for unknown:surefire-testng
[DEBUG] org.apache.maven.surefire:surefire-api:jar:2.4
[DEBUG] org.apache.maven.surefire:surefire-booter:jar:2.4
[DEBUG] org.codehaus.plexus:plexus-utils:jar:1.4.9
[DEBUG] org.apache.maven.surefire:surefire-testng:jar:2.4:test
(selected for test)

Am I doing something wrong oor have I found a bug ?

Best regards
Henrik

[Andreas Guther]

The information you have provided is not sufficient to make an assessment.  What about running the tests with 'mvn test'?  Does everything work as expected here?  What version of surefire are you using? 

Andreas

[Henrik Lynggaard]

Hi

Please ask for all the information you need, and I will try to provide
it

Running the test using mvn test appears to work as expected and
genereted the following xml

testng-results.xml:
<testng-results>
<reporter-output>
</reporter-output>
<suite name="Failed suite [Command line suite]">
<groups>
</groups>
<test name="Command line test(failed)">
<class name="net.apmoller.maventest.NgTest">
<test-method status="FAIL" signature="testFailNg()"
name="testFailNg" duration-ms="4" started-at="2008-01-16T15:40:09Z"
finished-at="2008-01-16T15:40:09Z">
<exception class="java.lang.AssertionError">
<message>
<![CDATA[expected:<true> but was:<false>]]>
</message>
<full-stacktrace>
<![CDATA[java.lang.AssertionError: expected:&lt;true&gt;
but was:&lt;false&gt;
at org.testng.Assert.fail(Assert.java:84)
..... clipped out....
]]>
</full-stacktrace>
</exception>
</test-method>
</class>
<class name="net.apmoller.test2.NgTest">
<test-method status="FAIL" signature="testFailNg()"
name="testFailNg" duration-ms="1" started-at="2008-01-16T15:40:10Z"
finished-at="2008-01-16T15:40:10Z">
<exception class="java.lang.AssertionError">
<message>
<![CDATA[expected:<true> but was:<false>]]>
</message>
<full-stacktrace>
<![CDATA[java.lang.AssertionError: expected:&lt;true&gt;
but was:&lt;false&gt;
at org.testng.Assert.fail(Assert.java:84)
..... clipped out....
]]>
</full-stacktrace>
</exception>
</test-method>
</class>
</test>
</suite>
</testng-results>

The TEST-TestSuite.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite failures="2" time="0.418" errors="0" skipped="0" tests="2"
name="TestSuite">
<properties>
<property name="java.runtime.name" value="Java(TM) SE Runtime
Environment"/>
<property name="sun.boot.library.path" value="/opt/jdk1.6.0_02/jre/
lib/i386"/>
<property name="java.vm.version" value="1.6.0_02-b05"/>
<property name="java.vm.vendor" value="Sun Microsystems Inc."/>
<property name="java.vendor.url" value="http://java.sun.com/"/>
<property name="path.separator" value=":"/>
<property name="java.vm.name" value="Java HotSpot(TM) Client VM"/>
<property name="file.encoding.pkg" value="sun.io"/>
<property name="user.country" value="DK"/>
<property name="sun.java.launcher" value="SUN_STANDARD"/>
<property name="sun.os.patch.level" value="unknown"/>
<property name="java.vm.specification.name" value="Java Virtual
Machine Specification"/>
<property name="user.dir" value="/home/hlyh/NetBeansProjects/
testproj"/>
<property name="java.runtime.version" value="1.6.0_02-b05"/>
<property name="java.awt.graphicsenv"
value="sun.awt.X11GraphicsEnvironment"/>
<property name="basedir" value="/home/hlyh/NetBeansProjects/
testproj"/>
<property name="java.endorsed.dirs" value="/opt/jdk1.6.0_02/jre/
lib/endorsed"/>
<property name="os.arch" value="i386"/>
<property name="java.io.tmpdir" value="/tmp"/>
<property name="line.separator" value="
"/>
<property name="java.vm.specification.vendor" value="Sun
Microsystems Inc."/>
<property name="os.name" value="Linux"/>
<property name="classworlds.conf" value="/opt/maven-2.0.8/bin/
m2.conf"/>
<property name="sun.jnu.encoding" value="UTF-8"/>
<property name="java.library.path" value="/opt/jdk1.6.0_02/jre/lib/
i386/client:/opt/jdk1.6.0_02/jre/lib/i386:/opt/jdk1.6.0_02/jre/../lib/
i386:/usr/java/packages/lib/i386:/lib:/usr/lib"/>
<property name="surefire.test.class.path" value="/home/hlyh/
NetBeansProjects/testproj/target/test-classes:/home/hlyh/
NetBeansProjects/testproj/target/classes:/home/hlyh/.m2/repository/org/
testng/testng/5.7/testng-5.7-jdk15.jar:/home/hlyh/.m2/repository/junit/
junit/3.8.1/junit-3.8.1.jar:"/>
<property name="java.specification.name" value="Java Platform API
Specification"/>
<property name="java.class.version" value="50.0"/>
<property name="sun.management.compiler" value="HotSpot Client
Compiler"/>
<property name="os.version" value="2.6.22-14-generic"/>
<property name="user.home" value="/home/hlyh"/>
<property name="user.timezone" value="Europe/Copenhagen"/>
<property name="java.awt.printerjob"
value="sun.print.PSPrinterJob"/>
<property name="file.encoding" value="UTF-8"/>
<property name="java.specification.version" value="1.6"/>
<property name="user.name" value="hlyh"/>
<property name="java.class.path" value="/opt/maven-2.0.8/boot/
classworlds-1.1.jar"/>
<property name="java.vm.specification.version" value="1.0"/>
<property name="sun.arch.data.model" value="32"/>
<property name="java.home" value="/opt/jdk1.6.0_02/jre"/>
<property name="java.specification.vendor" value="Sun Microsystems
Inc."/>
<property name="user.language" value="en"/>
<property name="java.vm.info" value="mixed mode, sharing"/>
<property name="java.version" value="1.6.0_02"/>
<property name="java.ext.dirs" value="/opt/jdk1.6.0_02/jre/lib/
ext:/usr/java/packages/lib/ext"/>
<property name="sun.boot.class.path" value="/opt/jdk1.6.0_02/jre/
lib/resources.jar:/opt/jdk1.6.0_02/jre/lib/rt.jar:/opt/jdk1.6.0_02/jre/
lib/sunrsasign.jar:/opt/jdk1.6.0_02/jre/lib/jsse.jar:/opt/jdk1.6.0_02/
jre/lib/jce.jar:/opt/jdk1.6.0_02/jre/lib/charsets.jar:/opt/jdk1.6.0_02/
jre/classes"/>
<property name="java.vendor" value="Sun Microsystems Inc."/>
<property name="maven.home" value="/opt/maven-2.0.8"/>
<property name="localRepository" value="/home/hlyh/.m2/repository"/
>
<property name="file.separator" value="/"/>
<property name="java.vendor.url.bug" value="http://java.sun.com/
cgi-bin/bugreport.cgi"/>
<property name="sun.cpu.endian" value="little"/>
<property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
<property name="sun.desktop" value="gnome"/>
<property name="sun.cpu.isalist" value=""/>
</properties>
<testcase time="0.006" name="testFailNg">
<failure message="expected:&amp;lt;true&amp;gt; but
was:&amp;lt;false&amp;gt;"
type="java.lang.AssertionError">java.lang.AssertionError:
expected:&amp;lt;true&amp;gt; but was:&amp;lt;false&amp;gt;
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.failNotEquals(Assert.java:438)
at org.testng.Assert.assertTrue(Assert.java:32)
at org.testng.Assert.assertTrue(Assert.java:42)
at net.apmoller.maventest.NgTest.testFailNg(NgTest.java:23)
</failure>
</testcase>
<testcase time="0" name="testFailNg">
<failure message="expected:&amp;lt;true&amp;gt; but
was:&amp;lt;false&amp;gt;"
type="java.lang.AssertionError">java.lang.AssertionError:
expected:&amp;lt;true&amp;gt; but was:&amp;lt;false&amp;gt;
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.failNotEquals(Assert.java:438)
at org.testng.Assert.assertTrue(Assert.java:32)
at org.testng.Assert.assertTrue(Assert.java:42)
at net.apmoller.test2.NgTest.testFailNg(NgTest.java:23)
</failure>
</testcase>
</testsuite>

If I run mvn site, then the resulting html contains this table for
package list
<table class="bodyTable"><tr class="a"><th>Package</th>
<th>Tests</th>
<th>Errors </th>
<th>Failures</th>
<th>Skipped</th>
<th>Success Rate</th>
<th>Time</th>
</tr>
<tr class="b"><td><a href="#"></a>
</td>
<td>2</td>
<td>0</td>
<td>2</td>
<td>0</td>
<td>0%</td>
<td>0.397</td>
</tr>
</table>

and this table for class list:
<table class="bodyTable"><tr class="a"><th></th>
<th>Class</th>
<th>Tests</th>
<th>Errors </th>
<th>Failures</th>
<th>Skipped</th>
<th>Success Rate</th>
<th>Time</th>
</tr>
<tr class="b"><td><a href="#TestSuite"><img src="images/
icon_warning_sml.gif" /></a>
</td>
<td><a href="#TestSuite">TestSuite</a>
</td>
<td>2</td>
<td>0</td>
<td>2</td>
<td>0</td>
<td>0%</td>
<td>0.397</td>
</tr>
</table>

I am running TestNG 5.7 and surefire 2.4, and using the following
testNg xml
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="5" skipfailedinvocationCounts="false" verbose="0"
name="Failed suite [Command line suite]" junit="false"
annotations="JDK">
<test name="Command line test(failed)" junit="false"
annotations="JDK">
<classes>
<class name="net.apmoller.test2.NgTest">
<methods>
<include name="testFailNg"/>
</methods>
</class>
<class name="net.apmoller.maventest.NgTest">
<methods>
<include name="testFailNg"/>
</methods>
</class>
</classes>
</test>
</suite>

please note that I have removed the testcases which succeded

best regards
Henrik

On Jan 16, 3:23 pm, "Andreas Guther" <andreas.gut...@gmail.com> wrote:
> The information you have provided is not sufficient to make an assessment.
> What about running the tests with 'mvn test'? Does everything work as
> expected here? What version of surefire are you using?
>
> Andreas
>

> On Jan 16, 2008 3:42 AM, Henrik Lynggaard <henrik.lyngga...@gmail.com>

[Henrik Lynggaard]

Hi again

I have had a brief look at the surefire 2.4 reports plugin and it
appears that it doesn't contain any code to handle testNG's style of
test reports. And the format of the two are somewhat different.

The junit runner creates a TEST-*.xml for each class with a testcase
for each failing method. the name attribute of the testsuite is the
fully qualified classname

The testng runner creates a single TEST-*.xml file, with a testcase
for each failing method regardless of class. The name attribute of the
testsuite is "TestSuite"

If my observation is correct then the question is how to fix this:

1)Fix testNG to generate the same output as junit ?
2) Fix the surefire testng provider to execute the tests correctly?
3) fix the reporter plugin to handle testNG output e.g. by using
testng-result.xml as input ?

Can someone confirm my observation and clarify what the intended
behaviour is ?

Best regards

Henrik

On Jan 16, 3:23 pm, "Andreas Guther" <andreas.gut...@gmail.com> wrote:

> The information you have provided is not sufficient to make an assessment.
> What about running the tests with 'mvn test'? Does everything work as
> expected here? What version of surefire are you using?
>
> Andreas
>

> On Jan 16, 2008 3:42 AM, Henrik Lynggaard <henrik.lyngga...@gmail.com>

[Dan Fabulich]

Henrik Lynggaard wrote:

> If my observation is correct then the question is how to fix this:
>

> 1) Fix testNG to generate the same output as junit ?

> 2) Fix the surefire testng provider to execute the tests correctly?
> 3) fix the reporter plugin to handle testNG output e.g. by using
> testng-result.xml as input ?

In this case, while TestNG is generating its own testng-result.xml file,
the TEST-*.xml files are actually generated by Surefire, not by TestNG.

Basically, Surefire instructs TestNG not to generate JUnit XML, while
Surefire registers itself with TestNG to be "notified" of test results as
they happen. Surefire then writes the XML file itself.

So, regardless how we ultimately fix it, this is a bug in Surefire.

Please file a bug here:
http://jira.codehaus.org/secure/CreateIssue!default.jspa

It would be very helpful if you were to attach a "reduced test case" in
the form of a minimal Maven project that demonstrates the problem.

Thanks!

-Dan

[Henrik Lynggaard]

Hello

Thanx for the clarification, with your help I actually think I found
the problem in the code.

I have created http://jira.codehaus.org/browse/SUREFIRE-433 covering
the issue including a test project

Best Regards
Henrik