BUG: JUnitReportReporter transposes errors and failures

[Thorson Little]

Note: I reported this issue two months ago but had not done the deep dive to determine if this is a clear bug.  I am now clear that it is.  The original post is here: https://groups.google.com/forum/?fromgroups#!topic/testng-dev/M_oR3lifxMM

Documentation:

The following is from JUnit: A Cooks tour (http://junit.sourceforge.net/doc/cookstour/cookstour.htm)

JUnit distinguishes between failures and errors. The possibility of a failure is anticipated and checked for with assertions. Errors are unanticipated problems like an ArrayIndexOutOfBoundsException. Failures are signaled with an AssertionFailedError error. 

To Repro:

TestClass:

public class JunitExample {

  @Test

  public void firstTest() {

 assertTrue( true, "true assertion number 1" ); 

  }

  

  @Test(enabled=true)

  public void secondTest() {

 assertTrue( true, "Assertion 1/3 passes" );

 assertTrue(true);

 assertTrue( false, "Assertion 3/3 fails and whole test fails");

  }

  

  @Test (enabled=true)

  public void thirdTest() {

 assertTrue( true, "true assertion number 2" );

 

 fail( "This is a failure" );

  }

  

  

  @Test

  public void fourthTest() {

 assertTrue( true, "true assertion number 3" ); 

 assertTrue( true, "true assertion number 4" ); 

 throw new RuntimeException("watch this fail");

  }

  

}

Expected Results: 

1 Success(firstTest), 2 failures(secondTest & thirdTest) and 1 error(fourthTest)

Actual Results:

1 Success(firstTest), 2 error(secondTest & thirdTest) and 1 failure(fourthTest)

<?xml version="1.0" encoding="UTF-8"?>

<!-- Generated by org.testng.reporters.JUnitReportReporter -->

<testsuite hostname="tlittle-lt" name="com.example.pkg.JunitExample" tests="4" failures="1" timestamp="29 May 2013 19:14:17 GMT" time="0.011" errors="2">

  <testcase name="thirdTest" time="0.000" classname="com.example.pkg.JunitExample">

    <error type="java.lang.AssertionError" message="This is a failure">

      <![CDATA[java.lang.AssertionError: This is a failure

at org.testng.Assert.fail(Assert.java:94)

...

]]>

    </error>

  </testcase> <!-- thirdTest -->

  <testcase name="fourthTest" time="0.001" classname="com.example.pkg.JunitExample">

    <failure type="java.lang.RuntimeException" message="watch this fail">

      <![CDATA[java.lang.RuntimeException: watch this fail

at com.example.pkg.JunitExample.fourthTest(JunitExample.java:31)

        ...

]]>

    </failure>

  </testcase> <!-- fourthTest -->

  <testcase name="secondTest" time="0.001" classname="com.example.pkg.JunitExample">

    <error type="java.lang.AssertionError" message="Assertion 3/3 fails and whole test fails expected [true] but found [false]">

      <![CDATA[java.lang.AssertionError: Assertion 3/3 fails and whole test fails expected [true] but found [false]

at org.testng.Assert.fail(Assert.java:94)

...

]]>

    </error>

  </testcase> <!-- secondTest -->

  <testcase name="firstTest" time="0.009" classname="com.example.pkg.JunitExample"/>

</testsuite> <!-- com.example.pkg.JunitExample -->

Where To Find The Bug:

In JUnitReportReporter.java

https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/reporters/JUnitReportReporter.java

lines 101-107:

       if (! isSuccess) {
          if (tr.getThrowable() instanceof AssertionError) {
            errors++;
          } else {
            failures++;
          }
        } 

line 126:

 testTag.errorTag = tr.getThrowable() instanceof AssertionError ? "error" : "failure";

[Thorson Little]

Attached environment