Inconsistent Suite Report Results. Tests Missing.

[Kevin Kucera]

Hello everyone. I'm getting random test methods excluded from my suite report. To make sure it wasn't one of my tests I setup as simple of an example as I could and I still get the same results.

Here's what I've got:

My xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Framework Tests" verbose="1" parallel="tests" thread-count="5">
<listeners>
   <listener class-name="org.testng.reporters.SuiteHTMLReporter" />
   <listener class-name="org.testng.reporters.TestHTMLReporter" />
</listeners>
<test name="Test-Stuff-1">
   <classes>
      <class name="smoketests.framework.TestStuff1" />
   </classes>
</test>
<test name="Test-Stuff-2">
   <classes>
      <class name="smoketests.framework.TestStuff2" />
   </classes>
</test>
<test name="Test-Stuff-3">
   <classes>
      <class name="smoketests.framework.TestStuff3" />
   </classes>
</test>
<test name="Test-Stuff-4">
   <classes>
      <class name="smoketests.framework.TestStuff4" />
   </classes>
</test>
</suite>

My tests only print out their name. They all look like this.

public class TestStuff1 {
   @Test
   public void test1() {
      System.out.println("test 1");
   }

   @Test
   public void test2() {
      System.out.println("test 2");
   }
}

When I run the suite all the tests execute successfully, but when I look at the suite report output (index.html) it doesn't contain all the methods that were run. Some times it shows 2, sometimes 4, sometimes 6, or 8. It seems to randomly fail to report on one of the <test> groups. It only seems to be a problem with the suite report being generated but not the individual test reports. The individual test files generate fine, but the suite report is almost always missing one or more of the tests defined by a <test> tag.