ant testng mixed mode gets NPE on certain Junit3 tests

[Mike Kienenberger]

I have 1425 junit tests in a 10-year-old project, and I'm doing a major upgrade on this project.   All of these tests currently pass under junit 3, both from eclipse and from ant.

Since the last time I worked on this project, I have migrated from using junit3 to testng on other projects, so I was overjoyed to find that testng now supports running my old junit3 tests directly -- before I had planned to maintain both kinds of tests with separate ant build files.

However, when I got everything reconfigured, I found that my testng 6.8.8 ant task only ran 657 of my tests, and some of these tests were isolated test cases not in my main test suite (but that's another issue).

Instead, there were 74 errors like this one hidden in the 389093 lines of output from running the tests:

   [testng] Failure in JUnit mode for class com.xyz.foo.barTest: could not create/run JUnit test suite: null

These only appear on the console and not in the test results summary files.

Attaching the ant process to the debugger, I note that each of these I've sorted through so far have this problem:

In JUnit3TestMethod.getMethod(Test) line: 21, nameMethod is always null.
   
    private static Method getMethod(Test t) {
        String name = null;
        try {
            Method nameMethod = t.getClass().getMethod("getName");
            name = (String) nameMethod.invoke(t);

I'm not sure why name would be null, nor am I certain what it's supposed to be doing.  I think I read that the name field for a junit3 test is populated with the current test method, but I don't know exactly how this gets populated in the testng junit3 runner.

For some of these, the tests being run are in an abstract testing superclass, a subclass of the abstract testing superclass, or a subclass of a testing superclass (poor man's parameterization implementation under junit3).  Others seem to be valid test classes.

I'm at a loss as to where I should investigate to solve this further.  I'll probably git clone the testng project and build it from source as a next step.




Side note: error reporting here needs to be improved as we're not getting the root causes or stack traces, only the topmost stack trace.   The root exception should be passed into runFailed() and added as the cause for the TestNGException created there.   Not that it helped me much in this situation.

        catch (Exception e) {  runFailed(testCase, "could not create/run JUnit test suite: " + e.getMessage()); }

This was the root stack trace:

   [testng] java.lang.NullPointerException
   [testng]     at org.testng.internal.ConstructorOrMethod.getDeclaringClass(ConstructorOrMethod.java:26)
   [testng]     at org.testng.internal.BaseTestMethod.<init>(BaseTestMethod.java:91)
   [testng]     at org.testng.internal.BaseTestMethod.<init>(BaseTestMethod.java:86)
   [testng]     at org.testng.junit.JUnitTestMethod.<init>(JUnitTestMethod.java:15)
   [testng]     at org.testng.junit.JUnit3TestMethod.<init>(JUnit3TestMethod.java:14)
   [testng]     at org.testng.junit.JUnitTestRunner.recordResults(JUnitTestRunner.java:118)
   [testng]     at org.testng.junit.JUnitTestRunner.endTest(JUnitTestRunner.java:106)
   [testng]     at junit.framework.TestResult.endTest(TestResult.java:72)
   [testng]     at junit.framework.TestResult.run(TestResult.java:111)
   [testng]     at junit.framework.TestCase.run(TestCase.java:118)
   [testng]     at junit.framework.TestSuite.runTest(TestSuite.java:208)
   [testng]     at junit.framework.TestSuite.run(TestSuite.java:203)
   [testng]     at org.testng.junit.JUnitTestRunner.doRun(JUnitTestRunner.java:287)
   [testng]     at org.testng.junit.JUnitTestRunner.start(JUnitTestRunner.java:260)
   [testng]     at org.testng.junit.JUnitTestRunner.run(JUnitTestRunner.java:248)
   [testng]     at org.testng.TestRunner$1.run(TestRunner.java:682)
   [testng]     at org.testng.TestRunner.runWorkers(TestRunner.java:1005)
   [testng]     at org.testng.TestRunner.privateRunJUnit(TestRunner.java:713)
   [testng]     at org.testng.TestRunner.run(TestRunner.java:614)
   [testng]     at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
   [testng]     at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
   [testng]     at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
   [testng]     at org.testng.SuiteRunner.run(SuiteRunner.java:254)
   [testng]     at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
   [testng]     at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
   [testng]     at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
   [testng]     at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
   [testng]     at org.testng.TestNG.run(TestNG.java:1057)
   [testng]     at org.testng.TestNG.privateMain(TestNG.java:1364)
   [testng]     at org.testng.TestNG.main(TestNG.java:1333)

[Mike Kienenberger]

On Monday, March 24, 2014 7:11:55 PM UTC-4, Mike Kienenberger wrote:

Instead, there were 74 errors like this one hidden in the 389093 lines of output from running the tests:

   [testng] Failure in JUnit mode for class com.xyz.foo.barTest: could not create/run JUnit test suite: null

These only appear on the console and not in the test results summary files.

Attaching the ant process to the debugger, I note that each of these I've sorted through so far have this problem:

In JUnit3TestMethod.getMethod(Test) line: 21, nameMethod is always null.
   
    private static Method getMethod(Test t) {
        String name = null;
        try {
            Method nameMethod = t.getClass().getMethod("getName");
            name = (String) nameMethod.invoke(t);

 
I solved this one already.   It turns out that a testing superclass was calling "setName(null)" in teardown, which then broke the testng's manipulation of the class from that point onward.