Terminating in the middle of a suite

Greg Martz

unread,

Jul 27, 2015, 1:52:21 PM7/27/15

to testng-users

Good morning,

I need a way to terminate the execution of a suite and not even attempt to run any remaining classes. I am using onTestFailure as below, but I do not want to just do a java terminate. I'd like to just stop the script and proceed to the final report generation, and do a graceful exit. This is an End-To-End scenario, and each test must have the test above it passing. There is no way I can see to set the remaining tests to Blocked which would be ideal, so I have to end. Running each and every test as skipped adds a lot to the execution speed as there are 43 tests that must be run. If the first fails, it attempts to run the remaining 42 and just wastes time. See my xml below also.

@Override

public void onTestFailure(ITestResult result)

{

// TODO Auto-generated method stub

Log.info("**********onTestFailure()");

try

{

Utility.logout(Utility.driver);

Utility.closeBrowser(Utility.driver);

}

catch (Exception e)

{

// TODO Auto-generated catch block

e.printStackTrace();

}

System.exit(0);

}

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</listeners>

</classes>

</test>

</suite>

Thanks!

Greg

Hubert Grzeskowiak

unread,

Aug 31, 2015, 11:00:30 AM8/31/15

to testng-users

Hi Greg,

I think the only way to do this is defining explicit group dependencies on the test classes. I.e. give every test class a group name (can even be same as class name) and make it depend on the previous one. Class level annotations can be also used together with method annotations, btw (see here: http://testng.org/doc/documentation-main.html#class-level).

Other than that you might try running the suite using an own test runner. The docs provide an example on that.

Cheers
Hubert Grzeskowiak

Krishnan Mahadevan

unread,

Sep 1, 2015, 12:57:16 AM9/1/15

to testng...@googlegroups.com

Hubert,

That for sure (the adding dependency amongst tests part) is going to pollute the test report, because you are now going to cause false positives. To make matters worse, if a test really needs to depend on other tests, then it becomes more difficult to figure out what was causing the test to fail.

That being said, here’s a hack that I could think of [ Yep, this is by no means a silver bullet ]

@Listeners (MySuperTestClass.CustomTestRunner.class)
public class MySuperTestClass {
    @Test
    public void testMethod1() {
        Reporter.log("testMethod1() was executed", true);
    }

    @Test
    public void testMethod2() {
        Reporter.log("testMethod2() was executed", true);
    }

    @Test
    public void testMethod3() {
        Assert.assertFalse(true, "Was asked to fail.");
        Reporter.log("testMethod3() was executed", true);
    }

    @Test
    public void testMethod4() {
        Reporter.log("testMethod4() was executed", true);
    }

    @Test
    public void testMethod5() {
        Reporter.log("testMethod5() was executed", true);
    }

    public static class CustomTestRunner implements IInvokedMethodListener {
        private boolean hasFailures = false;

        @Override
        public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
            if (hasFailures) {
                String msg = method.getTestMethod().getMethodName() + "() was disabled due to failures.";
                Reporter.log(msg, true);
                throw new SkipException(msg);
            }
        }

        @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
            if (! testResult.isSuccess()) {
                hasFailures = true;
            }
        }
    }

}

Here’s an output :

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running organized.chaos.testng.MySuperTestClass
Configuring TestNG with: TestNG652Configurator
testMethod1() was executed
testMethod2() was executed
testMethod4() was disabled due to failures.
testMethod5() was disabled due to failures.
Tests run: 5, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 0.676 sec <<< FAILURE! - in organized.chaos.testng.MySuperTestClass
testMethod3(organized.chaos.testng.MySuperTestClass)  Time elapsed: 0.008 sec  <<< FAILURE!
java.lang.AssertionError: Was asked to fail. expected [false] but found [true]
	at org.testng.Assert.fail(Assert.java:94)
	at org.testng.Assert.failNotEquals(Assert.java:494)
	at org.testng.Assert.assertFalse(Assert.java:63)
	at organized.chaos.testng.MySuperTestClass.testMethod3(MySuperTestClass.java:21)


Results :

Failed tests:
  MySuperTestClass.testMethod3:21 Was asked to fail. expected [false] but found [true]

Tests run: 5, Failures: 1, Errors: 0, Skipped: 2

Since the skip exception is being thrown from within a beforeInvocation() call, I don’t think there is going to be a lot of time spent in running other tests.

PS :
The caveat here is that your tests are NOT going to be running in parallel. If they do, then there can be race conditions and you would need to add some guards to the code to prevent them.


Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribbings @ http://rationaleemotions.wordpress.com/

-- 

You received this message because you are subscribed to the Google Groups "testng-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at http://groups.google.com/group/testng-users.

For more options, visit https://groups.google.com/d/optout.

Hubert Grzeskowiak

unread,

Sep 1, 2015, 11:44:28 AM9/1/15

to testng-users

Hey Greg,

your solution is similar to dependsOnMethods in that all methods with failed dependencies are skipped. I like the fact you have to write a lot less and that your solution scales well with increased tests number. However, keep in mind that by default the order of test method execution inside one class is not defined. If it happens to be the one you're expecting, then don't let it fool you - it's non-deterministic. (Using dependsOn* or priority it can be made deterministic. Another way is implementing an own AnnotationTransformer or MethodInterceptor.). However, if you don't care at which point in a class the list breaks, then you're fine I guess :-)

Cheers
Hubert Grzeskowiak

Reply all

Reply to author

Forward