how to stop a test suite if one method fails

[mwaschkowski]

Hi,

I've searched the forums, but couldn't find anything except old
threads that may not be applicable anymore - is there a way to cancel
all subsequent tests when one test fails?

Thank you!

Mark

[Cédric Beust ♔]

No, there is no such thing at the moment.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Mark Waschkowski]

OK, thank you.

Mark

2011/4/18 Cédric Beust ♔ <ced...@beust.com>

--
Mark Waschkowski
Senior Architect/Project Manager

RefinedData
174 West Beaver Creek
Richmond Hill, ON
L4B 1B4

ma...@refineddata.com
1-877-643-6439 x 789

[Krishnan Mahadevan]

Alexander,

I believe the "status-quo" remains the same and I am not aware of anything that would cause subsequent tests to cancel off if one test fails.

If we are referring to @Test annotated methods when we say "Tests", then this can be done by using dependsOn, but I dont think this can be done if we are referring to <test> tags in the suite file.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

On Wed, Aug 8, 2012 at 5:54 PM, Alexander Schikora <asb...@googlemail.com> wrote:

One year has passed, is there anything implemented by now that would allow this?

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/ElUq_cuEA_kJ.

[Harini]

Any update on this please?

One use case: If the test site is broken and the first test fails because we cannot log into the site anymore, I would like to end the test build since there is no point in running all the tests.

Appreciate any alternatives if this is not possible with TestNG.
 
Thanks,
Harini

[Jeff]

Unless I am missing something, add a @BeforeSuite method to validate that the site is up and functioning.  That should stop all tests if it fails. 

You could also create a set of validation @Test methods and put them in a "validation" group and make all other test classes depend on that group passing using the 'dependsOnGroups' value in the annotation.

 

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--

Jeff Vincent
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

[Harini]

Thanks Jeff!

I like the @BeforeSuite approach. I will try that.

[Andrew McElroy]

Here is my "solution" (i.e. work-around) for this issue.  It might not solve your specific case but it's good enough for my problem.

First, I added a variable with a getter and setting in my main class to flag whether or not we should abort the entire test suite:

 

    public static Boolean abortTestSuite = false;

    public static Boolean getAbortTestSuite() {
        return abortTestSuite;
    }

    public static void setAbortTestSuite(Boolean abortTestSuite) {
        TestBedEnv.abortTestSuite = abortTestSuite;
    }

Next I set "abortTestSuite" in any test case that might trigger the entire suite to fail (in my case I wanted to add setup stuff as test cases as opposed to putting it in @BeforeSuite).  So my test case could look something like this:

    @Test(priority = -10)
    public void test01() throws Exception {
        // if something catastrophic happens then....
        TestBedEnv.setAbortTestSuite(true);
    }

Finally, in my Listener, I add a check for this variable in "onStartTest":

    @Override
    public void onTestStart(ITestResult tr) {
        if (TestBedEnv.getAbortTestSuite()) {
            throw new SkipException ("Skipping Test: " + tr.getName());
        }
    }

This will result in ALL test cases (AFTER the one where we set "abortTestSuite" to true) to be skipped.  They will not execute and will show up in the log as skipped.  This way I do not need to remember to add "dependsOnGroups" to new test cases or anything else like that.

[djan...@gmail.com]

Here is how I do it.  I define it in my testing.xml  .   I have one test in the group called "critical"  and it is called "FastFailTest".

<suite name="Regression Test Suite" time-out="3600000" preserve-order="true" parallel="classes" thread-count="50">
  <listeners>
    <listener class-name="myFramework.engine.test.selenium.common.helper.CustomReport" />
  </listeners>
  <test name="Regression Tests" preserve-order="true">
    <groups>
            <dependencies>
                <group name="regression" depends-on="critical" />
            </dependencies>
        </groups>
    <classes>
    <class name="co.tests.UN00_FailFastSimpleSearch"/>
    <class name="co.tests.UN01_NameSearch" />
      ....


-Jon

[Nguyen Thai Hoang]

Hi all,

I met the same issue above. Below is my test suite, have 4 test cases, my expectation results is: if one test case running failed, other test cases in the test suite will not run, and TestNG will go ahead to run other test suite.

<?xml version="1.0" encoding="utf-8" standalone="yes"?>

<suite name="SOAPUI" verbose="3">

    <test name="SOAPUI024">

        <classes>

            <class name="TestLab.APOutsource.SOAPUI.SOAPUI024" />

        </classes>

    </test>

    <test name="SOAPUI025">

        <classes>

            <class name="TestLab.APOutsource.SOAPUI.SOAPUI025" />

        </classes>

    </test>

    <test name="SOAPUI026">

        <classes>

            <class name="TestLab.APOutsource.SOAPUI.SOAPUI026" />

        </classes>

    </test>

    <test name="SOAPUI027">

        <classes>

            <class name="TestLab.APOutsource.SOAPUI.SOAPUI027" />

        </classes>

    </test>

</suite>

For example: Test case SOAPUI025 failed while running, the TestNG will skip test case SOAPUI026 and SOAPUI027, and will run other test suite (if any)

I'm using TestNG 6.9.9 and interact with Selenium

Any suggest? Many thanks

PS: sorry my bad English

[Krishnan Mahadevan]

​One of the ways in which you can achieve this is by creating a Listener as below and wire it to your suite xml file.

@Listeners(ListenerSample.Listen.class)
public class ListenerSample {
    @Test
    public void test1() {
        System.err.println("test1() executed");
    }

    @Test
    public void test2() {
        System.err.println("test2() executed");
        throw new RuntimeException("Simulating a failure");
    }

    @Test
    public void test3() {
        System.err.println("test3() will never be executed");
    }

    public static class Listen implements IInvokedMethodListener {
        private boolean hasFailures = false;

        @Override
        public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
            synchronized (this) {
                if (hasFailures) {
                    throw new SkipException("Skipping this test");
                }
            }
        }

        @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
            if (! method.isTestMethod()) {
                return;
            }
            if (! testResult.isSuccess()) {
                synchronized (this) {
                    hasFailures = true;
                }
            }

        }
    }

}

Here's the output 

​

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

[Hoang Nguyen]

Hi all,

I've found out the solution for my scenrio by using dependsOnGroups and Dependencies in XML

1. I separate one real testcase into one class

2. I set testcase primary with Anotation @Test(groups = { "Primary" })










3. I set testcase dependency with Anotation @Test(groups = {"Dependency"}) 




---> It means testcase SOAPUI005 depends on SOAPUI003, when SOAPUI003 fail, it will not run testcase SOAPUI005

4. On testsuite.xml

5. When test case SOAPUI003 failed --> it will skip running SOAPUI005




[akshay....@agrostar.in]

Thanks Jeff! The @BeforeSuite approach works like a charm. :)

[Shruti]

Thank you! This helped me achieve what I need for establishing dependencies between tests!