automatically verify soft assertions for all tests

[Martin Phillips]

I've been persuaded to have a bash at TestNG as a replacement for JUnit and am struggling to nicely reproduce some common soft assertion functionality I had managed in JUnit to apply automatically to all of my test methods.

I have a base test class that all my other test classes extend.  This lets me define a single setup()@beforeMethod to set up a selenium driver instance (in the base class) for all of the child test classes to use.

Similarly, there's a teardown()@afterMethod that tears down the selenium driver.

I also want all of my tests to be able to easily use soft asserts. (http://beust.com/weblog/2012/07/29/reinventing-assertions/)

The easy way to do this seems to be to create a SoftAssert instance at the same time as I set up the Selenium Driver in @beforeMethod, and then call assertAll() in the @afterMethod.

I found the configfailurepolicy="continue" option to make all tests run even if an @afterMethod 'fails', and it kind of works, however the test method itself isn't marked as failed, just the teardown()@afterMethod.  This makes reporting not as clean as I'd like.  I want to see the names of the tests marked as failed, not the name of the teardown() method. From the various comments I've seen, @afterMethod is designed to be 'not really part of the test', but I want it to be! I want a failure there to fail the test it is associated with (as it does in Junit).

I've seen some stuff about @Verify for mock objects, which looks like it could possibly do what I'm looking for, but that means all my test methods need to specify @Verify in addition to @Test, and if another tester forgets the @Verify, then their tests could accidentally pass as the soft assertion will never be checked.  (http://beust.com/weblog/2010/03/23/better-mock-testing-with-testng/)

The descriptions of using softAssertions seem to require that each individual test call softAssert.assertAll() at the end of every method, which seems like boilerplate that could easily be abstracted away using similar functionality to @beforeMethod.

Can anyone think of a good way of having the @afterMethod (or something similar) common functionality for all of my tests which can make the test fail, rather than being reported as the @afterMethod method failing?

Hope someone can help.

Martin.

[Martin Phillips]

My colleague Dave just solved this for us using IInvokedMethodListener

On the common base class (BaseTestNGTestCase) he added a class level
  @Listeners({ TestMethodListener.class })

and created the listener which does the following: 
Get hold of the method that was just executed, make sure it is a test method,
get the class from the method, make sure it's an instance of our base class
get hold of the SoftAssert stored within it and do the assertAll on it,
if that shows failures, mark the TestResult as a fail.



public class TestMethodListener implements IInvokedMethodListener {

    public TestMethodListener() {
    }

    @Override
    public void beforeInvocation(final IInvokedMethod method,
            final ITestResult testResult) {
    }

    @Override
    public void afterInvocation(final IInvokedMethod method,
            final ITestResult testResult) {
        ITestClass invokingClass = method.getTestMethod().getTestClass();
        Object[] classInstance = invokingClass.getInstances(true);
        if (method.isTestMethod()) {
            if (classInstance[0] instanceof BaseTestNGTestCase) {
                BaseTestNGTestCase testCase = (BaseTestNGTestCase) classInstance[0];
                SoftAssert soft = testCase.getSoftAssert();
                AssertionError err = null;
                try {
                    soft.assertAll();
                } catch (AssertionError e) {
                     err = e;
                }
                if(null != err) {
                    testResult.setStatus(TestResult.FAILURE);
                    testResult.setThrowable(err);
                }

            }
        }
    }
}




Martin.

[Nalin]

Try passing in ITestResult into @AfterMethod and using ITestResult#setStatus().

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Sergey Kuts]

Hi,

First of all, SoftAssert class is already included into TestNG. And it looks the same as Cedric described in this topic http://beust.com/weblog/2012/07/29/reinventing-assertions/.

Regarding to your question. You can create a listener with overriden afterInvication method and perform assertAll() there. How to access soft assert instance from afterInvocation? Just put it into test context in beforeMethod.

Regarding to storing actual results. You can create your own test entities that will collect necessary info during test execution. If you're using reportng library for printing html results, you can easily put your custom test results structure into velocity context and use it in velocity templates.

Regards,
Sergey

2014/1/9 Martin Phillips <martin.ph...@gmail.com>

--

[Tomek Kaczanowski]

Martin, thank you for posting the solution!

2014/1/10 Martin Phillips <martin.ph...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to testng-users...@googlegroups.com.
> To post to this group, send email to testng...@googlegroups.com.
> Visit this group at http://groups.google.com/group/testng-users.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Regards / Pozdrawiam
Tomek Kaczanowski
http://practicalunittesting.com

[sunny sachdeva]

HI Martin,

I am trying this solution but stuck at below code

  SoftAssert soft = testCase.getSoftAssert();

Could you please provide some insight on this methof getSoftAssert.

I am wondering how to provide all the assertion back to afterInvocation so that assertall will fail the method.

Thank

Sunny

[Banh Duy]

Hi sunny,

I guess that all test classes always extend the BaseTestNGTestCase class.

BaseTestNGTestCase class has a protected SoftAssert field. So you have to make a getter for it, getSoftAssert().

protected SoftAssert softAssert;

public SoftAssert getSoftAssert() {
    return softAssert;
}

Vào 18:49:51 UTC+7 Thứ Năm, ngày 12 tháng 2 năm 2015, sunny sachdeva đã viết: