Test pass/fail after performing all asserts

[webtester]

Hi,

I'm not sure if this is the right place to ask this question. But I'll
give it a try.

At the moment my test is failing after the first failing assertion. I
have multiple asserts in my testmethod, because I need to test small
things. But I'm mostly interested in the total end-result.

assert 1: fail
assert 2: pass
assert 3: fail
assert 4: pass

I like to see in the output that assert 1 & 3 where failing. Instead
of just 1.

How can this be done with TestNg?


Thanks,

[Cédric Beust ♔]

Accumulate your test results in a variable, assert at the end.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Roman Hiden]

or use FlexibleAssert that would not fail the test immediately.

[Yevhen Bilevych]

Another approach I'd suggest is to split assertions into different methods that depend on main method. Thus you'll see four results in your report.

Thanks,

Yevhen 

Sent from my BlackBerry® PlayBook™
www.blackberry.com

---

From: "Roman Hiden" <rome...@gmail.com>
To: "testng...@googlegroups.com" <testng...@googlegroups.com>
CC: "ced...@beust.com" <ced...@beust.com>
Sent: October 3, 2012 8:29 PM
Subject: Re: [testng-users] Test pass/fail after performing all asserts

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/Oy3g5S2OCBIJ.

[Roman Hiden]

do you mean creating @Test for each method doing assert? because I don't see methods reporting as passed or failed in default testng reporting.

If you know how to report each method pass or fail could you please share you knowledge.

[Yevhen Bilevych]

Hi Roman,
Yes, each method doing assert should be annotated with @Test. dependsOnXXX is used to keep assertion order.

The code snippet is below. The main idea is to separate different test asserts in report. Also, if some additional action should be done before each assert, it's easy to read the code.

package testng.samples;

import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class MultipleAsserts {
    @BeforeClass
    public void beforeClass() {
        // you may initialize WebDriver there
    }
    @Test
    public void mainTest() {
        // navigate login page and login
        // Assert login succeeded
    }
   
    @Test(dependsOnMethods="mainTest")
    public void testCorrectPageLoaded() {
        // Wait for some page being loaded
        // Assert the url of the page is correct
    }
    @Test(dependsOnMethods="testCorrectPageLoaded")
    public void testLogoIsShown() {
        // Assert that logo is loaded and shown (this may need more than one
        // verification so we can use FlexibleAssert or accumulate results in
        // local array and then assert in the end of method, as Roman and Cedric
        // suggested)
    }
    @Test(dependsOnMethods="testCorrectPageLoaded")
    public void testCorrectUsernameIsShown() {
        // Assert welcome message shows correct username
    }
    @Test(dependsOnMethods="testCorrectPageLoaded")
    public void testMenuIsShown() {
        // Say menu depends in who has logged in. Assert that correct menu loaded
    }
    //...
}


Thanks,
Yevhen

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/YcGoUgXwfooJ.

[Roman Hiden]

Thanks for explaining, I tough there was another method.

[GUnit]

Hi Cedric,

I am currently having the same issue. Currently, I am using JUnit and what I did is to accumulate the test results in a failMessage variable and then call JUnit's fail(failMessage) method at the end of the test run. That fail() method is called not in the test method but in a super class.

I am now trying to migrate to TestNG and I see there is no corresponding fail() method that I can call. I know there is an Assert.fail(message) and take the same approach to accumulate the test results and then call this at the end of the test run. However, I believe that Assert.fail(allFailMessages) will need to be called inside the @Test method (at the end). What I would like to see is a way to do this outside of the @Test method. If I put this in an @AfterMethod, then I get an configuration failure. Is there any way for me to accomplish this?

The other thing I was thinking of is to override the onFinish(ITestContext) method on a listener and do the following:
1)  Retrieve my test failure messages
2)  Create a ITestResult to represent the failure
3)  Add the ITestResult to the list of failed tests from getFailedTests().

However, I can't find a way to accomplish #2 (to create a ITestResult). I don't see any javadoc on this.

Please help.

Thanks.

On Monday, October 1, 2012 12:11:21 PM UTC-4, Cédric Beust ♔ wrote:

[Cédric Beust ♔]

I'm not sure I understood correctly, it looks like you are trying to create a test method that will be automatically be invoked after each test method?

In this case, this article might help you (it's about mocks but you can reuse the same idea):

http://beust.com/weblog/2010/03/23/better-mock-testing-with-testng/

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/GmMrW64KMLoJ.

[GUnit]

I would like to create a method (not a test method) that will be automatically be invoked after each test method. The purpose of this method is to log the accumulation of all the failures, if there are any. If there are, then it will fail the test. I would like this to be done automatically, so I don't need to explicitly do this at the end of every test.

[Cédric Beust ♔]

How about using the new assertions in TestNG 6.8?

Details here:  http://beust.com/weblog/2012/07/29/reinventing-assertions/

-- 

Cédric

[GUnit]

Ok. I think the link you provided with mock testing might help (after reading it a bit). I will try to apply the same idea.

Thanks.

[Roman Hiden]

I would Follow Cedric Suggestion And Use Flexibleasserts

[GUnit]

The new Assertion class is nice, but it does not solve the issue. To illustrate, in that blog entry, you had the following code.

@Test

public class A {

private SoftAssert m_assert = new SoftAssert();

public void multiple() {

m_assert.assertTrue(true, "Success 1");

m_assert.assertTrue(true, "Success 2");

m_assert.assertTrue(false, "Failure 1");

m_assert.assertTrue(true, "Success 3");

m_assert.assertTrue(false, "Failure 2");

m_assert.assertAll();

}

}

If I add another test method, say multiple2() that is the same as the multiple() test method (just as an example), you can see that "m_assert.assertAll();" will need to be called again at the end of both test methods.

What I am looking for is, to use this as an example, is for "m_assert.assertAll();" to be in a different method, outside of the test method so that it can be used globally and called right at the end of the test, instead of explicitly calling it over and over again at the end of every test method. To me that's just repetitive code. It should have the same effect as it being called at the end of the each test method. Something like having @RightAfterTest is what I am looking for.

I hope it is clear and you understand what I am saying. The problem here is that I have tons of tests, which I do not want to go back and manually have to update each and every one of them to use the Assertion class. I want to be able to update just that one place (method) and it affects all the tests.


[Cédric Beust ♔]

Take a look at my email from yesterday where I point to an article on my blog describing how to do exactly that (to verify mocks, but the same idea will work for you).

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/FAwsGqVfBvYJ.

[GUnit]

I'm not sure if this is a bug in TestNG or not. I believe it is. I am applying the same idea as described in that article. Here is what I have in my test class.

    @Test
    public void testMissingFields() {
        logFailMessage("1");
        logFailMessage("2");
    }
   
    @TestAssertion
    @Test
    public void logTestResult() {
        if (!StringUtil.isEmpty(getFailMessage())) {
            Assert.fail(getFailMessage());
        }
   
Here is my IMethodInteceptor implementation.

public class TestInterceptor implements IMethodInterceptor {

    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods,
            ITestContext context) {
        List<IMethodInstance> result = new ArrayList<IMethodInstance>();
        IMethodInstance assertion = null;
       
        // find the assertion method
        for (IMethodInstance method : methods) {
            //if (method.getMethod().getConstructorOrMethod().getMethod().isAnnotationPresent(TestAssertion.class)) {
            Method m = method.getMethod().getConstructorOrMethod().getMethod();
            Annotation[] annotations = m.getAnnotations();
            if (m.isAnnotationPresent(TestAssertion.class)) {
            //if (method.getMethod().getMethod().isAnnotationPresent(TestAssertion.class)) {
                assertion = method;
                break;
            }
        }
       
        if (assertion == null) {
            return methods;
        }
       
        for (IMethodInstance method : methods) {
            if (method != assertion) {
                result.add(method);
                result.add(assertion); // so it runs after the @Test method
            }
        }
       
        return result;
    }

}

When I debug my test class, the method inteceptor can only find 1 annotation on each test method and that is @Test. It did not find the other annotations such as the @TestAssertion that I have.

I think I'm doing this right from what I understand.

[Cédric Beust ♔]

Is your @TestAnnotation's retention "runtime"?

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/9_cXAwjYBxsJ.

[GUnit]

That was the problem. So I added the runtime retention to my annotation and it is picking it up now. However, the problem now is that the test count is doubled. Because the "@TestVerifier" method needs to also be annotated with @Test so it gets picked up by the IMethodInterceptor and the fact that I have to run this method right after every single test, the total number of tests reported is now doubled. So if I have 2 tests, then it will be reported as 4.

Having gone through this issue, I feel like this is an area in which TestNG can improve on and should be able to handle and allow. Is this an enhancement that I can request and if so, where should I do this?

[Cédric Beust ♔]

Feel free to file an issue on the github tracker at http://github.com/cbeust/testng

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/Oy6-eb4_PdEJ.