Selenium + TestNG - TestNG skips execution of tests after assertion failure.

[Kartik]

Hi All,

I extend Selenium's SeleneseTestNGHelper which has exposed the method
checkForVerificationErrors. This method is annotated with
@AfterMethod.
http://grepcode.com/file/repo1.maven.org/maven2/org.seleniumhq.selenium.client-drivers/selenium-java-testng-helper/1.0-beta-2/com/thoughtworks/selenium/SeleneseTestNgHelper.java#SeleneseTestNgHelper.checkForVerificationErrors%28%29
From what I have read on TestNG threads, if a configuration fails,
TestNG test runner considers the test environment to be in an unstable
state and skips execution of all tests that would have followed the
test that generated verification errors. Have I understood this
correctly?

If this is correct, this is causing me two problems:
1. Selenium does not immediately cease test execution but stored all
verification errors until you check for those errors. That means that
the test is considered to be successful by TestNG thus causing a false
positive.
2. It skips all executions of the tests that would have followed the
actual failed test.

Has any one had this problem before? I would like to
1. Verify errors but be able to mark the test as failed. This would
prevent any false positives. I looked at this link for soft asserts
http://beust.com/weblog2/archives/000514.html but I don't think I can
use this approach.
2. Prevent any skipped execution of tests. In my base class that
extends SeleniumTestNGHelper, I have overridden the behavior as
follows:

/**
* Asserts that there were no verification errors during the
current test,
* failing immediately if any are found */
@AfterMethod(alwaysRun = true)
public void checkForVerificationErrors(Method m, ITestContext
context) {
try {
super.checkForVerificationErrors();
} catch (AssertionError e) {
Reporter.log(this.verificationErrors.toString());
@SuppressWarnings("unchecked")
Map<String, String> failedMethods =
(Map<String, String>)
context.getAttribute(FAILED_METHODS);
failedMethods.put(m.getDeclaringClass().getSimpleName() +
"_"
+ m.getName(),
this.verificationErrors.toString());
}
}

But this could give false positives for a test that failed. Is there a
way I can change the test result after execution? I looked at
implementing a variation of this strategy but this does not return any
configuration methods.
http://beust.com/weblog/2010/03/23/better-mock-testing-with-testng/

[Suk-Hyun Cho]

We discussed this issue here: http://groups.google.com/group/testng-users/browse_thread/thread/31543b11edd3e62a

On May 28, 1:10 pm, Kartik <krishnan.1...@gmail.com> wrote:
> Hi All,
>
> I extend Selenium's SeleneseTestNGHelper which has exposed the method
> checkForVerificationErrors. This method is annotated with

> @AfterMethod.http://grepcode.com/file/repo1.maven.org/maven2/org.seleniumhq.seleni...

>  From what I have read on TestNG threads, if a configuration fails,
> TestNG test runner considers the test environment to be in an unstable
> state and skips execution of all tests that would have followed the
> test that generated verification errors. Have I understood this
> correctly?
>
> If this is correct, this is causing me two problems:
> 1. Selenium does not immediately cease test execution but stored all
> verification errors until you check for those errors. That means that
> the test is considered to be successful by TestNG thus causing a false
> positive.
> 2. It skips all executions of the tests that would have followed the
> actual failed test.
>
> Has any one had this problem before? I would like to
> 1. Verify errors but be able to mark the test as failed. This would

> prevent any false positives. I looked at this link for soft assertshttp://beust.com/weblog2/archives/000514.htmlbut I don't think I can

[Cédric Beust ♔]

Hi Kartik,

On Fri, May 28, 2010 at 1:10 PM, Kartik <krishn...@gmail.com> wrote:

Hi All,

I extend Selenium's SeleneseTestNGHelper which has exposed the method
checkForVerificationErrors. This method is annotated with
@AfterMethod.

This seems unfortunate to me, for the exact reasons that you mention below:  this verification should be made in a test method, not in a configuration method...

I'll try to respond to the rest of your mail once I have some time to read it in details.

--
Cédric

 

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Kartik Kumar]

Hi Cedric,

I don't know if this is the right approach but I implemented something like this. I got this to work but the parent method is still called as the  two methods have different signatures.

   /**
     * Asserts that there were no verification errors during the current test,

     * failing immediately if any are found.
     *
     * <p>Selenium throws {@link AssertionError} if verification fails. As this
     * method is a configuration method, TestNG considers the test run to be
     * invalid and skips all subsequent test execution. Checks for any
     * assertion errors and changes the test result after the execution of test
     * method.

     */
    @AfterMethod(alwaysRun = true)
    public void checkForVerificationErrors(Method m, ITestContext context) {
        try {
            super.checkForVerificationErrors();
        } catch (AssertionError e) {

           // Only check for false positives.
           ITestNGMethod[] testMethods = context.getAllTestMethods();
           for (ITestNGMethod testMethod : testMethods) {
               if (testMethod.getMethod().getName().equals(m.getName())) {
                   IResultMap resultMap = context.getPassedTests();
                   Set<ITestResult> results = resultMap.getResults(testMethod);
                   for (ITestResult result : results) {
                       result.setStatus(ITestResult.FAILURE);
                   }
               }
           }
        }
    }

2010/5/28 Cédric Beust ♔ <ced...@beust.com>

[Kartik Kumar]

I forgot to mention that that I also overrode the checkVerificationErrors() to do nothing.