TestNG appears to double count if SkipException is thrown in @BeforeMethod

[Nagesh]

I have a @BeforeMethod that throws a SkipException if a condition is met.

I have noticed that when I run a suite via IDE or via Maven, both the total count and skip count are double counted when the run completes. If we view the report via a results file the counts are correct. But if we run 4 tests, with 2 that skip, in the command-line via Maven/Intellij would be 6 tests - 2 passed and 4 skipped.

My expectation is that the test which failed in the @BeforeMethod should be counted towards the overall count only once. Is this known behavior? Or is it possible via a TestNG configuration to not have it count a skipped test twice?

Thank you for your inputs.

[⇜Krishnan Mahadevan⇝]

Nagesh,

Do you have a sample which can be run to recreate the problem you are describing ? That would help add a bit more clarity in terms of what you are looking for here.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Nagesh]

Hi Krishnan,

Thanks for getting back regarding this issue. Sorry for the delay in the response as I was away for a few days.

Here's an example -

Excerpt from Base class:

@BeforeMethod
public void preCheck(Method method) {
    List<String> testMethodGroups = Arrays.asList(method.getAnnotation(Test.class).groups());
    if (!Collections.disjoint(groupsToBeSkippedList, testMethodGroups)) {
        String message = "Skipping " + method.getName() + " test because " + groupsToBeSkippedList + " was not available in " + environment;
        Reporter.log(message, true);
        throw new SkipException(message);
    }
}

Excerpt from test class that extends base class:

@Test(groups = {Groups.XYZ, Groups.ABC})
public void verifyServiceIsUp() {
    String baseUrl = "http://localhost:" + 9000;
    ResponseEntity<String> entity = restTemplate.getForEntity(baseUrl, String.class);
    Assert.assertTrue(entity.getStatusCode().is2xxSuccessful());
}

If Groups.XYZ happens to be in the groupsToBeSkippedList and in testMethodGroups, I throw a SkipException, so that the test is skipped immediately and moves on to the next test. What I see in the Maven output in this case, if I were to run just this 1 test, is Tests run: 2, Skipped: 2.

-Nagesh

[⇜Krishnan Mahadevan⇝]

Nagesh,

This is not an issue with TestNG, but it looks like a problem with the Maven surefire plugin code.

I tried to simulate the problem using Maven sure-fire plugin v 2.15 and realised that TestNGReporter in Maven surefire plugin is basically counting skipped methods even though they are configuration methods [ See here ] whereas the TestNG reporters count only Test methods when reporting skipped methods [ atleast that's what I think :) ]

So if you would like to fix this, then you would need to do one of the following :

Option 1:

Send a Pull Request to the Maven surefire plugin code

Option 2 :

This is more of a hack than a solution.

Add a dependency to the Maven surefire plugin version that you are working with. For e.g., I was using surefire plugin version 2.15, so I added a dependency as below :

<dependency>
    <groupId>org.apache.maven.surefire</groupId>
    <artifactId>surefire-testng</artifactId>
    <version>2.15</version>
</dependency>

Now resort to classpath overriding for  the class org.apache.maven.surefire.testng.TestNGReporter [ i.e., create the same package in your test project and copy paste the entire class code into it ] and alter the method TestNGReporter.onSkipped() to something like below :

public void onTestSkipped( ITestResult result ){
    ReportEntry report = new SimpleReportEntry( getSource( result ), getUserFriendlyTestName( result ) );
    if (!result.getMethod().isTest()) {
        return;
    }
    reporter.testSkipped( report );
}

This will now start making surefire plugin to count only test methods which are skipped and will not count configuration methods that were getting skipped.

That should solve the problem that you are experiencing.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Krishnan]

For the sake of tracking, I have filed the problem with surefire plugin as an issue : https://issues.apache.org/jira/browse/SUREFIRE-1248

Julien,

Thanks for sharing the bug tracking link for surefire.

[Nagesh]

Hi Krishnan, 

Interesting find! Thanks for the detailed follow up and work around. Also thanks for reporting the issue on the Surefire project.

Regards,

Nagesh