Quarantine tests and and setting status to skipped

[dr...@broad.mit.edu]

Hi there,

In out test environment we have a number of tests which we quarantine in bamboo. Bamboo does run these tests and since some of them are very long running I wanted to find a way, separate from disabling them to ...disable them.

To this end I created an annotation 'QuarantinedTest' which I put on tests I wish to disable.

To disable the tests, I created a class implementing IMethodInterceptor. In this class I successfully find all the QuarantinedTest annotated tests but the problem is the skipped test count is not correctly reflected in the test summary at the end of the test run. Any help would be greatly appreciated. Here is (some of) the code I use to update the status:

QuarantinedTest quarantinedTest =

    optionalMethod.map(m ->  

    m.getDeclaredAnnotation(QuarantinedTest.class)).orElse(null);

if (quarantinedTest != null) {
    TestResult testResult = 

        TestResult.newTestResultFor(method.getMethod());

    testResult.setStatus(TestResult.SKIP);

    testResult.setContext(context);

    context.getSkippedTests()

        .addResult(testResult, method.getMethod());
} else {
    testsToRun.add(method);
}

[Krishnan Mahadevan]

What version of TestNG are you using ? Can you please check if the problem persists when you use TestNG 7.5 ?

If its still reproducible can you please share a sample project that can be used to reproduce the problem ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

---

From: testng...@googlegroups.com <testng...@googlegroups.com> on behalf of dr...@broad.mit.edu <dr...@broad.mit.edu>
Sent: Friday, January 14, 2022 2:19:04 AM
To: testng-users <testng...@googlegroups.com>
Subject: [testng-users] Quarantine tests and and setting status to skipped

 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/97f41db4-480b-41a0-ac8c-6a16ecd4b0bdn%40googlegroups.com.

[dr...@broad.mit.edu]

Thanks Krishnan for replying. I should have given more information to begin with.

I see the issue with 7.3 and 7.5. When run from maven I don't see skips update, but if I run the test class from IntelliJ the skips does get updated. My tests are configured through @Test annotations while (I think) Intellij produces temporary suite files, so maybe this has to do with suites???

Here's the test code:

public class TestQuarantining {
    @Test
        public void testNotQuarantined(){
        Assert.assertTrue(true);

    }

    @Test
    @QuarantinedTest
        public void testQuarantined(){

        Assert.assertTrue(true);

      }

}
Intellij generated xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
  <test thread-count="5" name="test">
    <classes>
      <class name="org.etc.TestQuarantining"/>
    </classes>
  </test> <!-- test -->
</suite> <!-- Default Suite -->

Intellij output

===============================================
Default Suite
Total tests run: 1, Passes: 0, Failures: 0, Skips: 1
===============================================

maven output:

command: 

mvn test -Dtest=TestQuarantining

...

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.378 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------