What is the expected behaviour if a test method timeout occurs in a class with multiple test methods?

[robbiewinston]

Hi,

I see strange behaviour when a test timeout occurs within a class with multiple test methods.  The classes are also being run in parallel (classes NOT methods).

When a test timeout occurs I appear to "lose" tests.  My out reports the failed tests but if I add the number of failed tests to the number of passed tests, then the this is less then the actual number of tests.  There are no tests reported as skipped.

If a timeouit occurs in a test method, is the entire class "killed"?

Thanks

Robbie

[Cédric Beust ♔]

The entire class would become skipped if a @BeforeClass or a @BeforeMethod failed. If you are seeing something, feel free to post a code example.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[robbiewinston]

Cedric,

I want to achieve the following but am struggling to find a combination of annotations and configuration to achieve it;

Set up; 

- Classes contain multiple test methods

- Classes have @BeforeClass, @AfterClass methods which interact with private properties which are then referenced in each test;

-Classes extend a base test class with  @BeforeMethod and @AfterMethod inherited

- All test methods are annotated with @Test(timeOut = 300000)

- Tests run via Maven surefire plugin with;

<parallel>classes</parallel>

                    <threadCount>5</threadCount>

                    <perCoreThreadCount>true</perCoreThreadCount>

-Maven run via TeamCity on MacMini server

- Test results written by ReportNG listener

Aim;

If @BeforeClass fails, do not run any tests but mark all tests as skipped

if @BeforeTest fails, mark that test as failed or skipped but continue to next test method in class and re-run @BeforeTest (OR mark all tests in that class as skipped)

If test time out occurs after 5 minutes of test method running, mark that test as failed or skipped BUT continue to run other test methods in that class (OR mark all other tests in that class as skipped)

If error occurs during @AfterTest, still run other tests in class

Can this be achieved? I haven't worked out the pattern yet, but if tests timeout, then subsequent tests in that class are neither completed nor marked as skipped, so the total number of tests in the results is incorrect. My minimum requirement is that the total number of tests is correct even if subsequent tests are marked as skipped although ideally I would like them to run as well.

Interestingly, if I look at the output which contains thread info (sorry, I can't remember the name of the html file, and don't have access to TeamCity at the moment), then I can see a subsequent test start in the same thread as the timeout test but never finish and that thread isn't used again.

Is the problem that Timeout closes the thread, therefore making subsequent tests unable to run?

Thanks

Robbie

[Yevhen Bilevych]

Hi Robbie,

As if your initial problem description, you may struggle with the issues I faced somehow:
- If @BeforeClass fails, all methods of current class get skipped. Unfortunately, all methods of all classes with common parent class are also skipped unless configfailurepolicy="continue" is set
- If method fails, dependent methods of the same class get skipped. Unfortunately, if the failed method is defined in the common base class, methods from other derived classes with the same dependency are also skipped.

Both cases, I believe are design bugs that dependency rules are applied to the (base) class, not to the instance.
Haven't verified these with latest versions.

As of items of you aim, please see answers inline.

Thanks,
Yevhen

On 05/21/2013 11:23 AM, robbiewinston wrote:

Cedric,

I want to achieve the following but am struggling to find a combination of annotations and configuration to achieve it;

Set up; 

- Classes contain multiple test methods

- Classes have @BeforeClass, @AfterClass methods which interact with private properties which are then referenced in each test;

-Classes extend a base test class with  @BeforeMethod and @AfterMethod inherited

- All test methods are annotated with @Test(timeOut = 300000)

- Tests run via Maven surefire plugin with;

<parallel>classes</parallel>

                    <threadCount>5</threadCount>

                    <perCoreThreadCount>true</perCoreThreadCount>

-Maven run via TeamCity on MacMini server

- Test results written by ReportNG listener

Aim;

If @BeforeClass fails, do not run any tests but mark all tests as skipped

It is behavior by default except cases above.

if @BeforeTest fails, mark that test as failed or skipped but continue to next test method in class and re-run @BeforeTest (OR mark all tests in that class as skipped)

@BeforeTest refers to the tag <test/> in testng xml. It's abstraction level between <suite> and class.

If test time out occurs after 5 minutes of test method running, mark that test as failed or skipped BUT continue to run other test methods in that class (OR mark all other tests in that class as skipped)

Does @Test(timeOut=600) work for you? Again, dependency matters.

If error occurs during @AfterTest, still run other tests in class

This also may be default behavior.

Can this be achieved? I haven't worked out the pattern yet, but if tests timeout, then subsequent tests in that class are neither completed nor marked as skipped, so the total number of tests in the results is incorrect. My minimum requirement is that the total number of tests is correct even if subsequent tests are marked as skipped although ideally I would like them to run as well.

Interestingly, if I look at the output which contains thread info (sorry, I can't remember the name of the html file, and don't have access to TeamCity at the moment), then I can see a subsequent test start in the same thread as the timeout test but never finish and that thread isn't used again.

Is the problem that Timeout closes the thread, therefore making subsequent tests unable to run?

Thanks

Robbie