@AfterClass issue??

[Jeff]

I have test classes with some defined groups ("L1", "L2", "L3") such as:

@Test(singleThreaded = true, groups = {"L3"})

public class MyTest {...}

I also have a method annotated with @AfterClass(alwaysRun = true) that cleans up after the tests complete.

 

I am using Maven + maven-surefire plugin to run the tests.

 

If I run tests and specify a group such as:

 

mvn test -Ptest -Dgroups=L1

 

It still runs the @Factory for the classes annotated as "L2", "L3", etc. even though it won't run any test cases from that class.  This causes it to use system resources I don't need.  Additionally, it never calls my @AfterClass(alwaysRun = true) even though it says 'alwaysRun'.

 

Am I doing it wrong?

 

I added @AfterSuite(alwaysRun = true) and that seems to work for now but I would have expected the other to work as well.

 

Even if this is a bug, how do I get TestNG to skip calling my @Factory if there are no test cases to be run?  Each test class instance sets up a Selenium2 browser instance which is expensive and resource intensive and I want to avoid that if it won't be used.

 

Thanks!!

--

Jeff Vincent
preda...@gmail.com
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Cédric Beust ♔]

Hi Jeff,

@Factory methods are always run because they create test instances (which are then tested to make sure they belong to the groups being run).

Right now, @Factory methods don't receive any injections but I could improve this by allowing them to receive the current ITestContext. Then you can know which groups are being included and return a different set of instances in this case.

Would this help?

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Jeff]

It might...I need to understand ITestContext...that's yet another new area for me. 

 

I had assumed that TestNG looked at the class and methods before calling @Factory or attempting to instantiate anything but now that I think about it, that was not a valid assumption, given that the @Factory doesn't necessarily have to be part of the class being returned.

 

If using ITestContext, am I understanding it right that I'll have to manually search walk through all classes/methods to determine if they have any matching annotations with the specified group(s) defined?  If so, that sounds ... well not fun.  I will look for a different way unless I'm missing something (again).

 

Regardless, is it a bug that @AfterClass(alwaysRun=true) doesn't get run, even with no tests get executed?

 

2011/5/20 Cédric Beust ♔ <ced...@beust.com>

[Cédric Beust ♔]

Maybe something that will work better in your case would be for me to add a `groups` attribute to @Factory. Would this solve your problem?

In this case, TestNG would only run a @Factory method if it is part of a group you are launching.

The solution I was suggesting with ITestContext injection was more general in the sense that you can decide to run a factory or not based on a much broader criterion than "Are we running group 'foo'?".

As for your second question, can you show me a small test case where @AfterClass(alwaysRun = true) is not being run?

Thanks.

-- 

Cédric

[Jeff]

Thanks for the clarification.

 

My @Factory is currently defined in my base class but the groups are being defined on the subclass so  @Factory(groups = ... ) won't help in my case. 

 

As an alternative, what if TestNG checks the parent class containing the @Factory.  If it has 'groups' defined, but doesn't have a matching group then skip running @Factory?  (I haven't thought through this in any depth).

 

As for the @BeforeClass/@AfterClass issue, I've attached a simplified test class.  Here are the results when I ran it locally (maven):

 

RUN WITH MATCHING GROUP:

C:>mvn test -test=TestAfterClass -Dgroups=L3

 

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
In ctor(): RC_DEFAULT
In myFactory(): RC_DEFAULT
In ctor(): RC_DEFAULT
In initResources(): RC_ZERO
In ctor(): RC_DEFAULT
In initResources(): RC_ONE
In runBefore(): RC_ZERO
In testCaseOne(): RC_ZERO
In runAfter(): RC_ZERO
In runBefore(): RC_ONE
In testCaseOne(): RC_ONE
In runAfter(): RC_ONE
Tests run: 4, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.436 sec

Results :

Tests run: 4, Failures: 0, Errors: 0, Skipped: 2

 RUN WITH NON-MATCHING GROUP

C:>mvn test -test=TestAfterClass -Dgroups=L1

 

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
In ctor(): RC_DEFAULT
In myFactory(): RC_DEFAULT
In ctor(): RC_DEFAULT
In initResources(): RC_ZERO
In ctor(): RC_DEFAULT
In initResources(): RC_ONE
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.46 sec
There are no tests to run.

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Note that runBefore() and runAfter() are missing for both class instances if the group didn't match.  In general someone could theoretically set up resources in the constructor and/or @Factory method that needs to be freed when the class is done (as in my case) with the Selenium2 object.  If the class is instantiated at all, then I would have expected the before and after methods to run regardless (due to the alwaysRun flag).

 

I can't explain why there are 2 skipped in the first case.  The run count is right.

 

Hopefully this makes sense.

 

Thanks for your willingness to help!

 

2011/5/20 Cédric Beust ♔ <ced...@beust.com>

[Cédric Beust ♔]

On Fri, May 20, 2011 at 3:21 PM, Jeff <preda...@gmail.com> wrote:

Thanks for the clarification.

 

My @Factory is currently defined in my base class but the groups are being defined on the subclass so  @Factory(groups = ... ) won't help in my case. 

Ok.

 

 

As an alternative, what if TestNG checks the parent class containing the @Factory.  If it has 'groups' defined, but doesn't have a matching group then skip running @Factory?  (I haven't thought through this in any depth).

That's a bit too magical to my taste. The condition you use to decide whether the factory should run or not is quite subtle (depends on what groups the test classes in the subclass belong to), so I strongly suggest you take another look at the suggestion I made earlier (grab the ITestContext, look up what groups are currently being run and run the correct factory).

If you're running a group and your configuration methods don't belong to that group (and they don't set alwaysRun=true), then they won't be run. I think that's what's happening here.

-- 

Cédric

[Jeff]

The magic is really trying to answer the question, "How can I control @Factory execution based on groups AND be able to handle cases where there is inheritance/subclassing involved?"

 

Here is a simple example of how my @Factory is inherited and used:

 

@Test(groups={"L2"})

class Test1 extends TestBase { ... }

 

@Test(groups={"L2"})

class Test2 extends TestBase { ... }

 

@Test(groups={"L3"})

class Test3 extends TestBase { ... }

 

class TestBase {

  @Factory()

  public void myFactory { ... }  //shared by all subclasses via inheritance

 

  @AfterClass (alwaysRun=true) 

  public void cleanup() { ... }   //Should ALWAYS run...even if no test cases are run?

 }

Right now, my @Factory method runs for Test1, Test2 and Test3.

 

I really only want @Factory to run for Test3.  The end result is that the @Factory for Test1 and Test2 allocates resources that will never get used.

 

In addition, I have an inherited @AfterClass(alwaysRun=true) method that doesn't get called for Test1 and Test2, despite being set to 'alwaysRun=true'.

 

I *could* use ITestContext and manually read the groups annotations to conditionally create test classes in my @Factory, but then I feel like I'm circumventing TestNG's natural order  and creating a maintenance problem that will confuse people because it won't behave in a TestNG documented way.

 

Can we look at finding a mechanism to have TestNG extend conditional, group-based execution to the @Factory that works at the class-level?  Your initial suggestion to add  support for @Factory( groups = {...} ) might be the right direction if it can somehow be applied at the class level as well somehow.

 

I'm curious if others could benefit from this.

 

Thanks!

 

2011/5/23 Cédric Beust ♔ <ced...@beust.com>

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Cédric Beust ♔]

Isn't the problem that you are putting your factory in a base class even though it should only be run by certain subclasses?

Why not simply put your factory in the same class(es) that have a @Test(groups="L3") annotation?

-- 

Cédric

[Jeff]

Because it is boiler plate code that doesn't change currently between test classes.  I don't want to duplicate it. 

On May 23, 2011 4:08 PM, "Cédric Beust ♔" <ced...@beust.com> wrote:
> Isn't the problem that you are putting your factory in a base class even
> though it should only be run by certain subclasses?
>
> Why not simply put your factory in the same class(es) that have a
> @Test(groups="L3") annotation?
>
> --
> Cédric
>
>
>
>
> On Mon, May 23, 2011 at 2:46 PM, Jeff <preda...@gmail.com> wrote:
>
>> The magic is really trying to answer the question, "How can I control
>> @Factory execution based on groups AND be able to handle cases where there
>> is inheritance/subclassing involved?"
>>
>> Here is a simple example of how my @Factory is inherited and used:
>>
>> @Test(groups={"L2"})

>> class Test1 extends *TestBase* { ... }
>>
>> @Test(groups={"L2"})
>> class Test2 extends *TestBase* { ... }
>>
>> @Test(groups={"L3"})
>> class Test3 extends *TestBase* { ... }
>>
>> class *TestBase* {
>> @*Factory*()

>> public void myFactory { ... } //shared by all subclasses via inheritance
>>

>> @*AfterClass* (*alwaysRun*=true)

>> public void cleanup() { ... } //Should ALWAYS run...even if no test
>> cases are run?
>> }
>> Right now, my @Factory method runs for Test1, Test2 and Test3.
>>
>> I really only want @Factory to run for Test3. The end result is that the
>> @Factory for Test1 and Test2 allocates resources that will never get used.
>>

>> In addition, I have an inherited *@AfterClass(alwaysRun=true)* method that

>> doesn't get called for Test1 and Test2, despite being set to
>> 'alwaysRun=true'.
>>
>> I *could* use ITestContext and manually read the groups annotations to
>> conditionally create test classes in my @Factory, but then I feel like I'm
>> circumventing TestNG's natural order and creating a maintenance problem
>> that will confuse people because it won't behave in a TestNG documented way.
>>
>> Can we look at finding a mechanism to have TestNG extend conditional,
>> group-based execution to the @Factory that works at the class-level? Your

>> initial suggestion to add support for *@Factory( groups = {...} )* might
>> be the right direction if it can *somehow be applied at the class level *as

>> well somehow.
>>
>> I'm curious if others could benefit from this.
>>
>> Thanks!
>>
>> 2011/5/23 Cédric Beust ♔ <ced...@beust.com>
>>
>>>
>>>
>>>
>>> On Fri, May 20, 2011 at 3:21 PM, Jeff <preda...@gmail.com> wrote:
>>>
>>>> Thanks for the clarification.
>>>>
>>>> My @Factory is currently defined in my base class but the groups are
>>>> being defined on the subclass so @Factory(groups = ... ) won't help in my
>>>> case.
>>>>
>>>
>>> Ok.
>>>
>>>
>>>>
>>>> As an alternative, what if TestNG checks the parent class containing the
>>>> @Factory. If it has 'groups' defined, but doesn't have a matching group
>>>> then skip running @Factory? (I haven't thought through this in any depth).
>>>>
>>>
>>> That's a bit too magical to my taste. The condition you use to decide
>>> whether the factory should run or not is quite subtle (depends on what
>>> groups the test classes in the subclass belong to), so I strongly suggest
>>> you take another look at the suggestion I made earlier (grab the
>>> ITestContext, look up what groups are currently being run and run the
>>> correct factory).
>>>
>>>
>>>
>>>> As for the @BeforeClass/@AfterClass issue, I've attached a simplified
>>>> test class. Here are the results when I ran it locally (maven):
>>>>

>>>> *RUN WITH MATCHING GROUP:*

>>>>
>>>> C:>mvn test -test=TestAfterClass -Dgroups=L3
>>>>
>>>> -------------------------------------------------------
>>>> T E S T S
>>>> -------------------------------------------------------
>>>> Running TestSuite
>>>> In ctor(): RC_DEFAULT
>>>> In myFactory(): RC_DEFAULT
>>>> In ctor(): RC_DEFAULT
>>>> In initResources(): RC_ZERO
>>>> In ctor(): RC_DEFAULT
>>>> In initResources(): RC_ONE

>>>> In *runBefore*(): RC_ZERO
>>>> In testCaseOne(): RC_ZERO
>>>> In *runAfter*(): RC_ZERO
>>>> In *runBefore*(): RC_ONE
>>>> In testCaseOne(): RC_ONE
>>>> In *runAfter*(): RC_ONE

>>>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.436 sec
>>>> Results :
>>>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 2
>>>>

>>>> *RUN WITH NON-MATCHING GROUP*

>> I ♥ DropBox <http://db.tt/9O6LfBX> !!

>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "testng-users" group.
>> To post to this group, send email to testng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> testng-users...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/testng-users?hl=en.
>>
>

[Cédric Beust ♔]

On Mon, May 23, 2011 at 3:11 PM, Jeff <preda...@gmail.com> wrote:

Because it is boiler plate code that doesn't change currently between test classes.  I don't want to duplicate it. 

You don't have to duplicate it but maybe you could branch out your hierarchy a bit more.

Superclass1

   Superclass2 with factory

     L3

   Superclass 3 without factory

      L1

      L2

-- 

Cédric

[Jeff]

None of my test classes run without first being created and initialized through the factory.  The factory sets up the Selenium instances, determines which test environment (base URL, languages, type of browser to use , etc.) to use and creates the test class instances based on that information. 

I don't want to do all that work if the class is excluded because it is part of a different group.

[Cédric Beust ♔]

How about putting the factory in a totally different class, then? It doesn't have to even belong in the same hierarchy as your tests, all it does is create test instances.

-- 

Cédric

[Jeff]

That won't change the issue... I still need to either 1) only run the @Factory for classes that have the correct group defined, or 2) manually use ITestContext to do limit test instance creation based on the defined  groups. 

As you pointed out, option 2) can be done now but requires me to deviate from native TestNG behavior. 

I'm simply wondering if TestNG could be enhanced to do for @Factories at the class level, what it already does for class level @Test annotations.

[Jeff]

Our discussion took a bit of a tangent on the @Factory issue. 

 

Coming back to the @AfterClass(alwaysRun=true) scenario.  The documentation states for the 'alwaysRun' parameter:

"For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.

For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped." (See TestNG Annotations)

In the attached sample, none of the @Before or @After methods are being run when a matching group is not set/found on the class or methods.  If I'm reading the documentation correctly, then this seems to be a bug.

 

Thoughts?