[testng-users] @Factory execution order

[Santo Pfingsten]

I use @Factory to create multiple browser tests for Selenium, but currently
the execution order is:

beforeclass(instance1)
beforeclass(instance2)
beforeclass(instance3)
test(instance1)
test(instance2)
test(instance3)
afterclass(instance1)
afterclass(instance2)
afterclass(instance3)

Beforeclass starts a browser and logs into a user account and afterclass
logs out and closes the browser,
so this is a huge problem, since it means that there will be 3 browsers
opened at the same time logged in with the same user.
My goal is to have the instances run one after another:

beforeclass(instance1)
test(instance1)
afterclass(instance1)
beforeclass(instance2)
test(instance2)
afterclass(instance2)
beforeclass(instance3)
test(instance3)
afterclass(instance3)

is there any way to achieve that ?
It doesn't matter in which order the instances (or even the tests) are
executed, just that they are separate

Thanks in advance
--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31495933.html
Sent from the testng-users mailing list archive at Nabble.com.

[Santo Pfingsten]

Seems to be the dependsOn and priority values of the @Test annotation that
causes this behaviour,
so for now I solved it with a method interceptor and custom annotations for
priority.
Using the method interceptor with TestNG's priority or dependsOn set does
not work, so I had to create a custom one.

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31498075.html

[Nalin Makar]

use @[Before|After]Method instead of @XXXClass
-nalin

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Santo Pfingsten]

Thats like saying if your bike is broken use a skateboard.
I need the beforeclass/afterclass as well as the beforemethod/aftermethod.
The reason to create the browser window within beforeclass is so that I
don't have to recreate it for every test-method.
Starting the browser is the slowest task of a test-run with selenium.
starting it for every test method would result in a huge slowdown.

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31503266.html

[Nalin Makar]

Based on you description in the original post, changing the annotations to @XXXMethod will do exactly what you wanted. You didn't mention that you were also using @XXXMethod for some other methods.

Have you considered using a DataProvider?

Can you give a more concrete example of a test class. That'll help.

-nalin

[Santo Pfingsten]

As said in my other post, I found a solution for now, but I will provide a
better example on Monday. (Won't be home at the weekend, and I'm off in 5
minutes)
DataProvider provides data for a method, not the whole class, and besides I
don't think it'll solve the issue of the methods being called
between-instances.
Have a nice weekend.

Santo

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31506381.html

[Jeff]

I'll throw in my experience which is somewhat limited but relevant.  Note:  This is my first exposure to TestNG and Maven (and Selenium for that matter) ... so I'm still learning.

 

I'm using Selenium2 with TestNG and Maven to build a test framework within NetBeans 6.9.1.  I haven't used TestNG from the commandline yet to define and run TestSuites yet so i don't things may change once that happens.  I currently only run/debug single test classes from within the IDE.

I also was concerned about how ot handle multiple selenium instances since it is an expensive operation to spin up the browsers.  I ended up defining it such that there was a 1:1 association between a test class instance and a selenium object.

 

All my test classes inherit from a common SeleniumHelper base class that has my @Factory method.  The @Factory method loads a properties file based on the maven profile in use.  It then creates instances of the test class using information from the properties file.

 

This simplified the test classes and eliminates the need for subclasses to know anything about creating the selenium instances.   Behavior and browser configuration is controlled via a properties file selected using a Maven profile.

 

Each test class INSTANCE shares a WebDriver(and WebDriverBackedSelenium) object and thus the same browser instance. 

 

The next problem was that TestNG (within NetBeans at least) seemed to run my test methods in parallel even when running/debugging a single test class.  This must NOT happen with a shared selenium/browser instance.  The test methods were stomping all over each other. Sort of reminded me when my kids were little and would hit the keyboard or mouse buttons while I was trying to type...it just doesn't work.

 

To fix it I first require that each test class instance run test methods SEQUENTIALLY using the @Test(singleThreaded = true) annotation on the CLASS.  This is a MUST for testing with selenium, unless you start creating/managing multiple selenium instances.  However, this 1:1 relationship at the class level is very natural and works well so far.

 

When test methods were autonomous, this is all that was needed since the execution order didn't matter.  However, I tend to like smaller, more granular test methods that perform individual steps in a multi-step process and thus want run tests in a guaranteed order for a given browser session.  To do this, I used the @Test(dependsOnMethods = { ... } ) on each method in the sequence.  However, this is very brittle and hard to manage (Hint: I'm open for suggestions on a more maintainable method to guarantee ordering of test methods within a test class as opposed to hard coding the dependency relationship or relying on the .XML testsuite file). 

 

Here is my factory method.  I'm happy to share more detail if needed...just ping me.  I have a 'dev' mode that creates only one test class instance for dev/debugging purposes and defaults to FIREFOX36.  Otherwise, it takes a comma-separated list of browsers and will create a test class instance for each browser specified (same or different browsers) and an associated selenium instance.  So far it works well but there are improvements I need to make to it still:

 

  @Factory
  public Object[] testFactory() {
    ArrayList<Object> tests = new ArrayList<Object>();
    Object test = null;

    try {
      String[] browsers = config.getProperty("browser", "FIREFOX36").split(",");

      //Create a single Firefox instance for dev/debugging
      if (config.getProperty("devMode", "true").toLowerCase().equals("true")) {
        try {
          test = this.getClass().newInstance();
          ((SeleniumHelper) test).startSelenium(Browser.valueOf(browsers[0]), BASE_URL);
          tests.add(test);
        } catch (InstantiationException iex) {
          //Handle this
        } catch (IllegalAccessException iaex) {
          //Handle this
        }
      } else {
        // Generate test objects for all specified browser types, for the specified reference
        for (String bStr : browsers) {
          Browser b = Browser.valueOf(bStr);
          try {
            test = this.getClass().newInstance();
            ((SeleniumHelper) test).startSelenium(b, BASE_URL);
            tests.add(test);
          } catch (Exception ex) {
             logger.log(Level.SEVERE, "[ERROR] Creating test class: " + this.getClass().getName(), ex);
          }
        }
      }
    } catch (Exception ex) {
      logger.log(Level.SEVERE, "ERROR: no profile specified for class: " + this.getClass().getName(), ex);
      //don't rethrow in order to return empty array.
    }

    return tests.toArray();
  }

Jeff Vincent
preda...@gmail.com
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Cédric Beust ♔]

Hi Jeff,

Thank you so much for this very detailed message, this is by far the most helpful contribution I have seen regarding the usage of TestNG and Selenium so far! To everyone else using Selenium: please comment and add to the discussion, I'd love to get more of this documented officially.

A few thoughts:

TestNG should not be running your tests in parallel by default, so even though you worked around this problem by using @Test(singleThreaded=true), you might want to try to find out what is calling TestNG in parallel mode (I suggest NetBeans? Or maybe your own testng.xml file, if you're using one?). Understanding what's going on might save you some trouble down the line.

As you correctly pointed out, the only reliable way to order your test methods is either dependsOnMethods (Java) or preserve-order (XML). Like you, I would be happy to hear about alternatives that people would find superior to these two approaches.

Jeff, would you be interested in expanding these instructions into a more detailed "Selenium+TestNG HOWTO"? I would be happy to either link to it or even include it directly in the Selenium section of the TestNG documentation. Feel free to email me directly if you want to chat about it.

Thanks again for taking the time to share your experience.

-- 

Cédric

[Jeff]

I'd be happy to contribute to a HOW-TO.  I can't guarantee that what I'm doing is in the vicinity of "best-practices" though ;-),

 

The parallel thing is likely related to the maven-surefire-plugin, but that is an entirely new discussion topic I'm still figuring out as well.  I'm still unsure where NetBeans control ends and Maven begins, particularly when launching a debugging session for a test class.  The fact that I'm working at all is a miracle somedays! :)  I haven't found an option in NetBeans that controls TestNG behavior directly.

 

Regardless, I like that the test class is explicitly annotated as single threaded.  Hopefully that will force it to be single threaded regardless. 

 

As far as preserving order in the code, here are a couple of thoughts to toss around:

1. I've used reflection a fair amount but am unsure right off if there is any way to query for methods based on the defined order in the class.  If there is a way to do that, maybe an annotation option at the class level to trigger run order based on defined ordering of the methods (the way the code reads).
1. Maybe this could happen automatically if singlethreaded=true is used?
2. A class-level annotation option to define a sorting mechanism based on method name.
3. Class member annotation on a data structure (array, sorted map, etc.) listing the method names and their ordering.  Not dynamic, but centralizes the management.
4. Class level annotation option (runOrder = {}) that contains the list of method names an the order to execute them.  This is also not dynamic but centralizes the management.

I very much prefer dynamic mechanisms that minimize extra maintainance of option 1) and 2) above.  Running methods based on defined order would be nice since it mirrors the way the code reads.  Being able to define a generic sorting mechanism seems appealing as well.

 

Thanks!

 

2011/4/29 Cédric Beust ♔ <ced...@beust.com>

[Scal]

I didn't had the need to explore @Factory for my projects yet, maybe
it would actually be of use in my current one?
I hope not to "hijack" the topic though.

My project setup and functional requirement are like this (using
JAVA):
- A parallel runner which starts and stop Selenium server, creating a
new instance with @Before/After Test
- A test file with only one @Test method which will run Selenium
commands stored in a storage engine (MS SQL in my case)
- The database is designed to have such table structure:
--- projects
--- test suites
--- test case
--- Selenium actions + parameters
All actions are linked to test case(s) (1 to n)
All test case(s) are linked to test suite(s) (1 to n)
All test suites(s) are linked to project(s) (1 to n)

The @Test method basically loop through the projects table and get
it's (I pass a project ID parameter to the test method to know which
actions I need to run) test suites, test cases and actions.
Because I use a single test method, I have implemented my own
reporting instead of testNG's one. This because I don't want a whole
project to fail whenever an assertion on an action fails. My @test
never actually fails, I try{} catch{} everything and store failures in
a log table, which is the base of data to build the report after a
project has ran.
Little addition to the description is that since I manage all actions/
data dynamically through a DB, I don't have to worry about preserver-
order or dependsOnMethods since I build my dependencies at the data
storage level.

Cedric, if you'd like a more specific description and how things work
in my code for adding it into examples/documentation, let me know.
Although the use of testNG itself is limited in my case, it still
takes advantage of parallelism at least :)
My Java knowledge is a bit limited, so code would maybe not be best
example, but the idea behind the "framework" I build could help some
users of Selenium + testNG?

Cheers;

[Santo Pfingsten]

Nevermind the factory inside a class, I just found that it is a good idea to
keep it in a separate class so you can start the test with just one browser
instead of all.

Santo Pfingsten wrote:
>
> I wonder.. how did you get the @Factory to work inside your test class ?
> I had to create a separate class for the factory, otherwise it wouldn't
> work.
>
> I'm using NetBeans too and don't have MultiThreading (altho I thought it
> might be multithreaded when the problem described in the topic was
> happening, but it was just tests running mixed together).
>
> As for the ordering, I use a custom annotation @Priority in combination
> with an IMethodInterceptor.
> (sorting the methods by browser, class name and then priority)
> dependsOn* and priority of the @Test would not be instance-safe.
>
> Thanks for the detailed input, it was very interesting.
> I'm doing it kinda similar.
> I have a base class, which does all the selenium specific stuff like
> starting a browser, write screenshots/failure dumps, loading various
> .properties files for configuring the tests, etc.
> Then I have an Assert class which inherits from the base class and adds
> all the assert/verify/fail methods for convenience.
> Then I have a TestNG class, which simply adds an @afterclass to close the
> browser.
> That's the one every test inherits from.
> And then for every test I create an @factory class to start one instance
> for each browser (browsers loaded from a common properties file).
> If I could skip the factory class it would really be great, but all I
> found on google was that you'd have to create your own class for the
> @factory.

>> methods SEQUENTIALLY using the *@Test(singleThreaded = true)* annotation

>> on
>> the CLASS. This is a MUST for testing with selenium, unless you start
>> creating/managing multiple selenium instances. However, this 1:1
>> relationship at the class level is very natural and works well so far.
>>
>> When test methods were autonomous, this is all that was needed since the
>> execution order didn't matter. However, I tend to like smaller, more
>> granular test methods that perform individual steps in a multi-step
>> process
>> and thus want run tests in a guaranteed order for a given browser
>> session.

>> To do this, I used the *@Test(dependsOnMethods = { ... } )* on each

>> method
>> in the sequence. However, this is very brittle and hard to manage

>> *(Hint:

>> I'm open for suggestions on a more maintainable method to guarantee
>> ordering
>> of test methods within a test class as opposed to hard coding the
>> dependency

>> relationship or relying on the .XML testsuite file).*

>>
>> Here is my factory method. I'm happy to share more detail if
>> needed...just
>> ping me. I have a 'dev' mode that creates only one test class instance
>> for
>> dev/debugging purposes and defaults to FIREFOX36. Otherwise, it takes a
>> comma-separated list of browsers and will create a test class instance
>> for
>> each browser specified (same or different browsers) and an associated
>> selenium instance. So far it works well but there are improvements I
>> need
>> to make to it still:
>>
>> @Factory
>> public Object[] testFactory() {

>> ArrayList tests = new ArrayList();

>> I ♥ DropBox <http://db.tt/9O6LfBX> !!

>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "testng-users" group.
>> To post to this group, send email to testng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> testng-users...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/testng-users?hl=en.
>>
>>
>>
>
>

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31523767.html

[Santo Pfingsten]

> methods SEQUENTIALLY using the *@Test(singleThreaded = true)* annotation

> on
> the CLASS. This is a MUST for testing with selenium, unless you start
> creating/managing multiple selenium instances. However, this 1:1
> relationship at the class level is very natural and works well so far.
>
> When test methods were autonomous, this is all that was needed since the
> execution order didn't matter. However, I tend to like smaller, more
> granular test methods that perform individual steps in a multi-step
> process
> and thus want run tests in a guaranteed order for a given browser session.

> To do this, I used the *@Test(dependsOnMethods = { ... } )* on each method
> in the sequence. However, this is very brittle and hard to manage *(Hint:

> I'm open for suggestions on a more maintainable method to guarantee
> ordering
> of test methods within a test class as opposed to hard coding the
> dependency

> relationship or relying on the .XML testsuite file).*

>
> Here is my factory method. I'm happy to share more detail if
> needed...just
> ping me. I have a 'dev' mode that creates only one test class instance
> for
> dev/debugging purposes and defaults to FIREFOX36. Otherwise, it takes a
> comma-separated list of browsers and will create a test class instance for
> each browser specified (same or different browsers) and an associated
> selenium instance. So far it works well but there are improvements I need
> to make to it still:
>
> @Factory
> public Object[] testFactory() {

> ArrayList tests = new ArrayList();

> I ♥ DropBox <http://db.tt/9O6LfBX> !!

>
> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31522900.html

[Santo Pfingsten]

Here's the example:
http://nopaste.info/705a97c366.html

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31522795.html

[Jeff]

By separate do you mean outside the inheritance chain?

 

In your Demo example, how would you handle setup/configuration if the test class is being created independent of the @Factory?

 

I started with the @Factory method in the test class but was able to abstract it down to the base class.  I played with making the @Factory method static as well as keeping it instance specific and it seems to work either way.  Regardless, it is a method that is still part of the test class via inheritance.

 

The way it is set up now, I don't need anything in the test class beyond extending the base class and adding any needed @Test annotations.  The @Factory behavior is controlled via a maven profile which triggers loading the associated properties file.  There are two settings in the properties file (among others):

browser=[FIREFOX36|IE8|CHROME|HtmlUnit]

devMode=[true|false]

If you want to run the same test in parallel in the same browser (I just found a server-side concurrency bug in our software by doing this), you could do:

browser=FIREFOX36,FIREFOX36,FIREFOX36,FIREFOX36  //4 instances in parallel.

devMode=false

or

browser=FIREFOX36,IE8,CHROME

devMode=false

If I then need to restrict it to a single browser for debugging, change devMode=true.  I could have just listed one value for the 'browser' (and that does work), but chose to add a switch to enable/disable loading of multiple browsers for debugging at the @Factory. 

 

Eventually, devMode will crank up logging Levels, enable debug flags, load more plugins and could do other things to enhance development and debugging tasks as opposed to just limiting the scope of browser instantiation.

 

If I understand it correctly, the @Factory method (by virtue of its mere presence) forces test class object creation to always go through the factory as the single entry point (the whole point of @Factory?).  This means for my tests, the class itself will never be instantiated and run (as a test) without first going through the @Factory.  This is good in my current implementation since I need configuration information during instantiaton at run-time and didn't know of a better way to feed it those values.

 

If the @Factory is dissociated with the class, then it would seem other steps must be taken to handle test class instantiation for all other cases, causing a bit more headache (depending on needs).

Jeff Vincent

preda...@gmail.com

See my LinkedIn profile at:

http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

 

[Cédric Beust ♔]

On Mon, May 2, 2011 at 6:51 AM, Santo Pfingsten <s...@tetralog.de> wrote:

Nevermind the factory inside a class, I just found that it is a good idea to
keep it in a separate class so you can start the test with just one browser
instead of all.

Please note that it is no longer necessary to create a separate method/class for a factory, here is how you do it.

-- 

Cédric

[Santo Pfingsten]

Yes, I've seen the topic, but I'm still wondering how he got it working
without the new feature.

> it.<http://groups.google.com/group/testng-users/browse_thread/thread/9c21b07009904aef>
>
> --
> Cédric

>
> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/%40Factory-execution-order-tp31495933p31530392.html

[Roughael]

By separate I meant one class having multiple instances, but an
instance is not tested in a single run, but has to wait for other
instances to finish the same test methods first.
i.e. if a test method depends on another test method, that test method
has to be run on all instances of the class before the next test
method can be run.

> In your Demo example, how would you handle setup/configuration if the test
> class is being created independent of the @Factory?

I just use a default constructor

I created the @Factory in one class with my @Test and TestNG would
always choose the @Test over the @Factory.
Maybe it was because I started the tests via rightclick->test file

> >http://old.nabble.com/%40Factory-execution-order-tp31495933p31495933....

> > >> >> Sent from the testng-users mailing list archive at Nabble.com.
>
> > >> >> --
> > >> >> You received this message because you are subscribed to the Google
> > >> Groups
> > >> >> "testng-users" group.
> > >> >> To post to this group, send email to testng...@googlegroups.com.
> > >> >> To unsubscribe from this group, send email to
> > >> >> testng-users...@googlegroups.com.
> > >> >> For more options, visit this group at
> > >> >>http://groups.google.com/group/testng-users?hl=en.
>
> > >> > --
> > >> > You received this message because you are subscribed to the Google
> > >> Groups
> > >> > "testng-users" group.
> > >> > To post to this group, send email to testng...@googlegroups.com.
> > >> > To unsubscribe from this group, send email to
> > >> > testng-users...@googlegroups.com.
> > >> > For more options, visit this group at
> > >> >http://groups.google.com/group/testng-users?hl=en.
>
> > >> --
> > >> View this message in context:
>

> >http://old.nabble.com/%40Factory-execution-order-tp31495933p31503266....

> > >> Sent from the testng-users mailing list archive at Nabble.com.
>
> > >> --
> > >> You received this message because you are subscribed to the Google
> > Groups
> > >> "testng-users" group.
> > >> To post to this group, send email to testng...@googlegroups.com.
> > >> To unsubscribe from this group, send email to
> > >> testng-users...@googlegroups.com.
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/testng-users?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "testng-users" group.
> > > To post to this group, send email to testng...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > testng-users...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/testng-users?hl=en.
>
> > --
>
> >  View this message in context:

> >http://old.nabble.com/%40Factory-execution-order-tp31495933p31522795....

> > Sent from the testng-users mailing list archive at Nabble.com.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > testng-users...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
>
> Jeff Vincent

> predato...@gmail.com

> See my LinkedIn profile at:http://www.linkedin.com/in/rjeffreyvincent

> I ♥ DropBox <http://db.tt/9O6LfBX> !!