Re: [testng-users] How to run dependent test methods as soon as possible

[Cédric Beust ♔]

Hi Dale,

As far as I can tell, TestNG doesn't segregate this way, it just schedules methods as soon as they are "free". For example, if you have "a", "b" and "c" (no dependencies) and "d" that depends on "e", then all the free methods (a, b, c and e) are launched right away.

I described this algorithm in details here:  http://beust.com/weblog/2009/11/28/hard-core-multicore-with-testng/

However, this could be working differently from how I designed it (we call those "bugs" :-)) so if you think you could reproduce your problem on a TestNG-only (no Selenium) example, that would help me investigate.

Thanks.

-- 

Cédric

-- 

Cédric

On Mon, Feb 18, 2013 at 12:36 PM, Dale Emery <da...@dhemery.com> wrote:

TestNG segregates test methods into two batches:  Tests that depend on other tests, and tests that do not depend on other tests.  And it seems to run /all/ of the tests in the second batch before it runs any tests in the first batch.  This creates a problem for one of my clients that I can't figure out how to solve.

Here is our scenario:

1. We are testing a web app using Selenium RC.
2. The Selenium server is on Sauce Labs.
3. Because launching a browser session can take 20 seconds, we launch one session per test class, in a @BeforeClass.  All test methods in the class use the same browser session.
4. We instruct TestNG to use six threads, one test class per thread.
5. Many test classes have chains of dependent test methods.  Each dependency is on another test method in the same class; there are no dependencies from one class to another.

Here is the problem:

1. TestNG runs the @BeforeClass methods for some of the classes (six of them, I think, one per thread), which launches the browser sessions on Sauce Labs.
2. TestNG runs the non-dependent test methods for those classes.
3. The shortest-running test methods finish long before the longest-running non-dependent test methods.
4. The threads for the short-running non-dependent test methods remain idle while TestNG finishes the longer-running non-dependent test methods on the other threads.
5. The threads remain idle even though no thread has any dependency on other classes or threads.
6. While these threads are idle, the Sauce Labs Selenium sessions are also idle.
7. We pay Sauce Labs while the sessions are idle.
8. After several minutes without receiving a Selenium command, Sauce Labs assumes the test has gone awry, and aborts the session.

Because of this, we have had to abandon TestNG's built-in dependency mechanism use our own.  We would prefer to use dependsOnMethods if we could solve the problems I described above.

Is there a way to convince TestNG to run begin running dependent test methods immediately, even though other classes are still running their non-dependent test methods?

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dale Emery]

Hi Cédric,

Here is a sample project that demonstrates how TestNG's scheduling algorithm can inject long delays between a free method and the first dependent method:

https://github.com/dhemery/TestNGDemo

In this project, I simulate long-running free tests by putting the free test to sleep for 1 to 12 seconds.

I can see that I was mistaken about the algorithm.  In a few cases, the dependent methods in one class run before the free methods in another.

Still: The scheduling algorithm is clearly introducing delays between the free method and the dependent method in each class.  The delay even for this simple project can be more than 20 seconds.  The length of the delay seems to increase the more test classes I add.  Imagine these delays being minutes instead of seconds, and imagine that we're holding an expensive resource open for the duration of a class, and you can see the problem.

Dale

[Dale Emery]

Looking through the code, it appears that I can't use priority to solve my problem.  At the start of the run, the entire batch of initially free methods is queued.  Later, as additional methods become free, they go to the end of the queue, even if they are higher priority than the already-queued methods.

[Dale Emery]

I ran this experiment:

1. Change TestRunner to use a PriorityBlockingQueue instead of a LinkedBlockingQueue to queue workers.
   
2. Give each test class a different priority.
3. Give all of the methods within a class the same priority.
   

This makes it possible (if somewhat awkward) to get the effect I want:  When a test method completes, any newly freed methods in that class are queued to run next.

The awkwardness is that this scheme requires me to give every method a priority, and to coordinate priorities across classes.  Better for me (and maybe more dangerous) would be to allow the user to determine queueing order.  That would allow me to (for example) order the queue by class name, which would give the effect I want without requiring me to fiddle with priorities.

So I can think of a few possible changes:

1. Change TestNG to use a PriorityBlockingQueue instead of a LinkedBlockingQueue.
   
2. Use a PriorityBlockingQueue and allow the user to supply a comparator.
   
3. Default to a LinkedBlockingQueue, and allow the user to supply an alternate BlockingQueue implementation.
   

Do any of these appeal?  It seems that option 3 would both retain the current behavior and give the user sufficient control and flexibility.

Dale