Unable to execute tests in parallel

3,152 views
Skip to first unread message

Tarun Bhadauria

unread,
Jan 22, 2012, 2:40:39 PM1/22/12
to testng...@googlegroups.com
TestNG - 6.3.1

testng.xml -

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Test Suite" verbose="5">

        <parameter name="appURL" value="http://www.test.com/default.aspx"></parameter>
        <parameter name="locale" value="us"></parameter>

        <test name="test in Firefox" parallel="classes"
                thread-count="3">
                <parameter name="browser" value="FF"></parameter>
                <classes>
                        <class name="com.test.SanityTest"></class>
                        <class name="com.test.SignupTest"></class>
                        <class name="com.test.LoginTest">                    
                        </class>
                </classes>
        </test>
</suite>

Despite I have "parallel= classes" set at test tag level all tests are executed sequentially.
Did I miss something obviously wrong?

~tarun

Cédric Beust ♔

unread,
Jan 23, 2012, 12:05:47 AM1/23/12
to testng...@googlegroups.com
How are you determining this, are you printing the id of the current thread? Please post your source.

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/N12cB0vPyu4J.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

Tarun Bhadauria

unread,
Jan 23, 2012, 1:09:16 AM1/23/12
to testng...@googlegroups.com
here is the cut down version of test -

Base class which is extended by all test classes -

public class SelTestCase {

        protected WebDriver driver;
        private static final int MAX_WAIT_PERIOD = 20;
        private String appURL;
        private String locale;

        @BeforeClass(alwaysRun=true)
        @Parameters( { "appURL", "locale" })
        public void setTestBed(String appURL, String locale) {
                this.appURL = appURL;
                this.locale = locale;
        }

        @BeforeMethod(alwaysRun=true)
        @Parameters({"browser", "chromedriver"})
        public void startDriver(@Optional("Chrome") String browser, @Optional String chromeLocation) {
                if(browser.equalsIgnoreCase("FF")) {
                        driver = new FirefoxDriver();                   
                } else if (browser.equalsIgnoreCase("IE")) {
                        driver = new InternetExplorerDriver();
                } else if(browser.equalsIgnoreCase("Chrome")) {
                        System.setProperty("webdriver.chrome.driver",
                                        chromeLocation);
                        driver = new ChromeDriver();
                }

                driver.manage().timeouts().implicitlyWait(MAX_WAIT_PERIOD,
                                TimeUnit.SECONDS); 
        }

        @AfterMethod(alwaysRun=true)
        public void stopDriver() {
                driver.quit();
        }
}

And test classes -

public class SignupTest extends SelTestCase {

        /**
         * Registers new user
         */
        @Test
        public void signUpToFrecklebox() {
                // some steps here
                assert HomePage.getLogOutLink().isDisplayed() : "Logout link is not displayed after registration";
        }
        
        /**
         * Verifies mandatory field validation for new user sign up
         */
        @Test
        public void signUpWithEmptyMandatoryFields() {               
                // some steps here
                assert RegistrationPage.getMandatoryFieldErrorMessage().isDisplayed():"Error message is not displayed for empty mandatory fields";
        }


One more test class -

public class LoginTest extends SelTestCase {

        @Test
        public void validLogin() {
                // some steps here
                assert HomePage.getLogOutLink().isDisplayed() : "Logout link is not displayed when logging in for user: "
                                + data.getRegisteredUserEmail();
        }

        @Test
        public void loginWithInvalidEmailAddress() {
                // some steps here
                assert LoginPage.getInvalidLoginError().isDisplayed() : "Invalid login error message is not displayed when logging with invalid email and valid password";
        }

And some more test classes in the same manner.

Tarun Bhadauria

unread,
Jan 24, 2012, 2:47:12 PM1/24/12
to testng...@googlegroups.com
Hi Cedric,

Do you see some thing wrong with code snippets I posted, which would eventually block parallel execution of test.

Cédric Beust ♔

unread,
Jan 24, 2012, 2:49:37 PM1/24/12
to testng...@googlegroups.com
Nothing jumps out after a quick read of your source, how did you determine that your tests are not running in different threads?

-- 
Cédric




On Tue, Jan 24, 2012 at 11:47 AM, Tarun Bhadauria <tku...@gmail.com> wrote:
Hi Cedric,

Do you see some thing wrong with code snippets I posted, which would eventually block parallel execution of test.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/xyheEN26rS4J.

Tarun Bhadauria

unread,
Jan 24, 2012, 2:57:17 PM1/24/12
to testng...@googlegroups.com
I assume it as in case of parallel execution, I should be seeing three FF launching as defined in testng.xml file -

<class name="com.test.SanityTest"></class>
<class name="com.test.SignupTest"></class>
<class name="com.test.LoginTest"> 

But I see them coming one after another, which is as good as "parallel=false".

Tarun Bhadauria

unread,
Jan 26, 2012, 12:53:08 PM1/26/12
to testng...@googlegroups.com
I changed the configuration of parallel run to "methods" and I see multiple browsers launched for executing test methods in parallel for classes one by one.
But when I change parallel configuration back to "classes" browsers are launched one after another executing tests sequentially for different classes.

I think I am missing some thing very basis with respect to parallel execution of classes. Does not parallel execution set for classes mean that classes are run in parallel depending on the thread-count provided. Which in my case should mean that I should yet see multiple browsers being launched but only running test methods in different classes in parallel.

Cédric Beust ♔

unread,
Jan 26, 2012, 1:18:18 PM1/26/12
to testng...@googlegroups.com
Hi Tarun,

Indeed, this is odd. What happens if you set the parallel attribute on <suite> instead of <test>?

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/CaVK61TkJVoJ.

Tarun Bhadauria

unread,
Jan 26, 2012, 1:33:05 PM1/26/12
to testng...@googlegroups.com
Hi Cedric,

I get the same results, if I keep parallel attribute at suite level
parallel = methods work but parallel = classes does not

I yet think I am missing something very silly

Cédric Beust ♔

unread,
Jan 26, 2012, 1:49:20 PM1/26/12
to testng...@googlegroups.com
On Thu, Jan 26, 2012 at 10:33 AM, Tarun Bhadauria <tku...@gmail.com> wrote:
Hi Cedric,

I get the same results, if I keep parallel attribute at suite level
parallel = methods work but parallel = classes does not

Ok, that's a relief, it would have meant something more fundamental was broken :-)

I suggest you start with a small test case, say two classes each with two methods that simply print their thread id, and run them with parallel=classes. You should see that all the methods within the same class should have the same thread id, but the two classes should run in different threads. For example

C1#m1: id=3
C1#m2: id=3
C2#m1: id=4
C2#m2: id=4

-- 
Cédric

Tarun Bhadauria

unread,
Jan 26, 2012, 2:19:13 PM1/26/12
to testng...@googlegroups.com
Hi Cedric,

This is what I tried - 

<suite name="Test Suite" verbose="5" >
<test name="test" parallel="classes" thread-count="5" >
<classes>
<class name="com.temp.test.Temp1"></class>
<class name="com.temp.test.Temp2"></class>
</classes>
</test>
</suite>


Two small classes -

public class Temp1 {
@Test
public void temp1M1() {
System.out.println("temp1M1: " +Thread.currentThread().getId());
}
@Test
public void temp1M2() {
System.out.println("temp1M2: " +Thread.currentThread().getId());
}

}

and

public class Temp2 {
@Test
public void temp2M1() {
System.out.println("temp2M1: " +Thread.currentThread().getId());
}
@Test
public void temp2M2() {
System.out.println("temp2M2: " +Thread.currentThread().getId());
}

}

And this is the out put I see -


[Invoker 10776760] Invoking com.temp.test.Temp1.temp1M1
temp1M1: 1
[Invoker 10776760] Invoking com.temp.test.Temp1.temp1M2
temp1M2: 1
[Invoker 10776760] Invoking com.temp.test.Temp2.temp2M2
temp2M2: 1
[Invoker 10776760] Invoking com.temp.test.Temp2.temp2M1
temp2M1: 1

Tarun Bhadauria

unread,
Jan 27, 2012, 12:01:08 AM1/27/12
to testng...@googlegroups.com
So it means all classes are running on same thread, is not it.

Cédric Beust ♔

unread,
Jan 31, 2012, 2:31:17 PM1/31/12
to testng...@googlegroups.com
Hi Tarun,

I'm seeing a different output:

[Invoker 1864729679] Invoking test.tmp.Temp1.temp1M1
temp1M1: 12
[Invoker 1864729679] Invoking test.tmp.Temp1.temp1M2
temp1M2: 12
[Invoker 1416386650] Invoking test.tmp.Temp2.temp2M1
temp2M1: 13
[Invoker 1416386650] Invoking test.tmp.Temp2.temp2M2
temp2M2: 13

As you can see, each class is running in its own thread.

Are you using the latest version?

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/oDqEyKdC6yMJ.

Tarun Bhadauria

unread,
Jan 31, 2012, 3:13:29 PM1/31/12
to testng...@googlegroups.com
Hi Cedric,

Found something weird
I am using testng eclipse plug-in 5.8 with Eclipse 3.4.2release. 
Test NG jar used - 6.3.1

I see the classes running in same thread id when executing tests from eclipse testng plugin
But when I use ant build I see different thread id. 

thanks for all your responses all along.

Cédric Beust ♔

unread,
Jan 31, 2012, 3:47:56 PM1/31/12
to testng...@googlegroups.com
The Eclipse plug-in uses its own testng.jar, unless you specifically overrode it. Update to the latest Eclipse plug-in.

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/mh9ry-_h0lgJ.

AdiSankar

unread,
Feb 8, 2012, 3:43:03 PM2/8/12
to testng...@googlegroups.com
Hi Cedric and Tarun:

TestNG PlugIn: 6.3.0.20111017
Eclipse Version: Helios Service Release 2

Me too facing the almost similar problem.
Different classes are run by different Threads as expected, which i came to know with help of ThreadId(). But the classes are not running in parallel, they are running sequentially

Code:

public class Class1 {
  @Test
  public void method1() {
 System.out.println("Class1 : method1      " + Thread.currentThread().getId());
 System.out.println();
 try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
  }
  
  @Test
  public void method2() {
 System.out.println("Class1 : method2      " + Thread.currentThread().getId());
 System.out.println();
  
  }
}



public class Class2 {
  @Test
  public void method1() {
 System.out.println("Class2 : method1      " + Thread.currentThread().getId());
 System.out.println();
  }
  
  @Test
  public void method2() {
 System.out.println("Class2 : method2      " + Thread.currentThread().getId());
 System.out.println();
  }




  public class Class3 {
  @Test
  public void method1() {
 System.out.println("Class3 : method1      " + Thread.currentThread().getId());
 System.out.println();
  }
  
  @Test
  public void method2() {
 System.out.println("Class3 : method2      " + Thread.currentThread().getId());
 System.out.println();
  }
 
}


TestNG.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="SuiteName" parallel="classes"
thread-count="5">
<test  name="TestName">
<classes>
<class name="SamplePackage.Class1" />
<class name="SamplePackage.Class2" />
<class name="SamplePackage.Class3" />
</classes>

</test>
</suite>


Output:

Class1 : method1      10

Class1 : method2      10

Class2 : method1      11

Class2 : method2      11

Class3 : method1      12

Class3 : method2      12

Although i kept a Thread.sleep() in Class1, its still not executing class2 / class3.
It is waiting for Thread.sleep() to complete.

Please help me out in solving this issue.

 






Krishnan Mahadevan

unread,
Feb 9, 2012, 12:39:18 AM2/9/12
to testng...@googlegroups.com
Adi Sankar,


From your output 

Output:

Class1 : method1      10

Class1 : method2      10

Class2 : method1      11

Class2 : method2      11

Class3 : method1      12

Class3 : method2      12


It looks like there is parallelism here, since each of the Classes are running in different threads and the 2 test methods in each of the classes running on the thread for the class.

So am not quite sure if I got your issue correctly here !




Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/RiP4d96xUkMJ.

AdiSankar

unread,
Feb 9, 2012, 12:49:04 AM2/9/12
to testng...@googlegroups.com
If u closely observe the code,
I kept a Thread.Sleep() in Class1.
So, it should execute Class2 and Class3 methods before Class1 methods.
But, it is not doing so. It is executing Class1, Class2 and then Class3 methods.

Expected O/P:

Class1 : method1      10

Class2 : method1      11

Class2 : method2      11

Class3 : method1      12

Class3 : method2      12

Class1 : method2      10

Cédric Beust ♔

unread,
Feb 9, 2012, 1:35:25 AM2/9/12
to testng...@googlegroups.com
Thread.sleep() is unlikely to do what you expect in multithread situations. If you want to order your tests, use dependencies.

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/exbWvOZi9qEJ.

Jayaraman Balasubramaniyan

unread,
Feb 10, 2012, 2:26:46 AM2/10/12
to testng...@googlegroups.com
I think that you are acheiving the parallel run by class wise and it is working as expected.
 
Each class will be executed by unique thread and thats what i can sense from the output you shared.
 
Correct me if I am wrong..
 
Thanks,
Jay!!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/exbWvOZi9qEJ.

ADI SANKAR

unread,
Feb 10, 2012, 2:41:34 AM2/10/12
to testng...@googlegroups.com
Its not at all working as expected..
Each class is being run by different threads, that's correct.
But the threads are running sequentially, not parallel.
There seems to be a bug.
I'm facing this problem with latest TestNG version.
So, i copied the old testng version "6.0.1.20110426_1108" into my eclipse plugin folder and it is working fine as expected.
Thanks

sai kiran

unread,
Feb 22, 2012, 11:50:46 PM2/22/12
to testng...@googlegroups.com
Hi Guys,

I'm facing the same issue. Please help me what could be correct version to execute in parallel.

Right now, in my eclipse, i have installed testng 6.3.2 plugin and in my ANT framework of lib folder, it has testng-6.0.1-nobsh-noguice.jar.

Also, i have confusion with following points
i) When i execute testng.xml  using eclipse as TestNg suite,
    a) Does it compiles the code with testng plugin installed in the eclipse or in my lib folder of testng.jar?

Thanks
Sai


Krishnan Mahadevan

unread,
Feb 23, 2012, 1:09:37 AM2/23/12
to testng...@googlegroups.com

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"


AFAIK, unless you explicitly tell eclipse, its the testng plugin installed in eclipse that would get called.
You can configure eclipse to refer to the testng jar bundled in the project via 

Project > Properties > TestNG > Use Project TestNG Jar (check this checkbox)

 
Thanks
Sai



--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/fn9Qvtp16PAJ.

ADI SANKAR

unread,
Feb 23, 2012, 1:10:41 AM2/23/12
to testng...@googlegroups.com
Hi Sai:
It uses eclipse plugin. 
It doesn't use jar in bin folder of project.
And there is a problem with testng plugin 6.3.2.
Please revert to testng plugin 6.0.1.20110426_1108.jar in eclipse.

thanks,
Adisankar

On Thu, Feb 23, 2012 at 10:20 AM, sai kiran <skrn...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/fn9Qvtp16PAJ.

Cédric Beust ♔

unread,
Feb 23, 2012, 1:12:37 AM2/23/12
to testng...@googlegroups.com

On Wed, Feb 22, 2012 at 10:09 PM, Krishnan Mahadevan <krishnan.ma...@gmail.com> wrote:
AFAIK, unless you explicitly tell eclipse, its the testng plugin installed in eclipse that would get called.
You can configure eclipse to refer to the testng jar bundled in the project via 

Project > Properties > TestNG > Use Project TestNG Jar (check this checkbox)

Please only do this if you have a very good reason to want to use your own testng.jar, since it can cause the plug-in to lock up if the versions are not right. This option should pretty much only be used when you want to use a more recent testng.jar than the one the Eclipse plug-in is using (which is rare since I release new versions of the plug-in quite often).

-- 
Cédric

Cédric Beust ♔

unread,
Feb 23, 2012, 1:13:05 AM2/23/12
to testng...@googlegroups.com
On Wed, Feb 22, 2012 at 10:10 PM, ADI SANKAR <adisa...@gmail.com> wrote:
And there is a problem with testng plugin 6.3.2.

Can you please explain what's wrong with it?

Thanks.

-- 
Cédric




sai kiran

unread,
Feb 23, 2012, 2:02:17 AM2/23/12
to testng-users
Thanks every one for support. It worked by changing the version to
5.9.0 in my build path and classes are executing in parallel.

But, i found a new issue when i placed parallel= tests and thread-
count=2 in suite level and classes in tests level with thread-
count="3". tests are not running in parallel and only classes running
in parallel.

Again, it would be some version issues?

Testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" >

<parameter name="gridFlag" parallel="tests" thread-count="2"
value="false"></parameter>

<test name="Test2" parallel="classes" thread-count="3">
<parameter name="browser" value="firefox"></parameter>

<classes>
<class name="test.HomePageTest1"/>
<class name="test.HomePageTest2"/>

</classes>
</test> Test

<test name="Test3" parallel="classes" thread-count="3">
<parameter name="browser" value="ie7"></parameter>

<classes>
<class name="test.HomePageTest1"/>
<class name="test.HomePageTest2"/>

</classes>
</test> <!-- Test -->


</suite> <!-- Suite -->


Regards,
Sai

On Feb 23, 11:12 am, Cédric Beust ♔ <ced...@beust.com> wrote:
> On Wed, Feb 22, 2012 at 10:09 PM, Krishnan Mahadevan <
>

ssimmons

unread,
Mar 10, 2012, 12:02:27 PM3/10/12
to testng-users
Has anyone made any progress with this? I am seeing the same thing as
Adi in that my classes do run in different threads, but they run in
different threads sequentially. For my test I built two identical
simple test classes (ParallelTest and ParallelTest2) with a single
method that takes some time to execute:

public class ParallelTest {

@Test
public void f() {

System.out.println("test 1 " + Thread.currentThread().getId());
for(int i = 0; i < 1000000000; i++) {
for(int j = 0; j < 4; j++) {

}
}
}
}

When running with TestNG 6.4 with 'parallel="classes", It shows they
run in different threads, but you can see the first system out, then a
pause, then the second system out (with different thread numbers). The
tests take about 14 seconds to complete. If I switch to TestNG 6.1 and
run, you see both system outs immediately, then a pause, then the
tests complete in about 7 seconds. FYI, I am kicking these off using
the maven surefire plugin, starting a build from the command line.


On Jan 22, 2:40 pm, Tarun Bhadauria <tkum...@gmail.com> wrote:
> TestNG - 6.3.1
>
> testng.xml -
>
> <?xml version="1.0" encoding="utf-8"?><!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" ><suite name="Test Suite" verbose="5">        <parameter name="appURL" value="http://www.test.com/default.aspx"></parameter>        <parameter name="locale" value="us"></parameter>        <test name="test in Firefox" parallel="classes"                thread-count="3">                <parameter name="browser" value="FF"></parameter>                <classes>                        <class name="com.test.SanityTest"></class>                        <class name="com.test.SignupTest"></class>                        <class name="com.test.LoginTest">                                            </class>                </classes>        </test></suite>
>
> Despite I have "parallel= classes" set at test tag level all tests are executed sequentially.
>
> Did I miss something obviously wrong?
>
> ~tarun

ssimmons

unread,
Mar 10, 2012, 1:59:49 PM3/10/12
to testng-users
Update: further testing seems to point to 6.2.1 as the last version of
TestNG that runs parallel classes correctly. 6.3 and up all seem to
run the tests in different threads, but sequentially

Cédric Beust ♔

unread,
Mar 10, 2012, 2:06:52 PM3/10/12
to testng...@googlegroups.com
It's not TestNG that runs the threads sequentially, it's the JVM.

-- 
Cédric




--
You received this message because you are subscribed to the Google Groups "testng-users" group.

Cédric Beust ♔

unread,
Mar 10, 2012, 2:07:47 PM3/10/12
to testng...@googlegroups.com
By the way, "sequentially" doesn't mean much, all you can say is that you're seeing the output of your tests displayed sequentially. It doesn't mean much more than that, the only reliable information is the thread id's. You shouldn't worry about anything else.

-- 
Cédric

ssimmons

unread,
Mar 10, 2012, 3:50:16 PM3/10/12
to testng-users
The only issue is that the 2 tests consistently take twice a long to
execute (14 seconds instead of 7 seconds) when I change over from
version 6.2.1 to 6.3 and above, which would seem to indicate that they
are not running in parallel even if they are in different threads. In
my real world tests (Selenium based tests) this is more obvious when I
change over to versions 6.3+. When I remote desktop into my Selenium
Grid nodes and launch my tests with 6.2.1 with parallel="classes" and
thread-count="6", I can watch as 6 browsers are launched and the
associated test classes run in parallel. Then changing only my pom
dependency to version 6.3 and launching the same tests again, only one
browser is launched for one test class at a time and all test classes
take about 6 times as long to run.

On Mar 10, 2:07 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> By the way, "sequentially" doesn't mean much, all you can say is that
> you're seeing the output of your tests displayed sequentially. It doesn't
> mean much more than that, the only reliable information is the thread id's.
> You shouldn't worry about anything else.
>
> --
> Cédric
>
> On Sat, Mar 10, 2012 at 11:06 AM, Cédric Beust ♔ <ced...@beust.com> wrote:
>
>
>
>
>
>
>
> > It's not TestNG that runs the threads sequentially, it's the JVM.
>
> > --
> > Cédric
>
Reply all
Reply to author
Forward
0 new messages