Re: [testng-users] TestNG Parallel Execution Not working as expected from 6.3.+

[Cédric Beust ♔]

In matter of threading, the only source for truth is the thread id. If you are seeing different thread id's, then your methods are running in different threads. The fact that they appear to run sequentially is irrelevant, you can be assured that the JVM will dispatch them as best it can based on your cores/processors.

-- 

Cédric

On Tue, Jun 26, 2012 at 4:45 PM, KT <kaite...@gmail.com> wrote:

Hi,

I have the following tests:

public class TestA {

    @Test

    public void testA() {

        System.out.println("TestA Id: " + Thread.currentThread().getId());

        System.out.println("TestA Time: " + System.currentTimeMillis());

        for (int i = 0; i < 100; i++) {

            System.out.println("TestA Count: " + i);

        }

        System.out.println("TestA End Time: " + System.currentTimeMillis());

    }

}

public class TestB {

    @Test

    public void testB() {

        System.out.println("TestB Id: " + Thread.currentThread().getId());

        System.out.println("TestB Time: " + System.currentTimeMillis());

        for (int i = 0; i < 100; i++) {

            System.out.println("TestB Count: " + i);

        }

        System.out.println("TestB End Time: " + System.currentTimeMillis());

    }

}

public class TestC {

    @Test

    public void testC() {

        System.out.println("TestC Id: " + Thread.currentThread().getId());

        System.out.println("TestC Start Time: " + System.currentTimeMillis());

        for (int i = 0; i < 100; i++) {

            System.out.println("TestC Count: " + i);

        }

        System.out.println("TestC End Time: " + System.currentTimeMillis());

    }

}

<?xml version="1.0" encoding="UTF-8"?>

<suite name="Test Suite" parallel="classes" thread-count="20">

    <test name="Tests" parallel="classes">

        <packages>

            <package name="com.testng.test"/>

        </packages>

    </test>

</suite>

When I run these tests with TestNg 6.2.1 with parallel=classes, it seems to run as expected => each test in a different thread and each thread running simultaneously

Whereas with 6.5.2, each test runs in a different thread, but each thread seems to run one at a time, one after another.

Please see attached test result files for more details.

Thanks.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/wsKvpf63pYgJ.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[KT]

Thanks Cedric for the reply.

I agree my methods are running in different threads, but it simply does not make sense to me, that my tests would consistently run testA first, then testB, thenC, if the tests are set to run in parallel (in TestNg 6.5.2)

It could be that the tests are kicked off in different threads, but each test is blocked from running until the prior test completes. This would defeat the purpose of parallel test execution. The reason why I mentioned this is because in 6.5.2, I always see the same behavior with my tests: TestA starts, runs, completes, TestB starts, runs, completes, TestC starts, runs, completes.

Whereas, with 6.2.1, the test result output always varies. (Test A, B and C starting, test activity constantly switching between Test A,B, C and eventually all 3 tests complete)

Additional thoughts are greatly appreciated. Thanks.

-- 

Cédric

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Alexander Poulikakos]

Hi

 

Would it look different if you add a slight delay in the for loop?

 

/Alex

 

 

---

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/_eSawhhD-RwJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[KT]

Nope, I added a 2 second sleep in TestA, and it still runs first, taking 2 seconds between each count 1 to 100, then TestB runs, then TestC.

Again, this happens in 6.5.2 (and 6.3+) but not in 6.2.1

public class TestA {

    @Test

    public void testA() throws InterruptedException {

        System.out.println("TestA Id: " + Thread.currentThread().getId());

        System.out.println("TestA Time: " + System.currentTimeMillis());

        for (int i = 0; i < 100; i++) {

            System.out.println("TestA Count: " + i);

            Thread.sleep(2000);

        }

        System.out.println("TestA End Time: " + System.currentTimeMillis());

    }

}

On Wednesday, June 27, 2012 12:35:54 AM UTC-7, drapostolos wrote:

Hi

 

Would it look different if you add a slight delay in the for loop?

 

/Alex

 

 

[Alexander Poulikakos]

How about using Thread.yield() instead of Thread.sleep()?

 

 

---

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/XjNoJJP5K_QJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Abhay]

 Thread.yield()  too gives the same o/p . I am also facing same problem.Even with me Parallel Execution Not working as expected !! 

On Wednesday, 27 June 2012 13:19:08 UTC+5:30, drapostolos wrote:

How about using Thread.yield() instead of Thread.sleep()?

 

 

---

From: testng...@googlegroups.com [mailto:testng-users@googlegroups.com] On Behalf Of KT

[Aidan]

I did some more debugging, and found that DynamicGraph is determining that ParallelTestClassB depends on ParallelTestClassA. So getFreeNodes() is only returning ParallelTestClassA at first. So it really is running the tests in sequence.

On Wednesday, June 27, 2012 7:24:35 AM UTC-7, Aidan wrote:

I had tried to post this last night, but it doesn't seem to have gone through; otherwise, please forgive the double-post.

I created a similarly simple project:

ParallelTestClassA.java

public class ParallelTestClassA {

    @BeforeClass

    public void setup() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testA setup starting (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testA setup finished (thread id = " + Thread.currentThread().getId() + ")");

    }

    

    @Test

    public void testA() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testA start (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testA finish (thread id = " + Thread.currentThread().getId() + ")");

    }

    

    @AfterClass

    public void tearDown() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testA tearDown starting (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testA tearDown finished (thread id = " + Thread.currentThread().getId() + ")");

    }

}

ParallelTestClassB.java

public class ParallelTestClassB {

    @BeforeClass

    public void setup() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testB setup starting (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testB setup finished (thread id = " + Thread.currentThread().getId() + ")");

    }

    @Test

    public void testB() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testB start (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testB finish (thread id = " + Thread.currentThread().getId() + ")");

    }

    

    @AfterClass

    public void tearDown() throws InterruptedException {

        System.out.println("" + System.currentTimeMillis() + " - testB tearDown starting (thread id = " + Thread.currentThread().getId() + ")");

        Thread.sleep(1000);

        System.out.println("" + System.currentTimeMillis() + " - testB tearDown finished (thread id = " + Thread.currentThread().getId() + ")");

    }

}

testng.xml

<suite name="Parallel Test Suite" parallel="methods" thread-count="20">

    <test name="Parallel Test">

        <classes>

            <class name="parallel.ParallelTestClassA" />

            <class name="parallel.ParallelTestClassB" />

        </classes>

    </test>

</suite>

I set breakpoints in the @BeforeClass, @Test and @AfterClass methods in both classes.

When I debugged through it with 6.5.2, the debugger stopped at ParallelTestClassA.setup(), and there was only one testing thread. A second thread didn't get created until after ParallelTestClassA.tearDown() finished.

When I did the same thing with 6.2.1, when the debugger stopped at ParallelTestClassA.setup(), there were already 2 threads executing.

This issue is important to my organization because we have a large suite of browser-driven tests, and we've seen the duration of our tests increase dramatically since upgrading to 6.5.2. So if there's any information I can provide that would help, please let me know!

Thanks!

[Aidan]

I'm pretty sure I found it...

In 6.2.1, it checks for preserve order in TestRunner.java using this (line 1074):

if ("true".equalsIgnoreCase(getCurrentXmlTest().getPreserveOrder())) {

In 6.5.2, it does something way more convoluted (line 1093):

if (! hasDependencies && "true".equalsIgnoreCase(getCurrentXmlTest().getPreserveOrder())) {

This condition is getting evaluated as true, regardless of whether preserve-order="false" is specified in testng.xml.

Since it's treating preserve-order as if it's turned on, it's making B dependent on A.

It looks like this has already been updated in the trunk, but I haven't building it and testing.

[Cédric Beust ♔]

Hi Aidan,

Fantastic debugging work and I've been doing the same thing apparently at the same time (right now :-)).

I've come to the exact same conclusion you did, the fix is that preserver-order calculations should simply be skipped if run in parallel mode. I'm polishing up the fix right now.

-- 

Cédric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/-R-k2da7zVYJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

I just pushed the fix, you can try it at http://testng.org/beta

The version should read 6.6beta 20120617_0003

Aidan, thanks again for taking the time to dive in and identify the problem, and I apologize to everyone else who reported the problem for not paying more attention.

Please let me know how the beta works for you.

-- 

Cédric

[Aidan]

Awesome, thanks so much, Cedric. Testing now.

-Aidan

-- 

Cédric

-- 

Cédric

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Aidan]

Works perfectly for me.

Thanks again!

-Aidan

[Abhay]

Hi Cedric, hi all,

Making preserve-order="false" gives me o/p as below:

The Thread Id for TestNG3 is 15

Test Count:0

The Thread Id for TestNG1 is 13

Test Count:1

The Thread Id for TestNG2 is 14

Test Count:0

Test Count:2

Test Count:1

Test Count:3

Test Count:2

Test Count:4

Test Count:3

Test Count:5

Test Count:4

Test Count:6

Test Count:5

Test Count:7

Test Count:6

Test Count:8

Test Count:7

Test Count:9

Test Count:8

Test Completed @ 1340859612149

Test Count:0

Test Count:1

Test Count:2

Test Count:3

Test Count:4

Test Count:5

Test Count:6

Test Count:7

Test Count:8

Test Count:9

Test Completed @ 1340859612150

Test Count:9

Test Completed @ 1340859612150

===============================================

Suite

Total tests run: 3, Failures: 0, Skips: 0

===============================================

still I can see difference. is it normal !? or am I going wrong some where?

Thanks & Regards,

Abhay

[Cédric Beust ♔]

Hi Abhay,

I'm not sure what you're asking and I also don't know what "o/p" means.

The trace seems to indicate that the threads are now properly interleaved, am I reading this correctly?

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/0OI3BbykC5EJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Abhay]

Hi Cedric,

I just want to ask is , I have three different classes which have same methods. I have run it in parallel .

And I have "System.out.println("Test Completed @ "+System.currentTimeMillis());" which gives me completion of test of one class.

So the result I got is :

for class A :Test Completed @ 1340859612149

class B:Test Completed @ 1340859612150

& class C:Test Completed @ 1340859612150

So from this we can see that class A is getting executed first and after that both class B & class C are executed simultaneously.

So my question is why all three classes don't get finished at same time? 

Thanks & Regards,

Abhay.

[Cédric Beust ♔]

The odds of all classes finishing at the exact same time are minuscule, and even if that's what currentTimeMillis() tell you, it's not precise enough for this.

The important part is that your classes run on different threads and are properly interleaved.

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/N3VVbYD1BcoJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Abhay]

I got it, Thanks a lot Cedric for your instant reply

Abhay.

[Padmavathi Potanapalli]

Cédric,

Can you please define the exact problem, and solution to this.Also please let us know which version is having the problem and in which version we not encounter this problem.

Thanks,
Padma

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/7hPpW3pH1kUJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Krishnan Mahadevan]

Padma,

You should either downgrade to 6.2.1 or you should wait for the next release of TestNG (6.5.3) which I guess will have the fix for this.

From my understanding, the problem comes when TestNG is doing parallel execution, it looks like it is taking into account the value of preserve-order (which is always true) by default.

So you can try to explicitly set this to "false" and see if that helps. If not, like I said downgrade or wait for the next release of TestNG.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

[Krishnan Mahadevan]

Based on what Aidan troubleshooted,

if (! hasDependencies && "true".equalsIgnoreCase(getCurrentXmlTest().getPreserveOrder())) {

If the above line is what is causing problem, then you might want to skip "preserve-order" completely and just downgrade TestNG version that you are depending upon.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

[Cédric Beust ♔]

Hi Padma,

I already explained the problem, the solution and pushed a fix in the current beta at http://testng.org/beta

The fix will be included the next version of TestNG.

-- 

Cédric

[krishn...@gmail.com]

Hi Cédric,

Hi,

I am using Test NG 6.8.7 and am able to run tests in parallel using TestNG.

My problem is that the TestNG thread is still hanging around after the test execution is completed. It happens only when I set parallel on suite.

Please let me know how to get rid of hanging TestNG thread. I am doing the following:

public static void main(String[] args) {
        XmlSuite suite = new XmlSuite();
        suite.setName("TmpSuite");
        suite.setParallel("tests");
        suite.setThreadCount(4);
        
        List<XmlTest> myTests = new ArrayList<XmlTest>();
       
        XmlTest test = new XmlTest(suite);
        test.setName("TmpTest");
        List<XmlClass> classes = new ArrayList<XmlClass>();
        classes.add(new XmlClass("Test1"));
        test.setXmlClasses(classes) ;
        myTests.add(test);
       
        XmlTest test1 = new XmlTest(suite);
        test1.setName("TmpTest1");
        test1.setXmlClasses(classes) ;
        myTests.add(test1);
       
        suite.setTests(myTests);
        List<XmlSuite> suites = new ArrayList<XmlSuite>();
        suites.add(suite);
        TestNG tng = new TestNG();
        tng.setXmlSuites(suites);
        tng.run();
    }

public class Test1 { 
    @Test
    public void TestMethod() {
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        System.out.println("Test Executed");
    }
}
 
The problem occurs only when I set 
suite.setParallel("tests");

If I comment it out, the TestNG thread doesn't remain.

Thanks,
Krishna