Parallel execution of Tests
mjl
is there any way of parallel execution of tests? So what i would like
to achieve is about following:
Given a fixture like
<suite name="Suite1" verbose="1" >
<test name="Test1" >
<classes>
<class name="SomeTest" />
</classes>
</test>
<test name="Test2" >
<classes>
<class name="test.OtherTest1" />
<class name="test.OtherTest2" />
</classes>
</test>
</suite>
or maybe
<test name="Test">
<groups>
<run>
<include name="group1"/>
<include name="group2"/>
</run>
</groups>
<classes>
<class name="test.SomeTest"/>
<class name="test.OtherTest1"/>
<class name="test.OtherTest2"/>
</classes>
</test>
i would like e.g. in fixture 1 the tests 'Test*' to be executed in
parallel, but the individual methods (@Test) in either Test1 and Test2
in sequential order. Or for example 2, parallel executing of the given
groups, but sequential execution of individual test methods aggregated
in each group.
Does this make sense to anybody or is this even a stupid thing to try?
Cheers,
michael
Cédric Beust ♔
<suite name="Suite1" verbose="1" parallel="true">
At the end, you can look at the HTML reports in the section "chronological order" and you'll see a list of all the test methods along with the thread ID they ran on.
--
Cedric
Cédric
Michael Luckey
i will try this one again. But at first glance i got the impression,
that with parallel="true" *all* test methods defined in the suite will
run in parallel. Which isn't exactly what i tried to express. I'll try
to be more explicit:
public class Test1 {
@Test(groups = {"grp1"})
public void m1() {
assert true;
}
@Test(groups = {"grp1"})
public void m2() {
assert true;
}
}
public class Test2 {
@Test(groups = {"grp2"})
public void m1() {
assert true;
}
@Test(groups = {"grp2"})
public void m2() {
assert true;
}
}
I do need the methods in grp1 resp. grp2 running in sequentiell order,
but would like to have grp1 and grp2 parallely executed. Does this make
more sense?
michael
Cédric Beust ♔ schrieb:
Cédric Beust ♔
You can aslo specify this at the <test> level...
--
Cedric
Cédric
Alexandru Popescu
> Michael,
>
> You can aslo specify this at the <test> level...
>
> --
> Cedric
>
>
I've got the impression that Michael would like exactly the oposite: inside a <test> everything to
be run sequential, but the <test> to be parallel.
I don't think we are currently supporting this. The <test> are treated sequentially (iirc).
./alex
--
.w( the_mindstorm )p.
Michael Luckey
So i would try something like the following
<suite name="Suite1" verbose="1" parallel="true">
<test name="Test1" parallel="false">
<classes>
<class name="SomeTest" />
</classes>
</test>
<test name="Test2" parallel="false" >
<classes>
<class name="test.OtherTest1" />
<class name="test.OtherTest2" />
</classes>
</test>
</suite>
But i really got the impression, the attribute 'parallel' does apply at
method layer, which means as soon i specify it, all methods which
potentially can be executed in parallel will end in the same list,
ordered by classname and then will be executed...
Cédric Beust ♔ schrieb:
> Michael,
>
> You can aslo specify this at the <test> level...
>
> --
> Cedric
>
>
> On 12/10/05, *Michael Luckey* < luc...@lesspain.de
Michael Luckey
>
> #: Cédric Beust ♔ changed the world a bit at a time by saying on
> 12/10/2005 4:37 PM :#
>
>> Michael,
>>
>> You can aslo specify this at the <test> level...
>>
>> --
>> Cedric
>>
>>
>
> I've got the impression that Michael would like exactly the oposite:
> inside a <test> everything to be run sequential, but the <test> to be
> parallel.
>
>
> I don't think we are currently supporting this. The <test> are treated
> sequentially (iirc).
>
> ./alex
> --
> .w( the_mindstorm )p.
Yes, exactly this was, what i was trying to express. Does it make sense
at all?
michael
Cédric Beust ♔
Yes, exactly this was, what i was trying to express. Does it make sense
at all?
Ah sorry, I misunderstood. Alexandru is right, we don't support this currently.
It seems a bit odd to me: if certain methods within a <test> can be run in parallel, why would you want TestNG to run them sequentially?
--
Cédric
Michael Luckey
>
>
> On 12/10/05, *Michael Luckey* <luc...@lesspain.de
Cedric,
because they can not be run in parallel. I'll try to explain:
We got a group of testMethods accessing a (shared) resource. Another
group accessing another (shared) resource. So i could run both groups in
parallel, but as soon methods part of the same group are parallely
executed, my tests are messed up.
I was already thinking off starting different TestNGs (TestRunner?), but
then i loose a collective report (, and my IDE :-(
So well, i probably have to wait unitl they have sequentially finished :-(
Thanks for your assistance,
michael
Cédric Beust ♔
Cedric,
because they can not be run in parallel. I'll try to explain:
We got a group of testMethods accessing a (shared) resource. Another
group accessing another (shared) resource. So i could run both groups in
parallel, but as soon methods part of the same group are parallely
executed, my tests are messed up.
I was already thinking off starting different TestNGs (TestRunner?), but
then i loose a collective report (, and my IDE :-(
So well, i probably have to wait unitl they have sequentially finished :-(
I see.
Right now, the only way to do this would probably be to guarantee synchronization at the resource level: if your resource cannot be accessed from multiple threads, it should probably supply an API to guarantee that...
What do you think?
--
Cédric
Michael Luckey
>
>
> On 12/10/05, *Michael Luckey* <luc...@lesspain.de
Cedric,
you are probably right about these API-thingy. But i think, in our
special case, it is not caused by the resource, which is basically
accessible from multiple threads, more the way we use it. Say we use a
datastore (rw), which will be scrubbed and initialized before each test
method. I guess, this has to be done. Now surely those methods should
not be executed in parallel. So we would have to provide a separate
datastore for each running thread, which seems not to be always feasible
(some of them are a bit heavier).
So i guess, as always, if it hurts enough, someone will find a solution.
Reading this parallel-thingy of TestNG i hoped to get one for free. Bad
luck. Only wish to be smarter and able to fix my screwed fixtures...
Again, thanks for being so helpful.
Cheers,
michael
Alexandru Popescu
I have re-read this thread, and I find myself wondering:
[hypothesis]
you need to have 2 <test>-s running in parallel, but inside each test the methods should be run
sequential.
[/hypothesis]
Do you need to have them run in a specific order? (meaning do you have dependencies declared?). In
this case you can unify both <test>-s, specify the <test> as parallel and the 2 parallel sequences
will be run in parallel
If you don't have a specific order, than both <test>-s can be run in parallel as there is no safe
guarantee about the ordering.
hth,
Michael Luckey
>
> I have re-read this thread, and I find myself wondering:
>
> [hypothesis]
> you need to have 2 <test>-s running in parallel, but inside each test
> the methods should be run sequential.
> [/hypothesis]
>
> Do you need to have them run in a specific order? (meaning do you have
> dependencies declared?).
I read this as: do you have to run the individual test methods in a
specific order. No, i do not care, no dependencies involved.
> In this case you can unify both <test>-s, specify the <test> as
> parallel and the 2 parallel sequences will be run in parallel
This one i do not understand. Do you mean like e.g.
///////////////////////////////////////////////////////////////////
package parallel;
import org.testng.annotations.Test;
public class Test1 {
@Test(groups = { "grp1"})
public void m1() {
assert true;
}
@Test(dependsOnMethods = { "m1" }, groups = { "grp1"})
public void m2() {
assert true;
}
}
and then
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="4" verbose="3" name="Parallel" parallel="true"
annotations="1.5">
<test name="Test" annotations="1.5">
<classes>
<class name="parallel.Test1" />
<class name="parallel.Test2" />
</classes>
</test>
</suite>
I thought this could not work, because as soon as dependencies are
involved, these methods will be added to the sequential list and are not
liable to parallel execution at all. So obviously i have some
misunderstandings here.
For sure i could express an order, but actually i do not care, so i
would prefer not to do so.
>
> If you don't have a specific order, than both <test>-s can be run in
> parallel as there is no safe guarantee about the ordering.
Sorry, but this one i do not get either. I tried to express that i have
to guarantee, that no two methods of the same group are executed at the
same time (as they are hitting the same resource).
But neither do i care in which order these methods are executed, nor if
there are any methods of an other group executed in between. So why
could they be executed in parallel as soon as no order is involved? I
guess, i am to tired now to understand what exactly you are saying, have
to think about this a while.
michael
Cédric Beust ♔
Everything I say below assumes that you are running TestNG in parallel mode.
As of today, the only way TestNG lets you specify that certain methods should be run in the same thread is if if have dependencies on each other. Michael points out that he needs a different kind of control over the sequential/parallel categorization that TestNG is defining today.
This discussion makes me realize that indeed, TestNG could do a better job at this, and that dependencies do not necessarily need to be tied to sequential execution: you might want to guarantee that certain methods will run in the same thread even though they don't necessarily depend on each other.
Point well taken.
My first reaction to this is: we should be able to define that a group will always run sequentially. In other words, all the methods in that group will always run in the same thread (but in no particular order: you don't need to define dependencies between these methods).
Michael points out that another way to define this could be at the testng.xml level: all the tests inside a <test> stanza will run sequentially but each <test> can run in parallel to another <test> in that same testng.xml.
Michael, did I capture this correctly?
What do you think of the "thread group" idea? Would that help you or is the <test> approach the only one that would work for you?
--
Cédric
Michael Luckey
Cedric,
i'd say you did a pretty good job in capturing this.
Concerning the tread group idea, i am actually not experienced enough
with the workings of TestNG to make any profound statement. But my
feelings right now are, that (at least) we perfectly could do without
the <test> stanza approach.
But how would this "thread group" look like? Is it meant to be a new
attribute of the @Test annotation e.g.
@Test(threadGroup="treadGroup1", groups={"grp1", "grp2"})
<groups>
<run>
<include name="treadGroup1" />
</run>
</groups>
or could it be integrated into/used with the existing group mechanism?
So one could imagine the following
<groups>
<run>
<include name="grp1" parallel="false" />
</run>
</groups>
and
<groups>
<define name="threadgroup1" parallel="false">
<include name="grp1"/>
<include name="someOtherGrp"/>
</define>
</groups>
which might be really powerful, but could be just as well a (very)
stupid idea.
The last one would give the possibility not only to run a group of test
methods sequential, but also to assure sequentiality of more than one
group, in which case it could be easier to group things at a finer
granularity.
But i do not have any idea of the implications of that approach. It
might be totally stupid to define this threading issue at testng.xml
level. I don not knnow, wether it will make any sense to have a group of
methods which, in one configuration must be run sequentially, in another
could also be run in parallel. One might shoot oneself in the foot doing
this...
So, once again, i believe that a group approach will (more than) suffice
in our case. I only came up with this <test> stanza as we are still
working with those junit classes, which do not (yet?) have any grouping
defined.
Well, i'm curious about the solution you will come up with, if it should
be done at all, as i do not know if anybody else would appreciate such a
feature.
Cheers,
michael
Cédric Beust ♔
Right now, I was thinking of using the existing group system and letting you specify the list of groups that should be run in the same thread in testng.xml.
Something like:
<suite parallel="true" sequential-groups="database1 database2" />
After that, it's up to you to make your test methods belong to the right group, knowing that all the methods in "database1" will be run in the same thread (and same for "database2").
Just thinking out loud right now, but as you say, so far you are the only one requesting this feature so I'd like to hear from more people before I launch myself into this...
--
Cedric
Cédric
Michael Luckey
> Michael,
>
> Right now, I was thinking of using the existing group system and
> letting you specify the list of groups that should be run in the same
> thread in testng.xml.
>
> Something like:
>
> <suite parallel="true" sequential-groups="database1 database2" />
>
> After that, it's up to you to make your test methods belong to the
> right group, knowing that all the methods in "database1" will be run
> in the same thread (and same for "database2").
>
> Just thinking out loud right now, but as you say, so far you are the
> only one requesting this feature so I'd like to hear from more people
> before I launch myself into this...
>
> --
> Cedric
>
Cedric,
this perfectly makes sense to me.
Thanks again guys.
michael