Running all methods in a class sequentially
Adam
for the great software!
We would like a way we can run all methods in a class sequentially- the
order doesn't matter, but we have a lot of test classes and they are
not thread-safe.
The only way we've found to do this is make each test method depend on
the previous method... this is brittle and ugly... but it does do the
job. Is there a better way?
The reason we want this is:
1. It's a lot of effort to go back and make all our test classes and
methods thread safe.
2. It seems that it makes writing well factored test code harder, since
common variables and methods now have to be pulled out into separate
classes.
cheers
adam
--
Adam Feuer
Cédric Beust ♔
Hi, we're using TestNG successfully here in my group at Amazon. Thanks
for the great software!
That's great to hear!
We would like a way we can run all methods in a class sequentially- the
order doesn't matter, but we have a lot of test classes and they are
not thread-safe.
By "sequentially", do you mean that you want all the test methods in a given class to be run next to each other, without any other test methods from other classes being run in-between? This should be the default behavior, unless you are running in multithreaded mode? Can you give more details on what you are seeing exactly?
Cédric
Feuer, Adam
> By "sequentially", do you mean that you want all the test methods in
> a given class to be run next to each other, without any other test
> methods from other classes being run in-between? This should be the
> default behavior, unless you are running in multithreaded mode? Can
> you give more details on what you are seeing exactly?
We are running in multithreaded mode to get parallelization. What we
want is for TestNG to run each test class' methods sequentially, not
in parallel. The parallelization will still give us a lot of
performance gain since other test classes can run in their own
threads.
Does that make sense?
What happens now by default is that TestNG will run each test method
in its own thread... since most of our test classes are not
thread-safe, things break. It's hard to see how to write thread safe
test classes either without a lot of code duplication or without a lot
of complication (introducing a lot of thread-safe "helper" classes).
What we do to avoid this problem is to make each test method dependent
on the previous test method, so that TestNG will run the methods in a
test class sequentially. This avoids threading issues with overwriting
variables and multiple threads calling the same helper method at the
same time. However, it is ugly, cumbersome, and brittle. Is there a
better way?
cheers
adam
--
Adam Feuer
afe...@amazon.com
Amazon.com Enterprise Services
Cédric Beust ♔
Okay, thanks for clarifying.
I understand your reluctance to making your class thread-safe just for the purpose of testing... definitely not a good thing.
Maybe I could add an attribute to the @Test annotation that would mark the class as "sequential", or maybe a better word: "atomic":
@Test(atomic = true)
public class A { ... }
What does everyone think about the idea and the naming?
--
Cedric
Cédric
Justin Lee
Hash: RIPEMD160
Wouldn't it just be simpler to use a dependsOnMethod entry? It'd mean a
little more management on these tests, but less complexity in the
annotations and testng proper.
Cédric Beust ? wrote:
> Hi Adam,
>
> Okay, thanks for clarifying.
>
> I understand your reluctance to making your class thread-safe just for
> the purpose of testing... definitely not a good thing.
>
> Maybe I could add an attribute to the @Test annotation that would mark
> the class as "sequential", or maybe a better word: "atomic":
>
> @Test(atomic = true)
> public class A { ... }
>
> What does everyone think about the idea and the naming?
>
> --
> Cedric
>
>
> On 8/10/06, * Feuer, Adam* <afe...@amazon.com
> afe...@amazon.com <mailto:afe...@amazon.com>
> Amazon.com <http://Amazon.com> Enterprise Services
>
>
>
>
>
>
> --
> Cédric
>
> >
- --
Justin Lee
http://www.antwerkz.com
AIM : evan chooly
Skype : evanchooly
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)
iD8DBQFE24j7JnQfEGuJ90MRAxpOAKCgxHv0Pxy7AxCfwSkelTzzVo2KowCgrlIT
hj3+5xTWI+ZqLgBG4s2kwwk=
=VMYz
-----END PGP SIGNATURE-----
Cédric Beust ♔
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160
Wouldn't it just be simpler to use a dependsOnMethod entry? It'd mean a
little more management on these tests, but less complexity in the
annotations and testng proper.
This used to be my response to this kind of query, but Adam is making a reasonable case. Note that I said "reasonable", not "bullet-proof" :-)
I'm still waiting to hear more feedback about this, since so far, Adam seems to be the only one who had to face this limitation, and I'm not very keen on adding features for just one person (no offense, Adam :-)).
Cédric
BK Lau
You said:
"[b]Maybe I could add an attribute to the @Test annotation that would mark the
class as "sequential", or maybe a better word: "atomic":
@Test(atomic = true)
public class A { ... }
What does everyone think about the idea and the naming?[/b] "
IMO: I think a better name would be
@Test(serially = true)
public class A { ... }
Incidentally, I have the same problem here at my work place where the classes are not thread-safe...since the Java classes that our devs wrote are just JNI wrappers to unsafe **legacy** C++ code...
-BK-
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=39932&messageID=79365#79365
Cédric Beust ♔
Hi Cederic:
You said:
"[b]Maybe I could add an attribute to the @Test annotation that would mark the
class as "sequential", or maybe a better word: "atomic":
@Test(atomic = true)
public class A { ... }
What does everyone think about the idea and the naming?[/b] "
IMO: I think a better name would be
@Test(serially = true)
public class A { ... }
The problem with "serially" is that it just says that these methods will be run... well, "serially". There is no notion that no methods from other classes will be run in-between, which "atomic" conveys better in my opinion...
Incidentally, I have the same problem here at my work place where the classes are not thread-safe...since the Java classes that our devs wrote are just JNI wrappers to unsafe **legacy** C++ code...
So how do you solve this problem? You run this class in non-multithreaded code?
By the way, Adam, this would be one way to solve your problem: declare your suite parallel, but put all the classes that are not thread-safe in a non-parallel test:
<suite parallel="true" >
<test>
.. classes that are thread-safe
</test>
<test parallel="false">
.. classes that are not thread-safe
</test>
Did you try this?
Cédric
Alexandru Popescu
face once again the discussion about the annotation inheritance. Other
than this, I find it quite cool and as Adam is showing it is pretty
needed.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
Feuer, Adam
> problem: declare your suite parallel, but put all the
> classes that are not thread-safe in a non-parallel test:
This removes the thread-safety problem, but then also prevents most of our unit and integration tests from being run in parallel- which is the main reason we are using in TestNG. We were able to speed up our tests 10x by using TestNG's parallel mode - we went from a 40 minute run to a 4 minute run.
> Maybe I could add an attribute to the @Test annotation that
> would mark the class as "sequential", or maybe a better word:
> "atomic":
cheers
adam
--
Adam Feuer
Amazon.com Enterprise Commerce Services
Cédric Beust ♔
> By the way, Adam, this would be one way to solve your
> problem: declare your suite parallel, but put all the
> classes that are not thread-safe in a non-parallel test:
Cedric,
This removes the thread-safety problem, but then also prevents most of our unit and integration tests from being run in parallel- which is the main reason we are using in TestNG. We were able to speed up our tests 10x by using TestNG's parallel mode - we went from a 40 minute run to a 4 minute run.
Ok.
> Maybe I could add an attribute to the @Test annotation that
> would mark the class as "sequential", or maybe a better word:
> "atomic":
I like "sequential" better... by "atomic", do you mean that each test method from that class will be run by itself, and will not overlap with another test method from the same class? If so then "atomic" is also ok.
I meant atomic at the class-level ("all methods in this class will be run in one thread without any test method from other classes interfering"). I realize now that your requirement is a bit weaker than this: you just want the methods in this class to be run in the same thread (thereby guaranteeing that they won't overlap each other) but you don't really care if other test methods from different classes are invoked in-between. Correct?
If I got that right, it might then make more sense to be able to specify a thread-id:
@Test(threadId = "t1")
public class A { ... }
which means that all the test methods in this class will be run in the thread named "t1" (an arbitrary name, just a way for TestNG to differentiate certain threads from its thread pool).
Potential problem: you define a thread pool of five threads but you accidentally specify six threadId's throughout your test classes. I suppose TestNG could throw an exception in this case, or maybe merge threads together if they are shared. Which might bring up the need for another attribute:
@Test(threadId = "t1", shared = true)
public class A {
would allow TestNG to use this thread to run other test methods than on class A, while shared = false would make sure that only A methods run in this thread.
Thoughts?
Cédric
Feuer, Adam
> I meant atomic at the class-level ("all methods in this class
> will be run in one thread without any test method from other
> classes interfering").
This would be fine.
> I realize now that your requirement
> is a bit weaker than this: you just want the methods in this
> class to be run in the same thread (thereby guaranteeing that
> they won't overlap each other) but you don't really care if
> other test methods from different classes are invoked
> in-between. Correct?
> If I got that right, it might then make more sense to be able
> to specify a thread-id:
I prefer the "sequential" or "atomic" method you mentioned before- this would just tell TestNG not to overlap test methods from the same class. That way TestNG can decide the best way to do this.
-adam
Alexandru Popescu
What would be the real benefit of this? The user have no access to the
thread used to run the tests, so using names would help to almost
nothing imo (but I may be wrong).
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
> --
> Cédric
>
>
> >
>
Cédric Beust ♔
> If I got that right, it might then make more sense to be able to specify a
> thread-id:
>
> @Test(threadId = "t1")
> public class A { ... }
What would be the real benefit of this? The user have no access to the
thread used to run the tests, so using names would help to almost
nothing imo (but I may be wrong).
It's just a symbolic name, a way to tell TestNG "pick a thread from your thread pool, call it "t1" and only put methods from this class on that thread".
Cédric
Alexandru Popescu
IMO this looks more like an implementation-oriented annotation, than
the more generic/readable form sequential/atomic.
just my 2 eurocents,
BenLau
"@Test(threadId = "t1", shared = true)
public class A {
would allow TestNG to use this thread to run other test methods than on
class A, while shared = false would make sure that only A methods run
in
this thread. "
This looks like "micro-managing" the test too much.
I would prefer that TestNG pick a thread(any thread) and run all
methods of a class with this thread.
-BK-
mjl
here:
http://groups.google.com/group/testng-users/browse_frm/thread/1c7b144fd9272e52
So Adam not being the first person?!?
Cheers,
Michael
> ------=_Part_51781_29655677.1155238399947
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> X-Google-AttachSize: 1040
>
> <br><br><div><span class="gmail_quote">On 8/10/06, <b class="gmail_sendername">Justin Lee</b> <<a href="mailto:jl...@antwerkz.com">jl...@antwerkz.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> <br>-----BEGIN PGP SIGNED MESSAGE-----<br>Hash: RIPEMD160<br><br>Wouldn't it just be simpler to use a dependsOnMethod entry? It'd mean a<br>little more management on these tests, but less complexity in the<br>annotations and testng proper.
> </blockquote><div><br>This used to be my response to this kind of query, but Adam is making a reasonable case. Note that I said "reasonable", not "bullet-proof" :-)<br><br>I'm still waiting to hear more feedback about this, since so far, Adam seems to be the only one who had to face this limitation, and I'm not very keen on adding features for just one person (no offense, Adam :-)).
> <br><br></div></div>-- <br>Cédric<br>
>
> ------=_Part_51781_29655677.1155238399947--
Dies Koper
think we had a similar problem.
> I meant atomic at the class-level ("all methods in this class will be run in
> one thread without any test method from other classes interfering"). I
We had one test class's setup method messing up a running test's set up
as follows:
Class1.setup
Class1.t1()
Class2.setup
Class2.t1()
Class1.t2()
Class1.tearDown
Class2.t2()
Class2.tearDown
#i.e. class2's setup is called before class1's tearDown.
We don't run them in parallel, all test methods run "sequentially", it's
just that methods from different test classes are mixed.
My colleague solved that by running TestNG as follows:
<suite name="SUITE">
<test name="TEST">
<classes>
<class name="TestClass1"/>
</classes>
<classes>
<class name="TestClass2"/>
</classes>
</test>
</suite>
I haven't checked it out yet, but I hope there is a way to do this with
a filelist so we can use wildcards.
Regards,
Dies
BK Lau
I looked over the old thread that Cederic,Alex and you had last year 2005:
The statement that summed up best is by Cederic.
I quoted it here with liberty for those are who trying to follow this thread of discussion. I added my own clarifying tokens in brackets[... ]:
"[b]Everything I say below assumes that you are running TestNG in parallel mode.
As of today, the only way TestNG lets you specify that certain methods
should be run in the same thread [as if] it have dependencies on each other.
Michael[you] points out that he needs a different kind of control over the
sequential/parallel categorization that TestNG is defining today.
This discussion makes me[Cederic] realize that indeed, TestNG could do a better job at this, and that dependencies do not necessarily need[or merely] to be tied to
sequential executions[independent of calling threads]:
[that in some cases] you might want to guarantee that certain methods will
run in the same thread even though they don't necessarily depend on each
other.[to be executed sequentially]"[/b]
I really think this issue will come up again since we will have invariably more
single threaded JUnit 3.x tests to be converted as TestNG gained more and more traction.
-BK-
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=39932&messageID=79605#79605
Cédric Beust ♔
Thank you all for your patience.
--
Cedric
David Bernard
IMHO, like the parallele option is define in the xml, the not parallele
inside class must be define into xml like :
<suite parallel="true">
<test>
.. classes that are thread-safe
</test>
<test parallelInsideClass="false">
.. classes that are not thread-safe
</test>
my 2 cents suggestions
Alexandru Popescu
IMO doing this at the XML level may become a tedious job (once you
create a new test you need to edit 2 places, instead of a single one).
But I am still fighting the idea of having :
@Test(runMode=sequential)
public class MyTest {
/// my @Test-s
}
because of the reason I have already presented. Maybe we can introduce
another annotation: @ExecutionMode/@ExecutionStrategy. What do you
think?
