Running all methods in a class sequentially

4,657 views
Skip to first unread message

Adam

unread,
Aug 9, 2006, 6:27:23 PM8/9/06
to testng-users
Hi, we're using TestNG successfully here in my group at Amazon. Thanks
for the great software!

We would like a way we can run all methods in a class sequentially- the
order doesn't matter, but we have a lot of test classes and they are
not thread-safe.

The only way we've found to do this is make each test method depend on
the previous method... this is brittle and ugly... but it does do the
job. Is there a better way?

The reason we want this is:

1. It's a lot of effort to go back and make all our test classes and
methods thread safe.
2. It seems that it makes writing well factored test code harder, since
common variables and methods now have to be pulled out into separate
classes.

cheers
adam
--
Adam Feuer

Cédric Beust ♔

unread,
Aug 9, 2006, 6:31:37 PM8/9/06
to testng...@googlegroups.com
Hi Adam,

On 8/9/06, Adam <afe...@amazon.com> wrote:

Hi, we're using TestNG successfully here in my group at Amazon. Thanks
for the great software!

That's great to hear!

We would like a way we can run all methods in a class sequentially- the
order doesn't matter, but we have a lot of test classes and they are
not thread-safe.

By "sequentially", do you mean that you want all the test methods in a given class to be run next to each other, without any other test methods from other classes being run in-between?  This should be the default behavior, unless you are running in multithreaded mode?  Can you give more details on what you are seeing exactly?

--
Cédric

Feuer, Adam

unread,
Aug 10, 2006, 2:27:13 PM8/10/06
to testng...@googlegroups.com
Cedric Beust wrote:

> By "sequentially", do you mean that you want all the test methods in
> a given class to be run next to each other, without any other test
> methods from other classes being run in-between? This should be the
> default behavior, unless you are running in multithreaded mode? Can
> you give more details on what you are seeing exactly?

We are running in multithreaded mode to get parallelization. What we
want is for TestNG to run each test class' methods sequentially, not
in parallel. The parallelization will still give us a lot of
performance gain since other test classes can run in their own
threads.

Does that make sense?

What happens now by default is that TestNG will run each test method
in its own thread... since most of our test classes are not
thread-safe, things break. It's hard to see how to write thread safe
test classes either without a lot of code duplication or without a lot
of complication (introducing a lot of thread-safe "helper" classes).

What we do to avoid this problem is to make each test method dependent
on the previous test method, so that TestNG will run the methods in a
test class sequentially. This avoids threading issues with overwriting
variables and multiple threads calling the same helper method at the
same time. However, it is ugly, cumbersome, and brittle. Is there a
better way?

cheers
adam
--
Adam Feuer
afe...@amazon.com
Amazon.com Enterprise Services

Cédric Beust ♔

unread,
Aug 10, 2006, 3:10:51 PM8/10/06
to testng...@googlegroups.com
Hi Adam,

Okay, thanks for clarifying.

I understand your reluctance to making your class thread-safe just for the purpose of testing...  definitely not a good thing.

Maybe I could add an attribute to the @Test annotation that would mark the class as "sequential", or maybe a better word:  "atomic":

@Test(atomic = true)
public class A { ... }

What does everyone think about the idea and the naming?

--
Cedric
--
Cédric

Justin Lee

unread,
Aug 10, 2006, 3:28:59 PM8/10/06
to testng...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Wouldn't it just be simpler to use a dependsOnMethod entry? It'd mean a
little more management on these tests, but less complexity in the
annotations and testng proper.

Cédric Beust ? wrote:
> Hi Adam,
>
> Okay, thanks for clarifying.
>
> I understand your reluctance to making your class thread-safe just for
> the purpose of testing... definitely not a good thing.
>
> Maybe I could add an attribute to the @Test annotation that would mark
> the class as "sequential", or maybe a better word: "atomic":
>
> @Test(atomic = true)
> public class A { ... }
>
> What does everyone think about the idea and the naming?
>
> --
> Cedric
>
>

> On 8/10/06, * Feuer, Adam* <afe...@amazon.com

> afe...@amazon.com <mailto:afe...@amazon.com>
> Amazon.com <http://Amazon.com> Enterprise Services
>
>
>
>
>
>
> --
> Cédric
>
> >

- --
Justin Lee
http://www.antwerkz.com
AIM : evan chooly
Skype : evanchooly
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)

iD8DBQFE24j7JnQfEGuJ90MRAxpOAKCgxHv0Pxy7AxCfwSkelTzzVo2KowCgrlIT
hj3+5xTWI+ZqLgBG4s2kwwk=
=VMYz
-----END PGP SIGNATURE-----

Cédric Beust ♔

unread,
Aug 10, 2006, 3:33:19 PM8/10/06
to testng...@googlegroups.com
On 8/10/06, Justin Lee <jl...@antwerkz.com> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Wouldn't it just be simpler to use a dependsOnMethod entry?  It'd mean a
little more management on these tests, but less complexity in the
annotations and testng proper.

This used to be my response to this kind of query, but Adam is making a reasonable case.  Note that I said "reasonable", not "bullet-proof" :-)

I'm still waiting to hear more feedback about this, since so far, Adam seems to be the only one who had to face this limitation, and I'm not very keen on adding features for just one person (no offense, Adam :-)).

--
Cédric

BK Lau

unread,
Aug 10, 2006, 3:47:42 PM8/10/06
to testng...@googlegroups.com
Hi Cederic:

You said:
"[b]Maybe I could add an attribute to the @Test annotation that would mark the

class as "sequential", or maybe a better word: "atomic":
@Test(atomic = true)
public class A { ... }

What does everyone think about the idea and the naming?[/b] "

IMO: I think a better name would be

@Test(serially = true)
public class A { ... }


Incidentally, I have the same problem here at my work place where the classes are not thread-safe...since the Java classes that our devs wrote are just JNI wrappers to unsafe **legacy** C++ code...

-BK-
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=39932&messageID=79365#79365

Cédric Beust ♔

unread,
Aug 10, 2006, 3:54:50 PM8/10/06
to testng...@googlegroups.com, afe...@amazon.com
On 8/10/06, BK Lau <testng...@opensymphony.com> wrote:

Hi Cederic:

You said:
"[b]Maybe I could add an attribute to the @Test annotation that would mark the
class as "sequential", or maybe a better word: "atomic":
@Test(atomic = true)
public class A { ... }
What does everyone think about the idea and the naming?[/b] "

IMO: I think a better name would be

@Test(serially = true)
public class A { ... }

The problem with "serially" is that it just says that these methods will be run... well, "serially".  There is no notion that no methods from other classes will be run in-between, which "atomic" conveys better in my opinion...

Incidentally, I have the same problem here at my work place where the classes are not thread-safe...since the Java  classes that our devs wrote are just JNI wrappers to unsafe **legacy** C++ code...

So how do you solve this problem?  You run this class in non-multithreaded code?

By the way, Adam, this would be one way to solve your problem:  declare your suite parallel, but put all the classes that are not thread-safe in a non-parallel test:

<suite parallel="true" >

<test>
  .. classes that are thread-safe
</test>

<test parallel="false">
  .. classes that are not thread-safe
</test>

Did you try this?

--
Cédric

Alexandru Popescu

unread,
Aug 10, 2006, 4:01:27 PM8/10/06
to testng...@googlegroups.com
The only thing I don't like about this idea is that we are going to
face once again the discussion about the annotation inheritance. Other
than this, I find it quite cool and as Adam is showing it is pretty
needed.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder

Feuer, Adam

unread,
Aug 10, 2006, 4:41:37 PM8/10/06
to Cédric Beust ♔, testng...@googlegroups.com
> By the way, Adam, this would be one way to solve your
> problem: declare your suite parallel, but put all the
> classes that are not thread-safe in a non-parallel test:

Cedric,

This removes the thread-safety problem, but then also prevents most of our unit and integration tests from being run in parallel- which is the main reason we are using in TestNG. We were able to speed up our tests 10x by using TestNG's parallel mode - we went from a 40 minute run to a 4 minute run.

> Maybe I could add an attribute to the @Test annotation that
> would mark the class as "sequential", or maybe a better word:
> "atomic":

I like "sequential" better... by "atomic", do you mean that each test method from that class will be run by itself, and will not overlap with another test method from the same class? If so then "atomic" is also ok.

cheers
adam
--
Adam Feuer
afe...@amazon.com 206-266-4726
Amazon.com Enterprise Commerce Services

Cédric Beust ♔

unread,
Aug 10, 2006, 4:50:26 PM8/10/06
to Feuer, Adam, testng...@googlegroups.com
On 8/10/06, Feuer, Adam <afe...@amazon.com> wrote:
> By the way, Adam, this would be one way to solve your
> problem:  declare your suite parallel, but put all the
> classes that are not thread-safe in a non-parallel test:

Cedric,

This removes the thread-safety problem, but then also prevents most of our unit and integration tests from being run in parallel- which is the main reason we are using in TestNG. We were able to speed up our tests 10x by using TestNG's parallel mode - we went from a 40 minute run to a 4 minute run.

Ok.

> Maybe I could add an attribute to the @Test annotation that
> would mark the class as "sequential", or maybe a better word:
> "atomic":

I like "sequential" better... by "atomic", do you mean that each test method from that class will be run by itself, and will not overlap with another test method from the same class? If so then "atomic" is also ok.

I meant atomic at the class-level ("all methods in this class will be run in one thread without any test method from other classes interfering").  I realize now that your requirement is a bit weaker than this:  you just want the methods in this class to be run in the same thread (thereby guaranteeing that they won't overlap each other) but you don't really care if other test methods from different classes are invoked in-between.  Correct?

If I got that right, it might then make more sense to be able to specify a thread-id:

@Test(threadId = "t1")
public class A { ... }

which means that all the test methods in this class will be run in the thread named "t1" (an arbitrary name, just a way for TestNG to differentiate certain threads from its thread pool).

Potential problem:  you define a thread pool of five threads but you accidentally specify six threadId's throughout your test classes.  I suppose TestNG could throw an exception in this case, or maybe merge threads together if they are shared.  Which might bring up the need for another attribute:

@Test(threadId = "t1", shared = true)
public class A {

would allow TestNG to use this thread to run other test methods than on class A, while shared = false would make sure that only A methods run in this thread.

Thoughts?

--
Cédric

Feuer, Adam

unread,
Aug 10, 2006, 5:02:28 PM8/10/06
to Cédric Beust ♔, testng...@googlegroups.com
Cedric Beust wrote:

> I meant atomic at the class-level ("all methods in this class
> will be run in one thread without any test method from other
> classes interfering").

Cedric,

This would be fine.

> I realize now that your requirement
> is a bit weaker than this: you just want the methods in this
> class to be run in the same thread (thereby guaranteeing that
> they won't overlap each other) but you don't really care if
> other test methods from different classes are invoked
> in-between. Correct?

Yes, this is correct. We want test methods from one class not to overlap with each other- we don't care whether other test methods from different classes are running at the same time or in between them.

> If I got that right, it might then make more sense to be able
> to specify a thread-id:

This seems cumbersome, as it removes the possibility that TestNG can optimize which classes should be run in which threads. I don't really want that kind of control, as I'd rather not think about which threads are doing what. As you mentioned, the possibility for me screwing things up seems great.

I prefer the "sequential" or "atomic" method you mentioned before- this would just tell TestNG not to overlap test methods from the same class. That way TestNG can decide the best way to do this.

-adam

Alexandru Popescu

unread,
Aug 10, 2006, 5:25:49 PM8/10/06
to testng...@googlegroups.com, Feuer, Adam
On 8/10/06, Cédric Beust ♔ <cbe...@google.com> wrote:
>
>
>

What would be the real benefit of this? The user have no access to the
thread used to run the tests, so using names would help to almost
nothing imo (but I may be wrong).

./alex
--
.w( the_mindstorm )p.
TestNG co-founder

> --
> Cédric
>
>
> >
>

Cédric Beust ♔

unread,
Aug 10, 2006, 5:29:56 PM8/10/06
to testng...@googlegroups.com, Feuer, Adam
On 8/10/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

> If I got that right, it might then make more sense to be able to specify a
> thread-id:
>
> @Test(threadId = "t1")
> public class A { ... }

What would be the real benefit of this? The user have no access to the
thread used to run the tests, so using names would help to almost
nothing imo (but I may be wrong).

It's just a symbolic name, a way to tell TestNG "pick a thread from your thread pool, call it "t1" and only put methods from this class on that thread".

--
Cédric

Alexandru Popescu

unread,
Aug 10, 2006, 5:36:58 PM8/10/06
to testng...@googlegroups.com

IMO this looks more like an implementation-oriented annotation, than
the more generic/readable form sequential/atomic.

just my 2 eurocents,

BenLau

unread,
Aug 10, 2006, 5:41:03 PM8/10/06
to testng-users
Cederic:

"@Test(threadId = "t1", shared = true)
public class A {


would allow TestNG to use this thread to run other test methods than on

class A, while shared = false would make sure that only A methods run
in
this thread. "

This looks like "micro-managing" the test too much.
I would prefer that TestNG pick a thread(any thread) and run all
methods of a class with this thread.

-BK-

mjl

unread,
Aug 10, 2006, 7:24:28 PM8/10/06
to testng-users
I'm not sure, but isn't this somehow the same as previously discussed
here:
http://groups.google.com/group/testng-users/browse_frm/thread/1c7b144fd9272e52

So Adam not being the first person?!?

Cheers,

Michael

> ------=_Part_51781_29655677.1155238399947
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> X-Google-AttachSize: 1040
>
> <br><br><div><span class="gmail_quote">On 8/10/06, <b class="gmail_sendername">Justin Lee</b> &lt;<a href="mailto:jl...@antwerkz.com">jl...@antwerkz.com</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> <br>-----BEGIN PGP SIGNED MESSAGE-----<br>Hash: RIPEMD160<br><br>Wouldn't it just be simpler to use a dependsOnMethod entry?&nbsp;&nbsp;It'd mean a<br>little more management on these tests, but less complexity in the<br>annotations and testng proper.
> </blockquote><div><br>This used to be my response to this kind of query, but Adam is making a reasonable case.&nbsp; Note that I said &quot;reasonable&quot;, not &quot;bullet-proof&quot; :-)<br><br>I'm still waiting to hear more feedback about this, since so far, Adam seems to be the only one who had to face this limitation, and I'm not very keen on adding features for just one person (no offense, Adam :-)).
> <br><br></div></div>-- <br>Cédric<br>
>
> ------=_Part_51781_29655677.1155238399947--

Dies Koper

unread,
Aug 10, 2006, 8:22:32 PM8/10/06
to testng...@googlegroups.com
Not really adding much to the conversation, just letting you know I
think we had a similar problem.

> I meant atomic at the class-level ("all methods in this class will be run in
> one thread without any test method from other classes interfering"). I

We had one test class's setup method messing up a running test's set up
as follows:

Class1.setup

Class1.t1()

Class2.setup

Class2.t1()
Class1.t2()

Class1.tearDown

Class2.t2()

Class2.tearDown

#i.e. class2's setup is called before class1's tearDown.

We don't run them in parallel, all test methods run "sequentially", it's
just that methods from different test classes are mixed.

My colleague solved that by running TestNG as follows:

<suite name="SUITE">
<test name="TEST">
<classes>
<class name="TestClass1"/>
</classes>
<classes>
<class name="TestClass2"/>
</classes>
</test>
</suite>

I haven't checked it out yet, but I hope there is a way to do this with
a filelist so we can use wildcards.

Regards,
Dies

BK Lau

unread,
Aug 11, 2006, 4:00:27 PM8/11/06
to testng...@googlegroups.com
Michael:
I looked over the old thread that Cederic,Alex and you had last year 2005:
The statement that summed up best is by Cederic.
I quoted it here with liberty for those are who trying to follow this thread of discussion. I added my own clarifying tokens in brackets[... ]:

"[b]Everything I say below assumes that you are running TestNG in parallel mode.
As of today, the only way TestNG lets you specify that certain methods
should be run in the same thread [as if] it have dependencies on each other.
Michael[you] points out that he needs a different kind of control over the
sequential/parallel categorization that TestNG is defining today.

This discussion makes me[Cederic] realize that indeed, TestNG could do a better job at this, and that dependencies do not necessarily need[or merely] to be tied to
sequential executions[independent of calling threads]:
[that in some cases] you might want to guarantee that certain methods will
run in the same thread even though they don't necessarily depend on each
other.[to be executed sequentially]"[/b]

I really think this issue will come up again since we will have invariably more
single threaded JUnit 3.x tests to be converted as TestNG gained more and more traction.

-BK-
---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=39932&messageID=79605#79605

Cédric Beust ♔

unread,
Aug 11, 2006, 4:32:15 PM8/11/06
to testng...@googlegroups.com
Point taken, you no longer need to convince me of the usefulness of being able to specify that all the test methods of a given class should not overlap.  I'll think of general way to solve this problem.

Thank you all for your patience.

--
Cedric

David Bernard

unread,
Aug 11, 2006, 4:37:09 PM8/11/06
to testng-users
Hi,

IMHO, like the parallele option is define in the xml, the not parallele
inside class must be define into xml like :

<suite parallel="true">

<test>
.. classes that are thread-safe
</test>

<test parallelInsideClass="false">
.. classes that are not thread-safe
</test>


my 2 cents suggestions

Alexandru Popescu

unread,
Aug 11, 2006, 4:47:19 PM8/11/06
to testng...@googlegroups.com

IMO doing this at the XML level may become a tedious job (once you
create a new test you need to edit 2 places, instead of a single one).
But I am still fighting the idea of having :

@Test(runMode=sequential)
public class MyTest {
/// my @Test-s
}

because of the reason I have already presented. Maybe we can introduce
another annotation: @ExecutionMode/@ExecutionStrategy. What do you
think?

BK Lau

unread,
Aug 11, 2006, 5:08:53 PM8/11/06
to testng...@googlegroups.com
Pls explain your thoughts further. Some examples.

Shashi Prakash Mall

unread,
Jan 25, 2019, 6:13:23 PM1/25/19
to testng-users
its 2019. New to testNG.
Do we have the solution available to run classes in sequence independent of others.

Thanks,
Shashi
Reply all
Reply to author
Forward
0 new messages