Does parallel="false" actually work?

[JR Boyens]

I've been trying for quite a while to stop the parallel execution of tests by putting parallel="false" or thread-count="1" or both in both the <suite> tag and the <test> tag with absolutely no results. With verbose="10" on I see that all the tests are added to the "PARALLEL LIST" regardless of what I do in the TestNG xml file. This of course is only necessary because I'm trying to work around another problem...

My tests extend a common class. In this class I have a beforeTestSuite method and a few beforeTestMethod and afterTestMethod methods. I feel this is a valid use as these are common to ALL of my tests and reproducing them is just silly. Unfortunately even though my tests are in the same suite, the beforeTestSuite method runs twice causing an error because part of the test is not thread-safe.

Because of these two errors I generally need to run tests a few times until I get past the thread-safe problem by sheer luck, or I need to disable individual classes in the xml file. Which means that I can't use <packages> only <classes>.

Any ideas?

---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=7207&messageID=16012#16012

[JR Boyens]

I seem to have fixed the beforeSuite() problem by moving the method out into a separate non-extending class and making the common test class abstract.

Although the parallel problem still stands.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=7207&messageID=16014#16014

[Cédric Beust ♔]

Hi JR,

About the parallel thing:  don't worry too much about what the message says (it's a debug message after all :-)).

When it gives the "parallel list", it's just a list of all the methods that *could* be invoked in parallel (in other words, methods that don't have dependencies on each other).

They won't be invoked in parallel if you set parallel="false" or parallel="true" with thread-count="1".  You can convince yourself of that by looking at the chronological list of methods after your tests have run and see the thread ID for each of them:  it should be the same everywhere.

Does this make sense?

--
Cedric

--
Cédric

[Cédric Beust ♔]

The beforeSuite() problem is more tricky, and we dicussed it here not long ago (last 2-3 weeks I believe).

If you have a beforeSuite() in a base class and two subclasses A and B, it will appear in both A and B, so TestNG will see it twice and therefore, register it twice.

We were discussing the possibility to make exceptions to this rule for beforeSuite (and possibly, beforeTest and beforeClass), but I'm a bit reluctant to the idea because it breaks consistency and it makes the feature harder to describe (usually a design smell in my opinion).

The workaround is to move these methods you only want TestNG to see once in a separate class.  Unfortunately, it's not always possible or easy (these methods may want to share state with the others), so I'm still left on the fence on what the best way to solve this problem is.

Here is a new idea:  how about adding a unique attribute to @Test and @Configuration?  If it is set to true, TestNG guarantees that this method will only be invoked once during the suite run...

Just thinking out loud, haven't thought about the consequences yet...

What do you think?

--
Cedric

--
Cédric

[JR Boyens]

> The beforeSuite() problem is more tricky, and we
> dicussed it here not long
> ago (last 2-3 weeks I believe).
>
> If you have a beforeSuite() in a base class and two
> subclasses A and B, it
> will appear in both A and B, so TestNG will see it
> twice and therefore,
> register it twice.
>
> We were discussing the possibility to make exceptions
> to this rule for
> beforeSuite (and possibly, beforeTest and
> beforeClass), but I'm a bit
> reluctant to the idea because it breaks consistency
> and it makes the feature
> harder to describe (usually a design smell in my
> opinion).

I don't think this is the case at all. It runs before the suite. Period. It doesn't matter if 100 classes extend the exact same class. It should only ever run once. I just don't see any reason for it to run more than once in any circumstance. As far as I'm concerned, I don't EVER expect anything to run twice unless I ask it to. If I ask for beforeMethod I realize that it will be run before the method. But should I add checks to see that I doesn't run 3 times before the method? Or 10 for that matter?

It's an unexpected behavior, which in my opinion means it's a bug. I just can't seem to think of a reason why I would want a test or before* method to run an un-obvious number of times, ever.

>
> The workaround is to move these methods you only want
> TestNG to see once in
> a separate class. Unfortunately, it's not always
> possible or easy (these
> methods may want to share state with the others), so
> I'm still left on the
> fence on what the best way to solve this problem is.
>
> Here is a new idea: how about adding a unique
> attribute to @Test and
> @Configuration? If it is set to true, TestNG
> guarantees that this method
> will only be invoked once during the suite run...
>
> Just thinking out loud, haven't thought about the
> consequences yet...
>
> What do you think?
>
> --
> Cedric

As I said above, I just don't think that adding an attribute makes design sense. I've moved to your suggested fix and yes, it does work fine. But that seems to be a workaround to a deeper problem with TestNG's inspection of the test classes.

I'm finally getting back to using TestNG after a long hiatus because of severe problems I had with the inspection and the detection of test methods and the running order of the methods. It seems like TestNG has come a long way, but IIRC this was one of the most annoying problems.

As a user my impression was that I should just be able to mark a method beforeSuite and it would be run before the suite once and only once. As a developer I understand the reasoning behind why it's doing what it's doing. But I think that you can certainly agree that running a method more than once without, at the very least, erroring or warning the user about is a bad thing.

I hope I didn't come off as angry or arrogant here, if I did you have my sincerest apologies. TestNG is a tool that I see as something that will really really help me in my work and so I have a vested interest in seeing it be all that it can be and to be designed as well as it can be.

The next step after this, of course, is getting the Eclipse plugin to read the console data and provide the graphical feedback even for more... exotic testing setups. :)

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=7207&messageID=16190#16190

[Alexandru Popescu]

#: JR Boyens changed the world a bit at a time by saying on 9/28/2005 7:40 AM :#

JR I would like to askk you how do you see/expect TestNG to behave in the following scenarios:

ParentClass:
- beforeTestClass
- beforeTestMethod
- testMethodOnParent1
- testMethodOnParent2
- afterTestMethod
- afterTestClass

ChildClass extends ParentClass:
- testMethodOnChild

scenario:

1. run TestNG with only ChildClass

2. run TestNG with both classes

I think the above will make me better understand your expectations. Thanks a lot.

./alex
--
.the_mindstorm.

[Cédric Beust ♔]

Okay JR, I'm (almost) convinced about before/afterSuite.

Now, what do you think of beforeClass and beforeTest?

--
Cedric

--
Cédric

[JR Boyens]

I see exactly where you are going with this. I'm having a narrow viewpoint and seeing parent classes as only parents and not as full-fledged test classes. This makes my scenario 2 not exactly what I would ever want to do, but it would be what would make the most sense. I guess my problem mostly exists with beforeSuite then. The clearest fix would be to have a Set of the fully-qualified method names of the beforeSuite methods and then run each one. That would allow for only one no matter the parent-child relationships. In my mind though, the parent class is an abstract class used only to provide a base for the extending classes. The problem lies with multiple extending classes and deciding whether or not to run each beforeTestClass before each child class or before the single ParentClass. In other words, running on a flattened namespace (run for each child) or on a globally-unique one (run once no matter the number of children).

In the context of beforeTestClass it becomes a more difficult problem. beforeTestMethod and beforeSuite however are a different story. Since beforeSuite by nature runs before multiple tests thus it seems you'd want it to run only once at the beginning of the tests in that suite. beforeTestMethod should be run before every test method in the scope of the classes below it in the tree. In other words, a beforeTestMethod in ParentClass1 shouldn't affect ParentClass2, but should affect ChildOfParentClass1.

So then the problem lies with the behavior of beforeTestClass and which class you mean. Before the Parent, before the Child, or both. I think the best idea is for it to run before both and state so in the documentation. It's what seems most natural. Having a globally-unique namespace for beforeTestClass (or afterTestClass) just doesn't seem like that hot of an idea. The interactions here could be... interesting depending on the pre-conceptions the developer comes to the table.

Which would, of course, be a strong argument to define the behavior in the documentation very bluntly.

You've put the problem in a different perspective for me.
Thanks.

Scenario 1:
beforeTestClass
beforeTestMethod
testMethodOnParent1
afterTestMethod

beforeTestMethod
testMethodOnParent2
afterTestMethod

beforeTestMethod
testMethodOnChild
afterTestMethod
afterTestClass

Scenario 2:
ParentClass:
Same as above except for the testMethodOnChild set
ChildClass:
Same as above

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=7207&messageID=16201#16201

[Alexandru Popescu]

Hi!

Resuming:

1/ @Configuration method at suite and test level should be considered in a flattened namespace
2/ @Configuration method at class and method level should behave in a hierarchical namespace

I tend to agree that this is making sense to me too.

./alex
--
.the_mindstorm.


#: JR Boyens changed the world a bit at a time by saying on 9/28/2005 9:21 AM :#

[Justin Lee]

I agree with this, too. with beforeTestClass, each subclass *is* a
separate test so i would expect to see multiple runs. beforeTestMethod
I would expect to behave the same. Suites, however, are not tied to a
particular class, so I would expect to see that run only once.

--
Justin Lee
http://www.antwerkz.com
AIM : evan chooly
720.299.0101