order of execution of tests

[shashank ( shantaram )]

Is there a guarantee that multiple tests in a testng.xml file will be
executed in the order they are declared in the file ?
Assume parallel option is none and no dependson annotations are used.

I have exhaustively tested this and the test are getting executed in
right order.
But wanted a confirmation before my team gets down to writing the
xml's
eg -- in follopwing example, will method2 always run after method1

<suite name="TESTNG-59 Suite" >
<test name="TESTNG-1">
<classes>
<class name="test.class1">
<methods>
<include name="method1"/>
</methods>
</class>
</classes>
</test>
<test name="TESTNG-2">
<classes>
<class name="test.class2">
<methods>
<include name="method2"/>
</methods>
</class>
</classes>
</test>

</suite>

[Cédric Beust ♔]

On Sun, Nov 8, 2009 at 6:18 AM, shashank ( shantaram ) <svwain...@gmail.com> wrote:

Is there a guarantee that multiple tests in a testng.xml file will be
executed in the order they are declared in the file ?

No, the XML file only contains the list of classes to be analyzed by TestNG.

If you need a specific ordering, you should use dependsOnGroups/dependsOnMethods.

--

Cédric

[Shantaram]

Please suggest how i can implement the following requirement.

I have 3 @test methods A,B and C testing three tasks.

I want to test multiple test flows

eg

test 1  - run A -> run B -> run C

test 2  - run A -> run C -> run B
test 3  - run B -> run C -> run A

I want to write many such flows

I dont mind writing an xml for each flow but i should be able to do it without changing the annotations in the java test code (needing re compiling)

-- 
Regards,

Shantaram Waingankar
9969035764
www.shantaram.co.nr

Twenty years from now you will be more disappointed by the things you didn't do than by the ones you did do. 
So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. 
Explore. Dream. Discover.

2009/11/8 Cédric Beust ♔ <cbe...@google.com>

Cédric

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com
To unsubscribe from this group, send email to testng-users...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en

-~----------~----~----~----~------~----~------~--~---

[Harihara Vinayakaram]

Technically the order of tests in testng.xml can change. But for now in a sequential flow this works .  So you can write your testng.xml as

<test .... >
   <include name="A" />
</test>
<test ...>
    <include name="B" />
</test>

and you can pass the appropriate testng.xml to your TestNG ant task

etc.
But beware that this is not fixed and can change as Cedric mentioned in a post some time back. 

Regards
Hari

--

You received this message because you are subscribed to the Google Groups "testng-users" group.

To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[anil raju]

As a part of our test framework we have multiple test files and some
of the workflows demand to include the tests of a particular suite
into another one and also ensure that the tests of the included suite
files are run in the sequence of inclusion.

On looking up what i see is that the suite-files tag can have multiple
test files and all the tests in them run too. But the sequence of
running the tests is not guaranteed.

A standard testng file with included suites will look like this.
eg.
<suite name="Suite1" verbose="1">
<suite-files>
<suite-file path="testng1.xml" />
<suite-file path="testng2.xml" />
</suite-files>
<test name="test1">
<classes>
<class name="com.abc.test">
<methods>
<include name="test1()" />
</methods>
</class>
</classes>

<./test>
</suite>

What i want to ensure is that the tests in testng1.xml are run before
testng2.xml and the tests in this file needs to be run at the end. Is
there any existing method which i could use to make the tests run in
this order.

Regards,
Anil

--
Don't talk unless you can improve the silence.
Visit my blog at
http://anilraju.co.nr

[Anna-onNabble]

Anil,
Have you figured this out?
Thanks,
Anna

--
View this message in context: http://old.nabble.com/order-of-execution-of-tests-tp26254836p29474824.html
Sent from the testng-users mailing list archive at Nabble.com.

[Chuck Brunson]

Cedric recently made a fix to make preserve-order work and its working
now for me. It was NOT working for me using 5.13.1.

You can get it from:
http://testng.org/beta/

It is testng-5.14beta.zip

Chuck Brunson
719-651-6259(w)
719-282-1373(h)
chuckb...@yahoo.com

--- On Wed, 8/18/10, Anna-onNabble <asul...@yahoo.com> wrote:

[Cédric Beust ♔]

On Wed, Aug 18, 2010 at 12:45 PM, Chuck Brunson <chuckb...@yahoo.com> wrote:

Cedric recently made a fix to make preserve-order work and its working
now for me. It was NOT working for me using 5.13.1.

You can get it from:
http://testng.org/beta/

It is testng-5.14beta.zip

I think the bug I fixed for Chuck will not help for the issue at hand (I can give more details if there is interest).

The issue is to run the <test> tags in the order in which they are encountered, which is not guaranteed right now.

I'm wondering about adding a preserve-test-order attribute at the suite level:

<suite name="foo" preserve-test-order="true">

Things get a bit more unclear in the presence of suite-files, since these define hierarchies of suite files. In this case, TestNG would have to pick a graph order traversal (depth first I guess) to choose a test ordering.

--
Cédric

[Felipe Knorr Kuhn]

Anna,

A while ago I wrote a Method Interceptor that parses the XML file and executes the tests in the exact order as defined there.

You can check it out here: http://gist.github.com/416310

However, with Cédric's recent implementation of the preserve-order attribute, it's probably not as useful anymore, specially because you have to declare every method you want to run and it also breaks TestNG's dependency resolver algorithm.

As we are discussing on a different thread, if you want to preserve the order of the test methods, you still have to declare them on the XML file.

I haven't compared the results produced by TestNG against my own Interceptor class yet, so feel free to experiment with it and share your results :)

FK

On Wed, Aug 18, 2010 at 8:33 PM, Anna-onNabble <asul...@yahoo.com> wrote:

Cedric,

What I really need is a way to guarantee execution order of test classes.
(The classes have internal method dependencies in them.)
If it is at all possible, what is the best way to achieve this right now?

Thanks,
Anna

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/order-of-execution-of-tests-tp26254836p29477095.html

[Anna-onNabble]

Cedric,

What I really need is a way to guarantee execution order of test classes.
(The classes have internal method dependencies in them.)
If it is at all possible, what is the best way to achieve this right now?

Thanks,
Anna

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/order-of-execution-of-tests-tp26254836p29477095.html

[Cédric Beust ♔]

Anna,

<test preserve-order="true"> should do exactly what you want.

--
Cédric

Cédric

[Anna-onNabble]

Yes, I have seen that in the doc. But I also read somewhere that
"preserve-order was not preserving class order with dependent methods" in
5.13.1. Which version has that been fixed in?

>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

>> .
>> > For more options, visit this group at
>> > http://groups.google.com/group/testng-users?hl=en.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/order-of-execution-of-tests-tp26254836p29477095.html
>> Sent from the testng-users mailing list archive at Nabble.com.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "testng-users" group.
>> To post to this group, send email to testng...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

>> .
>> For more options, visit this group at
>> http://groups.google.com/group/testng-users?hl=en.
>>
>>
>
>
> --
> Cédric
>

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/order-of-execution-of-tests-tp26254836p29477494.html

[Cédric Beust ♔]

On Wed, Aug 18, 2010 at 5:47 PM, Anna-onNabble <asul...@yahoo.com> wrote:

Yes, I have seen that in the doc. But I also read somewhere that
"preserve-order was not preserving class order with dependent methods" in
5.13.1. Which version has that been fixed in?

If your methods only depend on each other in the same class, this will work.

The case that doesn't work is when methods depend on each other across classes, because in this case, the two orderings can conflict with each other.

--
Cédric

[Anna-onNabble]

Great. Except maven-surefire-plugin throws the following exception after I
updated my pom.xml to declare testng 5.13.1 (confirmed this does not happen
with 5.12.1) :

org.apache.maven.surefire.booter.SurefireExecutionException:
java.util.ArrayList cannot be cast to java.lang.String;
nested exception is java.lang.ClassCastException: java.util.ArrayList cannot
be cast to java.lang.String
java.lang.ClassCastException: java.util.ArrayList cannot be cast to
java.lang.String
at org.testng.TestNG.configure(TestNG.java:1370)
at
org.apache.maven.surefire.testng.conf.TestNGMapConfigurator.configure(TestNGMapConfigurator.java:97)
at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:58)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:144)
at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at
org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)

Here is my pared down testng.xml:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Component Tests" verbose="10" annotations="JDK">
<test name="Functional Tests" preserve-order="true">
<classes>
<class name="server.PrgTest" />
<class name="server.SibTest" />
<class name="server.SchTest" />
<class name="server.PpvTest" />
<class name="server.EdnTest" />
</classes>
</test>
</suite>

And the surefire plugin configuration in the pom is as follows:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<version>2.6</version>
<configuration>
<suiteXmlFiles>

<suiteXmlFile>${project.basedir}/src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
<properties>
<c.home>${c.home}</c.home>
<c.config.file>c.props</c.config.file>
</properties>
</configuration>
</plugin>

Am I missing something here?

Thanks,
Anna

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>
>
>

--
View this message in context: http://old.nabble.com/order-of-execution-of-tests-tp26254836p29478898.html

[Cédric Beust ♔]

Anna, can you email me the pom.xml that you are using?

Thanks.

--
Cédric

Cédric

[Cédric Beust ♔]

Anna,

I think TestNG is working as expected: you have dependencies across your classes which conflict with "preserve-order".

For example, preserve-order tries to run SibTest first, but SibTest has dependencies on group "g3", and group g3 contains methods in all three classes, so TestNG needs to run all these methods first.

Hope this clarifies things.

--
Cédric

2010/8/19 Anna Sulkina <asul...@yahoo.com>

Hi Cédric,

Thanks for such fast responses!

I created a new test project with only 3 short test classes that have method dependencies and groups, which to some degree reflects my real scenario.
And I don't get that exception. In fact, when I removed the parent declaration from my production POM, no exception is thrown.

BUT, the test classes are not executed in the order declared in the suite xml file.
I am attaching my simple test project, so you can see for yourself. (The zip files also includes the output of the test.)

Thanks,
Anna

---

From: Cédric Beust ♔ <ced...@beust.com>

To: Anna Sulkina <asul...@yahoo.com>; Brett Porter <brett....@gmail.com>
Sent: Thu, August 19, 2010 11:33:03 AM

Subject: Re: [testng-users] order of execution of tests

+Brett

Anna, I can't seem to reproduce the problem.

I am using the same configuration as you: Surefire 2.6 and:

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>5.13.1</version>
            <scope>test</scope>
        </dependency>

        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.6</version>
        <configuration>
          <properties>
            <cats.home>catsHome</cats.home>
            <cats.config.file>catsProps</cats.config.file>
          </properties>
          <suiteXmlFiles>
            <suiteXmlFile>src/test/resources/testng-single.xml</suiteXmlFile>
          </suiteXmlFiles>
        </configuration>

The problem seems to be that Surefire is passing a String[] and TestNG expects a String, but I don't see any of this neither in the Surefire code nor in TestNG. I am able to run my tests with Surefire and TestNG 5.13.1 without seeing the ClassCastException that you are seeing, which I'm reproducing for Brett below.

Brett, any idea where Surefire might be passing a String[] in that map that TestNG might cast impropriately? And also any thought why I can't reproduce this? Can you?

Anna, are you positive that I am reproducing the correct scenario?


org.apache.maven.surefire.

booter.SurefireExecutionException:
java.util.ArrayList cannot be cast to java.lang.String;
nested exception is java.lang.ClassCastException: java.util.ArrayList cannot
be cast to java.lang.String
java.lang.ClassCastException: java.util.ArrayList cannot be cast to
java.lang.String
       at org.testng.TestNG.configure(TestNG.java:1370)
       at
org.apache.maven.surefire.testng.conf.TestNGMapConfigurator.configure(TestNGMapConfigurator.java:97)
       at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:58)
       at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:144)
       at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at
org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
       at
org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)

2010/8/19 Anna Sulkina <asul...@yahoo.com>

Hi Cédric,

The pom.xml file is attached.

Thanks,
Anna

---

From: Cédric Beust ♔ <ced...@beust.com>
To: testng...@googlegroups.com; asul...@yahoo.com
Sent: Thu, August 19, 2010 7:30:13 AM
Subject: Re: [testng-users] order of execution of tests

--
Cédric

--
Cédric

--
Cédric