Ordering test methods per instance?

[Torsten Reinhard]

Hi,

how can I execute (dependant) test methods on instances of one class,
instance by instance?

I have a class like:

public MyClass
{
Object testdata;
public MyClass( Object testdata )
{
this.testdata = testdata;
}

@Test
public void test1()

@Test(dependsOnMethod="test1")
public void test2()

@Test(dependsOnMethod="test2")
public void test3()
}

And I have a factory:

public MyClassFActory
{
@Factory
public Object[] create()
{
return new Object[] {
new MyClass1( getTestData(1)),
new MyClass1( getTestData(2))
};
}
}

What I want is the following order:

MyClass(id=1).test1();
MyClass(id=1).test2();
MyClass(id=1).test3();

MyClass(id=2).test1();
MyClass(id=2).test2();
MyClass(id=2).test3();

But TestNG does:

MyClass(id=1).test1();
MyClass(id=2).test1();

MyClass(id=1).test2();
MyClass(id=2).test2();

MyClass(id=1).test3();
MyClass(id=2).test3();

Is thera any way instead of implementing a special IMethodInterceptor,
that manages the "dependsOn" logic, after having sort the methods by
instance?

Thanx, Torsten

[Cédric Beust ♔]

Hi Torsten,

This is currently not possible since it goes against the semantic of "dependsOnMethods": no matter how many instances of a method test1() are run, if test2() depends on it, then test2() can't start until after all the instances of test1() have completed successfully.

To implement your own ordering, your best bet is probably to remove all the dependencies, implement an IMethodInterceptor and do your own ordering there...

--
Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[torsten....@gi-de.com]

Hi,

thanx, but that´s sometimes not so easy...
It´s much more easier and clearer to define the execution order manually, than to re-sort it afterwards by analyzing the instances, the methodnames or even annotations.

I guess it´s a common request to have a complex test scenario based on dependant test steps, that should be run an varying testdata:

public class ClientTest
{
        private Client client;

        @Test
        public void testCreateClient()
        {
        }
        @Test
        public void testConfigureClient()
        {
        }
        @Test
        public void testCreateCardsClient()
        {
        }
        @Test
        public void testPrintCardsClient()
        {
        }
        @Test
        public void testDeleteClient()
        {
        }
}

Isn´t there any equivalent to the old "JUnit TestSuite" where Tests could be added manually and than executed in exactly that order?

thanx, Torsten

Cédric Beust ♔ <ced...@beust.com> Gesendet von: testng...@googlegroups.com 24.08.2010 18:20 Bitte antworten an testng...@googlegroups.com

An testng...@googlegroups.com Kopie Thema Re: [testng-users] Ordering test methods per instance?

[torsten....@gi-de.com]

Hi,

I have some testng.xml (suite) files,

01_setup.xml
02_basetests.xml
03_clienttests.xml
04_cardtests.xml

that should run in exactly that order.

When I would define groups at the @Test annotation at the test methods or classes within each suite, they will be executed in the order they depend on I guess, but I loose the ability to execute
a suite separately:

org.testng.TestNGException:
Method "com....basetests.Start.testStepLoginSysAdmin()" depends on nonexistent group "setupsystem"
        at org.testng.TestRunner.computeAlternateTestList(TestRunner.java:1095)
        at org.testng.TestRunner.privateRun(TestRunner.java:692)
        at org.testng.TestRunner.run(TestRunner.java:553)

the reason why the tests dependson other tests is some created data and state in a common database.
So what I want is:

run the tests in numerical order (01_setup, 02_basetests, 03_clienttests, 04_cardtests) if run "all at once" from testng.xml:

<suite name="Uebergabe">
        <suite-file path="01_setup.xml"/>
        <suite-file path="02_basetests.xml"/>
        <suite-file path="03_clienttests.xml"/>
        <suite-file path="04_cardtests.xml"/>
</suite>

be able to rerun 03_clienttests or 04_cardtests.xml, after 01_setup.xml and 02_basetests.xml have been run at least one time.

I guess what could help would be the ability to define the dependencies not at source-level (@Test(groups="myGroup"), @Test(dependsOnGroups="myGroup") ,
but at suite.xml level.

=> any other idea how to manage the execution order of suites?

Thanx, Torsten

[torsten....@gi-de.com]

no answer to the question about how to keep the execution order of suites ? Why isn´t there a "preserve-order" Flag on Suite level ?
So here´s a workaround:

To keep the execution order of

01_setup.xml
02_basetest.xml
03_clienttests.xml
04_cardtests.xml

AND the ability to restart each suite itself (after predecessor suite succeeded) just add the following class to each suite configuration:


@Test(groups={"setupsystem",..., ...})
public class DependsOnGroupResolver
{
    /**
     * empty test method - does nothing
     */
    public void resolved()
    {
    }
}

@Test(dependsOnGroups="setupsystem")
public class Start extends AbstractCamsTests
{

<suite name="02_Basistests" verbose="1" parallel="methods" thread-count="10">
        <test name="02_Basistests">
                <classes>
                        <!--  workaround - enables standalone execution of this suite -->
                        <class name="com.gide.gdcams.testspez.TS_01_92_01.DependsOnGroupResolver"/>
                        <class name="com.gide.gdcams.testspez.TS_01_92_01.Start"/>
                        <class name="...."/>

=> the "dependsOnGroups" of the testcase is satisfied by a "group mock testcase" if the suite will run standalone (executing 02_basetest.xml only)
=> the "dependsOnGroups" of the testcase is satisfied by the testcases of 01_setup.xml, if the suite will run as part of other suites (executing all *.xml )

Please again, have some thoughts of how to keep the execution order of suite.xml files - without defining "dependsOnGroups" on implementation level  !

Cheers,

Torsten

torsten....@gi-de.com Gesendet von: testng...@googlegroups.com 22.11.2010 14:41 Bitte antworten an testng...@googlegroups.com

An testng...@googlegroups.com Kopie Thema [testng-users] how to organize order of test execution, keeping separate execution ability, too?

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at

http://groups.google.com/group/testng-users?hl=en.

[Cédric Beust ♔]

Hi Torsten,

We recently discussed the possibility of being able to specify group dependencies inside your testng.xml file (take a look at the archive for today).

Would this solve your problem? 

-- 

Cédric

--
Cédric

[torsten....@gi-de.com]

Hi Cedric,

thanx for your reply now, and also thanx to Ansgar who might have similar problems...

In complex systems there´s often the need to execute a complex test plan, like Ansgar described in his thread:
first suite parallel, than a complex suite sequential, after that the next suite parallel again and so on - but all in a (user defined) order.

But it´s not only the question of how to order the suites, testclasses and test methods - it´s additionally the question of how to be able to
start or restart testing. As I experienced, the more you make use of "dependsOnMethod" or "dependsOnGroup" the more you loose the
ability to start (or restart) tests, begining on 2nd or 3rd suite - because TestNG claimes about those unresolved dependencies.

So for me, the easiest thing would be a setup where the user could define the Order of Suites and Testclasses manually,
which could be easily be done in a long (and complex) testng.xml file.

Much more complex than writing a "user controlled testng.xml" is it to write MethodInterceptors or a lot of dependsOn=".." tags in the code.

hope this and my last posts will help,

thanx

Torsten



 

Cédric Beust ♔ <ced...@beust.com> Gesendet von: testng...@googlegroups.com

26.11.2010 19:52 Bitte antworten an testng...@googlegroups.com

An testng...@googlegroups.com Kopie Thema Re: [testng-users] how to organize order of test execution, keeping separate execution ability, too?

[Kaj Kandler]

Torsten,
I would not make the test suites dependent on each other.

You can avoid that by defining some @BeforeTest or @BeforeXXX methods
that set the system into the state that you'd expect for the @Test
methods to run. So in essence replace the outcome you'd expect from
the test run before with some test setup.

That kind of setup allows you to cleanly use methods outside of the
API you want to use, such as loading a database dump, etc. in order to
get you system into the state that you need it to be in. You are also
not depending on previous tests having been successful before you can
run the rest of the tests.

K<o>

[Cédric Beust ♔]

On Mon, Nov 29, 2010 at 5:53 AM, <torsten....@gi-de.com> wrote:

But it´s not only the question of how to order the suites, testclasses and test methods - it´s additionally the question of how to be able to
start or restart testing. As I experienced, the more you make use of "dependsOnMethod" or "dependsOnGroup" the more you loose the
ability to start (or restart) tests, begining on 2nd or 3rd suite - because TestNG claimes about those unresolved dependencies.

It shouldn't. If you launch a single method that depends on other methods/groups, TestNG will automatically launch these first.

However, for this to happen, you need to tell TestNG where to find these classes (they must still be listed in your testng.xml), maybe that's what you're not doing?

--
Cédric

[torsten....@gi-de.com]

you´re right, I´m not doing that if I just want to (re)run one single suite - and than I have these unresolved dependencies...
but I need to (re)run one single suite - if there was a failure in environment setup, the application I´m going to test or whatever.

as Kai Kandler mentioned "You can avoid that by defining some @BeforeTest or @BeforeXXX methods

that set the system into the state that you'd expect for the @Test
methods to run."

that is said easy - but gets very difficult in complex (and time expensive) test scenarios.

imagine the following:
Test1 will create a client, some client users in the database
Test2 will than create some cards or whatever products, using that client
Test3 will thant modify the created cards - terminate the card, suspend the card, desuspend the card and so on.
Test4 will than create "follower" cards - based on the first created cards

It´s really hard for Test4 to implement a @BeforeTest method, that will perform the action of Test1-3 at a whole!

So for me, the easiest way is to define an execution order of all my suites - and implement a @BeforeTest method,
that picks up the state, dynamically created objectId´s or whatever from the suite run before.

But actually there´s no option to "preserver-order" the suite execution - and as written before, if I use dependencies
to force the execution order, I loose the "loosely coupled" testcases......

So far,

thanx, torsten

Cédric Beust ♔ <ced...@beust.com> Gesendet von: testng...@googlegroups.com

29.11.2010 16:42 Bitte antworten an testng...@googlegroups.com

An testng...@googlegroups.com Kopie Thema Re: Re: [testng-users] how to organize order of test execution, keeping separate execution ability, too?