Execution order of <suite> child elements?

[Ansgar Konermann]

Hi Cedric, hi all,

is there an easy way to specify the execution order of child elements of
an XML <suite> element?

There is a "preserve-order" attribute for the <test> tag, can I have a
similar feature for <suite> child elements?

Our use case is as follows: we would like to run all our fast unit tests
at the beginning of the overall test suite in parallel. Then, before
continuing with the more integrational tests, we want to run one
specific single test class, and after that, all our integration tests
(in parallel, again).

Does TestNG already support this use case in tandem with the maven
surefire plugin?

Thanks in advance + best regards

Ansgar

[Cédric Beust ♔]

Hi Ansgar,

You are right this is currently not supported but I agree it would make sense to extend this concept to suites as well.

I would have to use a different name to be consistent with TestNG's inheritance rules, though. More specifically:

<suite preserve-order="true">

is the same as using <test preserve-order="true"> on all your test tags

<suite suite-preserve-order="true">

means that the <test> tags will be run in the order they are found in your testng.xml file

Would this solve your problem?

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Ansgar Konermann]

On 26.11.2010 16:20, Cédric Beust ♔ wrote:

<suite suite-preserve-order="true">

means that the <test> tags will be run in the order they are found in your testng.xml file

Would this solve your problem?

Not fully I fear. Our top-level suite currently looks like this:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="toplevel suite" parallel="false">

  <suite-files>
    <suite-file path="testng-allFastTests.xml" />
  </suite-files>

  <test name="Spring Bean Instantiation Test">
    <classes>
      <class name="com.example.product.spring.SpringBeanInstantiationTest" />
    </classes>
  </test>

  <suite-files>
    <suite-file path="testng-allSlowTests.xml" />
  </suite-files>

  <suite-files>
    <suite-file path="testng-allDatabaseTests.xml" />
  </suite-files>

  <test name="Ungrouped Tests (forbidden!)">
    <groups>
      <run>
        <exclude name="fastTest" />
        <exclude name="slowTest" />
        <exclude name="databaseTests" />
        <exclude name="seleniumTest" />
      </run>
    </groups>
    <packages>
      <package name="com.example.product..*" />
    </packages>
  </test>
</suite>

What we would like to achieve is that the order of *all* the child elements of the suite gets preserved, not only the <test> elements.

The resulting execution sequence should work like this:
1. Execute all fast tests (inside this suite file, we specify parallel="tests", so run them in parallel)
2. SpringBeanInstantiationTest
3. all slow tests (in parallel, defined in suite file as above)
4. all remaining tests (in parallel, defined in suite file as above)

OTOH, our "suite hierarchy" is quite simple, so we would probably be able to refactor it to only one suite with multiple <tests>, i. e. your initial suggestion would suffice.

Regarding hierarchical suites: does TestNG actually model this as a suite hierarchy internally (each with its own parallel=... setting), or will all suites be merged into one (with merged attributes)?

Kind regards

Ansgar

[Cédric Beust ♔]

Hi Ansgar,

Yes, TestNG models the suite inclusions as a hierarchy internally.

I'm afraid that what you are asking is basically a brand new way of specifying an entire ordering. Not only is this error prone in XML but I think that groups are a much better way of doing that since you can mix methods that have different speeds and semantics in the same class/test/package. Right now, it looks like whenever a developer adds a new method they need to go add it at a very specific place in the XML file, which doesn't seem optimal to me...

The "TestNG way" is to put all your classes and packages in testng.xml in no particular order and then to ask TestNG to run the "fast" tests, then the "slow" tests, etc...

Does this make sense?

-- 

Cédric

[Ansgar Konermann]

On 26.11.2010 17:38, Cédric Beust ♔ wrote:
> Hi Ansgar,
>
> Yes, TestNG models the suite inclusions as a hierarchy internally.
>
> I'm afraid that what you are asking is basically a brand new way of
> specifying an entire ordering. Not only is this error prone in XML but
> I think that groups are a much better way of doing that since you can
> mix methods that have different speeds and semantics in the same
> class/test/package. Right now, it looks like whenever a developer adds
> a new method they need to go add it at a very specific place in the
> XML file, which doesn't seem optimal to me...
>
> The "TestNG way" is to put all your classes and packages in testng.xml
> in no particular order and then to ask TestNG to run the "fast" tests,
> then the "slow" tests, etc...
>
> Does this make sense?

Hi Cédric,

yes, does make sense. We don't really need the sub-suites.

We could probably even do without the possibility to order <test>s in
the <suite>, but then need a possibility to define dependencies between
test groups in the <suite> xml.

With group dependencies, our approach could look as follows:

- put all tests into their respective groups
- reserve an extra group for the spring beans instantiation test
- specify group dependencies (run this group after groups x, y, z have
completed) in the <suite>

Is it possible to specify group dependencies in the testng.xml? I'm only
aware of the @Test(dependsOnGroups) annotation-style dependency
specification. Searching the testng DTD, i did not find any attribute
named like .*depend.* -- and we certainly don't want to add
dependsOnGroups attributes to each new test method, but define this
dependency in one place (DRY).

Best regards

Ansgar

[Cédric Beust ♔]

Hi Angar,

On Fri, Nov 26, 2010 at 9:14 AM, Ansgar Konermann <ansgar.k...@googlemail.com> wrote:

Hi Cédric,

yes, does make sense. We don't really need the sub-suites.

We could probably even do without the possibility to order <test>s in the <suite>, but then need a possibility to define dependencies between test groups in the <suite> xml.

With group dependencies, our approach could look as follows:

- put all tests into their respective groups
- reserve an extra group for the spring beans instantiation test
- specify group dependencies (run this group after groups x, y, z have completed) in the <suite>

Is it possible to specify group dependencies in the testng.xml? I'm only aware of the @Test(dependsOnGroups) annotation-style dependency specification. Searching the testng DTD, i did not find any attribute named like .*depend.* -- and we certainly don't want to add dependsOnGroups attributes to each new test method, but define this dependency in one place (DRY).

This is currently not supported but I agree it's more in line with TestNG's philosophy, and I'm certainly sensitive to the DRY argument.

I like the idea of being able to specify group dependencies in your testng.xml, I'll think about that a bit more.

Anyone else have any thoughts on this feature?

--
Cédric

[Ansgar Konermann]

On 26.11.2010 17:38, Cédric Beust ♔ wrote:

> Hi Ansgar,
>
> Yes, TestNG models the suite inclusions as a hierarchy internally.
>
> I'm afraid that what you are asking is basically a brand new way of
> specifying an entire ordering. Not only is this error prone in XML but
> I think that groups are a much better way of doing that since you can
> mix methods that have different speeds and semantics in the same
> class/test/package. Right now, it looks like whenever a developer adds
> a new method they need to go add it at a very specific place in the
> XML file, which doesn't seem optimal to me...

Hi Cedric,

thinking about your comment one more time, I don't quite see where the
*total* ordering is imposed.

So I took my favourite low-tech tools and drew a little sketch to
illustrate what we would like to order, and which parts should still be
subject to TestNGs "unspecified execution order":

http://passion.forco.de/content/ordering-sub-elements-testng-suite

The "sub-suites" are relatively thin, all they contain is a single
<test> element which collects all tests having a certain group from all
java packages beneath a certain root package. Each of these could easily
be inlined into one <test> tag inside the top-level suite.

I just want to make sure we're talking about the same things; besides
that, the solution we already discussed last friday (using testng
groups) would suffice for our use case.

Best regards and thanks in advance.

Ansgar

[djan...@gmail.com]

Hello, it is now June 2014 .   Was any of this ever implemented?   I need to run a suite that looks like this, but it is critical that the critical tests run before any of the other tests.   If the critical tests fail, then there is no reason to run any of the other tests.    Does TestNG have group dependencies yet?   Suite-files preserve order yet?

    <test name="Critical Tests">
        <classes>
            <class name="qa.tt.tests.X00_PropertiesTest" />
            <class name="qa.tt.tests.X01_CriticalTest" />  <!-- 1 test passes or it fails the whole suite -->
        </classes>
    </test>
    <suite-files>
        <suite-file path="./ValidationTests.xml" />  <!-- 5 tests verifies back end systems -->
        <suite-file path="./ClientSmokeTests.xml" />  <!-- 5 tests of customized configurations -->
        <suite-file path="./GenericRegression.xml" />  <!-- 100 generic tests covering all app features -->
    </suite-files>   
</suite>

[djan...@gmail.com]

Turns out that it isn't a problem.  I just refactored my suite file to not use "suite-files" any longer and instead I now use one suite file with mulitple "test" tag sections in it.  I force order by the preserve order arg on the suite tag.

[stelios82]

And how do you ensure that if one critical test fails, the whole rest of tests will fail?