Running only failed tests with DataProvider and XML fed parameters

[edwolb]

We use XML parameters to feed runtime parameters to our tests, and then use DataProviders to further iterate through different test scenarios.  We've setup our framework such that a CSV file can be passed in through the XML file and used to run the test against specific data.  The xml looks like this:

<suite name="Test Suite" verbose="1" parallel="tests" thread-count="10">

  <test name="Test" >

    <classes>

  <parameter name="csvFile" value="path/to/file.csv" />

           <class name="com.example.automation.Test.java" />

    </classes>

  </test>

</suite>

This works well for us, as it means we can use the same test against different sets of data, for different databases or one file for smoke testing and one for full regression.  We have a generic DataProvider that we reuse that always looks for the "csvFile" parameter and creates a set of data to pass to the test using it.

The problem is that when tests fail for specific rows of data, and we try to replay only failed tests, the csvFile parameter goes missing.  This results in a total failure, and the only way we can replay certain scenarios is by doing the entire test, which kills us for heavy data driven tests.  How can we ensure that parameters are passed into the failed test XML files?

Thanks

[Cédric Beust ♔]

Hi Ed,

Probably an oversight in how I generate testng-failed.xml (I don't believe I look up the <parameter> tag in the original testng.xml file).

I'll look into it.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dusvJ6H6hhcJ.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[edwolb]

Thanks Cédric, let me know if I can provide any more input.

-- 

Cédric

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Cédric Beust ♔]

Hi Ed,

I'm seeing these parameters being carried over to testng-failed.xml, although not exactly where they should be. Still, they are here and you should see them:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Failed suite [Suite]">

  <parameter name="n" value="42"/>

  <parameter name="o" value="43"/>

  <test name="Test(failed)" preserve-order="true">

    <parameter name="n" value="42"/>

    <parameter name="o" value="43"/>

    <classes>

      <class name="test.reports.FailedSampleTest">

        <methods>

          <include name="failed"/>

        </methods>

      </class> <!-- test.reports.FailedSampleTest -->

    </classes>

  </test> <!-- Test(failed) -->

</suite> <!-- Failed suite [Suite] -->

Can you post a small testng.xml and then the testng-failed.xml you are getting?

Thanks.

-- 

Cédric

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

Hi Ed,

I just pushed a fix in the beta that should fix this, can you try it and report back?

http://testng.org/beta

Run with verbose="2" and make sure the version is at least 6.6beta 20120616_1332

Thanks.

-- 

Cédric

[Chris]

Hi Cédric,

It looks like the test successfully ran again, however instead of only rerunning the one failed instance provided by the data provider, it ran all instances again.  So you've gotten the 1 test out of the suite to run, but we're still running all scenarios, even those that previously passed.  Is this intended?

Also, from what I can tell, the parameters are now being included twice, once at the suite level, and once at the test level, even though the original XML file only included the parameters at the class level.  I know the DTD may only recommend that parameters be included at the test level, and not the class level.  Could this be causing the issue?  Including it at the class level seems to be working otherwise.

Here's the failed xml file:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Failed suite [suite]">

  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv"/>

  <parameter name="browser" value="chrome"/>

  <test name="TC56_verifyOrder(failed)" preserve-order="true">

    <parameter name="csvFile" value=" com/automation/tests/orders/TC56.csv "/>

    <parameter name="browser" value="chrome"/>

    <classes>

      <class name="com.automation.tests.orders.TC56">

        <methods>

          <include name="setupNavigation"/>

          <include name="closeDrivers"/>

          <include name="login"/>

          <include name="navigateToOrders"/>

          <include name="cleanup"/>

          <include name="DCorder" invocation-numbers="0"/>

          <include name="setLoginProfile"/>

          <include name="setupDriver"/>

        </methods>

      </class> <!-- com.automation.tests.orders.TC56 -->

    </classes>

  </test> <!-- TC56_verifyDiscontinueAndStikeOutActiveOrder(failed) -->

</suite> <!-- Failed suite [suite] -->

and the original:

<suite name="suite">

  <test name="TC56">

    <classes>

  <parameter name="csvFile" value="com/automation/tests/orders/TC56.csv" />

  <parameter name="browser" value="chrome" />

       <class name="com.automation.tests.orders.TC56" />

    </classes>

  </test>

</suite>

Thanks

--
Chris

[Frank Escobar]

Fixed in version 6.10

https://mvnrepository.com/artifact/org.testng/testng/6.10

https://github.com/cbeust/testng/blob/master/CHANGES.txt#L95

https://github.com/cbeust/testng/issues/1250

--
Chris

-- 

Cédric

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/dU-C5Ff8XugJ.

To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.