How to use dataProvider and only re-run failed test instances?

[sazz]

I am using dataProvider to pass parameters to a test method. Say
dataProvider method passes 10 data sets and it fails 3 times. And now
I just want to run the test method only 3 times with the failed data,
am struggling a bit to make sure only that happens and does not run 10
times again. Please guide me in this.

In one single test method I have both passed and failed test cases
present at a time hence when ever I invoke this perticular method I
have with me both the passed and failed report generated at the same
time. But the issue is, when ever I specifically want to run the
Failed test cases from the testng-failed.xml I come across again with
all the cases(passed as well as failed cases), which I do not want. I
only want to get the failed cases here only not the passed one even
though it is in the same method.

For eg ::

public class BaseTest{
String id;
BaseTest(String id) {
this.id = id;
}

@Test(dataProvider="CheckTest")
public void testGenX(String testMe) {
System.out.println("id=" + id +" testMe="+ testMe);
if("A".equals(id))
Assert.assertEquals("james", testMe); //ListenerTest fails for
james
else {
Assert.assertEquals("andrew", testMe); //AnotherTest fails for
andrew
}
}

@DataProvider(name="CheckTest")
public Object[][] permanentGiver() {

return new Object[][] {
{"andrew"}, {"james"},
{"andrew"}, {"james"}
};
}
}


Result is::

===============================================
Default test
Tests run: 4, Failures: 2, Skips: 0
===============================================


now when I am running the testng-failure.xml for the fail result. I
again come across the same result(both the fail as well as the pass).
But I want only the fail result to execute. Kindly give a solution.

[Cédric Beust ♔]

On Mon, Apr 25, 2011 at 1:57 AM, sazz <suj...@gmail.com> wrote:

I am using dataProvider to pass parameters to a test method. Say
dataProvider method passes 10 data sets and it fails 3 times. And now
I just want to run the test method only 3 times with the failed data,
am struggling a bit to make sure only that happens and does not run 10
times again. Please guide me in this.

TestNG already does this automatically for you. Try it and take a look at the testng-failed.xml file that it generates, you will understand how this works.

--
Cédric

[Raghuram Devarakonda]

2011/4/25 Cédric Beust ♔ <ced...@beust.com>

Very interesting. Out of curiosity, I wrote the following test method and it failed as expected for values 1 and 3.

----------
    @DataProvider
    public Object[][] dpEven() {
        return new Object[][] { {1}, {2}, {3}};
    }
   
    @Test(dataProvider="dpEven")
    public void testEven(int num) {
        Assert.assertEquals(num%2, 0);
    }
----------

Running with testng-failed.xml then ran the test for only failed parameters 1 and 3. So TestNG is actually saving all the parameters passed to the test method in testng-failed.xml (using "invocation-numbers" in this case). Is there a limitation on the types of parameters that can be supported this way?

Thanks,
Raghu

[Cédric Beust ♔]

Hi Raghuram,

On Mon, Apr 25, 2011 at 9:55 AM, Raghuram Devarakonda <drag...@gmail.com> wrote:

Very interesting. Out of curiosity, I wrote the following test method and it failed as expected for values 1 and 3.

----------
    @DataProvider
    public Object[][] dpEven() {
        return new Object[][] { {1}, {2}, {3}};
    }
   
    @Test(dataProvider="dpEven")
    public void testEven(int num) {
        Assert.assertEquals(num%2, 0);
    }
----------

Running with testng-failed.xml then ran the test for only failed parameters 1 and 3. So TestNG is actually saving all the parameters passed to the test method in testng-failed.xml (using "invocation-numbers" in this case). Is there a limitation on the types of parameters that can be supported this way?

There is no limitation on this for a simple reason: testng-failed.xml only records the indices that failed, not the values. When you rerun your test, TestNG will call your data provider as usual but it will only invoke your test method with the indices in the XML file, and it will discard the others.

I can see why you're asking this question, though: in your example, the values you return happen to be integers that match the indices :-)

--
Cédric

[Raghuram Devarakonda]

2011/4/25 Cédric Beust ♔ <ced...@beust.com>

I can see why you're asking this question, though: in your example, the values you return happen to be integers that match the indices :-)

There is no limitation on this for a simple reason: testng-failed.xml only records the indices that failed, not the values. When you rerun your test, TestNG will call your data provider as usual but it will only invoke your test method with the indices in the XML file, and it will discard the others.--

Hi Cédric,

Nice solution to what I thought was a tricky problem. The user needs to be careful not to tinker with the DataProvider code in between the runs, though. Yes, my example test case did contribute to my question :-). 

My interest was piqued because coincidentally, a similar question was asked on nose forums today and the answer there was that it is not possible to run tests with only the parameters that failed. To be fair, DataProvider tests in TestNG have some differences with python generator based tests in nose.
    http://groups.google.com/group/nose-users/browse_thread/thread/7a27f571e9c56e36

Thanks,
Raghu

[Cédric Beust ♔]

Hi Raguram,

2011/4/25 Raghuram Devarakonda <drag...@gmail.com>

Hi Cédric,

Nice solution to what I thought was a tricky problem. The user needs to be careful not to tinker with the DataProvider code in between the runs, though.

Indeed, but if you break this assumption by modifying your tests in-between runs, then you should not be surprised if you get different results. And this is true whether you run testng.xml or testng-failed.xml.

 

Yes, my example test case did contribute to my question :-). 

My interest was piqued because coincidentally, a similar question was asked on nose forums today and the answer there was that it is not possible to run tests with only the parameters that failed. To be fair, DataProvider tests in TestNG have some differences with python generator based tests in nose.
    http://groups.google.com/group/nose-users/browse_thread/thread/7a27f571e9c56e36

I'm not familiar with Nose but I don't see any reason why it couldn't use the same trick I used with TestNG: just remember the number of the invocation that failed and discard the others when rerunning your tests...

--
Cédric

[Raghuram Devarakonda]

2011/4/25 Cédric Beust ♔ <ced...@beust.com>

I'm not familiar with Nose but I don't see any reason why it couldn't use the same trick I used with TestNG: just remember the number of the invocation that failed and discard the others when rerunning your tests...

Indeed. I have suggested the same on nose list.

[Frank Escobar]

Fixed in version 6.10

https://mvnrepository.com/artifact/org.testng/testng/6.10

https://github.com/cbeust/testng/blob/master/CHANGES.txt#L95

https://github.com/cbeust/testng/issues/1250