Re: Retry Listener works once for @Factory test method instance

[Pranav Verma]

Sorry for my earlier incomplete mail, Here I go again :

List,
This is the situation :

1) I have a testclass with one test method
2) I create 6 instances of that test class using @Factory ( named
Test1, Test2, Test3 ... Test6 )
3) Odd number Test always fail ( Test 1 , Test3 and Test5 will fail, but Test2
, Test4 and Test6 will pass )
4) The test class has a retry listener implemented, with retry count as 2

Now after the test run I expect to see this

Failed tests:
Test1(com.test.RetryTest): expected:<true> but was:<false>
Test1(com.test.RetryTest): expected:<true> but was:<false>
Test3(com.test.RetryTest): expected:<true> but was:<false>
Test3(com.test.RetryTest): expected:<true> but was:<false>
Test5(com.test.RetryTest): expected:<true> but was:<false>
Test5(com.test.RetryTest): expected:<true> but was:<false>

Tests run: 9, Failures: 6, Errors: 0, Skipped: 0

but I got this

Failed tests:
Test1(com.test.RetryTest): expected:<true> but was:<false>
Test1(com.test.RetryTest): expected:<true> but was:<false>
Test3(com.test.RetryTest): expected:<true> but was:<false>
Test5(com.test.RetryTest): expected:<true> but was:<false>

Tests run: 7, Failures: 4, Errors: 0, Skipped: 0

As you can see only Test1 was retried , rest all tests just failed
after first failure. My guess is, its happening because while retrying
TestNG consider all instance of one method ( created by @factory) as
one method, hence once the retry quota is over for first method, it
does not bother to run it for other instances of same method. Seems
like a bug , what do you guys say ?

I am attaching the sample code with this email, and I am TestNG 6.3.1.

best,
Pranav

On Wed, Dec 7, 2011 at 1:48 AM, Pranav Verma <pranav...@gmail.com> wrote:
> List,
> This is the situation :
>
> 1) I have a testclass with one test method
> 2) I create 4 instances of that test class using @Factory ( named
> Test1, Test2, test3 and Test4 )
> 3) Odd number Test always fail ( Test 1 and Test3 will fail, but Test2
> and Test4 will pass )
> 4) The test class has a retry listener implemented, with retry count as 2
>
> Now after the test run I expect to see this
>
>
> Failed tests:
>  Test1(com.test.RetryTest): expected:<true> but was:<false>
>  Test1(com.test.RetryTest): expected:<true> but was:<false>
>  Test3(com.test.RetryTest): expected:<true> but was:<false>
> Test3(com.test.RetryTest): expected:<true> but was:<false>
>
> Tests run: 5, Failures: 3, Errors: 0, Skipped: 0
>
>
>
> Test Run : 6, Failed :3, Skipped  15 ( coz 3 tests will fail, and as
> they should get retried 5 times each, hence 3*5=15)
>
> bit I got this

[pranav]

Cédric , can you please have a look ...and let me know if I am doing
something wrong or its a bug ?

TIA,
Pranav

[Cédric Beust ♔]

Hi Pranav,

I'll try as soon as I get a chance, but I'm pretty busy right now...

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[pranav]

Hey Cédric,


Happy New Year :), did you get a chance to have a look at this issue ?

Best,
Pranav

[Cédric Beust ♔]

Hi Pranav,

Sorry, not yet...

-- 

Cédric

Pranav

[Krishnan Mahadevan]

Pranav,

I dont find any issues with IRetryAnalyzer is TestNG and it works fine.

Here's a full fledged working sample that substantiates my claim.

public class RetryAnalyzerDemo implements ITest{

private String instanceName;

private int initialValue=1;

public RetryAnalyzerDemo(String instanceName, int i){

this.instanceName = instanceName;

this.initialValue = i;

}

@Override

public String getTestName() {

return "Test : " + this.instanceName ;

}

@Factory

public static Object[] getInstance(){

Object[] object = new Object[2];

object[0] = new RetryAnalyzerDemo("Java",1);

object[1] = new RetryAnalyzerDemo("Oracle", 3);

return object;

}

@Test(retryAnalyzer=MyGlobalListener.class)

public void testMethod(){

Assert.assertTrue(((this.initialValue++)%5)==0);

}

}

public class MyGlobalListener implements IRetryAnalyzer, ITestNGListener{

public ThreadLocal<Integer> retryCount = new ThreadLocal<Integer>();

public MyGlobalListener(){

System.out.println("RetryListener instantiated");

retryCount.set(new Integer(0));

}

@Override

public boolean retry(ITestResult result) {

int currentValue = 0;

if (retryCount.get() != null){

                      currentValue = retryCount.get().intValue();

}

System.out.println("Current Value = " + currentValue);

if (currentValue >= 5){

return false;

}

retryCount.set(new Integer(currentValue++));

System.out.println("Another iteration pending");

return true;

}

}

Output 

===============================================

    Default test

    Tests run: 8, Failures: 6, Skips: 0

===============================================

===============================================

BlueFin Test

Total tests run: 8, Failures: 6, Skips: 0

===============================================

As you can see, 

There were two instances that I created

The "Java" instance is supposed to fail 4 times, before it passes

The "Oracle" instance is supposed to fail 2 times, before it passes

which explains the result of 6 failures of which 2 passed.

I hard-coded the retry count to 5 times, but you can get creative and change that as well.

Thanks & Regards

Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

2012/1/10 Cédric Beust ♔ <ced...@beust.com>

[pranav]

Krishnan,

Thanks for trying to look into this issue, but your example is
different from my problem. Your example does not have a max limit for
retry , it will run tests as long as it does not pass, which I agree
works just fine. My problem is with fixing a max number of retries,
and reporting them properly .... please try the source zip attached in
my original post, and you will understand the problem better. Thanks
again.

Best,
Pranav

[shankar KC]

I think you are missing sth here.

@Override

public boolean retry(ITestResult result) {

int currentValue = 0;

if (retryCount.get() != null){

                      currentValue = retryCount.get().intValue();

}

System.out.println("Current Value = " + currentValue);

if (currentValue >= 5){

return false;

}

retryCount.set(new Integer(currentValue++));

System.out.println("Another iteration pending");

return true;

}

THis example has max limit.