[testng-users] IInvokedMethodListener returns Pass result status for failed configuration methods

[Nalin]

Hi

I have the following code in a class:

public class DerivedClass extends BaseClass
{
@Test
public void test()
{
//do something here
}

@AfterMethod
public boolean testCleanUp()
{
//some clean up code
}

@BeforeMethod
public void testSetUp()
{
AssertTrue(false);
return false;
}

}

and an IInvokedMethodListener implementation as following:

public class ResultPrinter implements IInvokedMethodListener
{
private static Logger log =
LoggerFactory.getLogger(ResultPrinter.class);

public void afterInvocation(IInvokedMethod m, ITestResult tr)
{
log.info(tr.getStatus() + " " +
m.getTestMethod().getMethodName());
}
}

public void beforeInvocation(IInvokedMethod arg0, ITestResult arg1)
{
// TODO Auto-generated method stub
}
}

On running this test, I see the following statements for the testSetup
method:

1 testSetUp

Why is the status return as PASS when the testSetUp method fails? The
test and cleanup methods are skipped as expected.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Kartik Kumar]

Are you sure this compiles? The method

@BeforeMethod
  public void testSetUp()
  {
     AssertTrue(false);
     return false;
  }

returns void and you are returning false at the end of the method.

[Nalin]

It should be:

@BeforeMethod
public boolean testSetUp()
{
AssertTrue(false);
return false;
}

or

@BeforeMethod
public void testSetUp()
{
AssertTrue(false);
}

The code compiles. Also noticed that when the test is skipped, the
status for test is 16 which maps to STARTED.


-nalin

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to testng-users...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/testng-users?hl=en.

[Nalin]

Kartik,

I think you implemented IInvokedMethodListener2. I tried using
afterInvocation() methods in that and still see the same result.

-nalin

[Kartik Kumar]

It is initialized to SUCCESS in the init method of its implementation. See http://www.google.com/codesearch/p?hl=en#8L6k9hc6tZg/src/org/testng/internal/TestResult.java&q=TestResult%20package:http://testng\.googlecode\.com&sa=N&cd=1&ct=rc&l=55

[Nalin]

Yes, it is. But when a configuration method fails, shouldn't the
corresponding ITestResult's status be set to FAIL?



On Apr 28, 5:11 pm, Kartik Kumar <krishnan.1...@gmail.com> wrote:
> It is initialized to SUCCESS in the init method of its implementation. Seehttp://www.google.com/codesearch/p?hl=en#8L6k9hc6tZg/src/org/testng/i....googlecode\.com&sa=N&cd=1&ct=rc&l=55

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

[Nalin]

Ok. So it's a bug in Invoker.invokeConfigurationMethod() function. It
doesn't catch any exception that the configuration method might throw.
This means that the invocation of IInvokedMethodListener happens
without the knowledge of whether the config method passed or failed. I
changed the for loop in this method to

for(Object targetInstance : instances) {
InvokedMethod im= new InvokedMethod(targetInstance,
tm,
params,
false, /* isTest */
isClass, /* ??? */
System.currentTimeMillis());

runInvokedMethodListeners(true /* before */, im, testResult);
m_notifier.addInvokedMethod(im);
boolean confMethodFailed = true;
try {
Reporter.setCurrentTestResult(testResult);
MethodHelper.invokeMethod(tm.getMethod(), targetInstance,
params);
//using this to avoid catching exception
confMethodFailed = false;
// Only run the method once if it's @BeforeSuite or
@AfterSuite
if (isSuite) break;
}
finally {
if (confMethodFailed) {
testResult.setStatus(ITestResult.FAILURE);
}
Reporter.setCurrentTestResult(testResult);
runInvokedMethodListeners(false /* after */, im, testResult);
}
}

and it started to work as expected.

Cedric, any thoughts?

-nalin

On Apr 28, 5:17 pm, Nalin <nalin.ma...@gmail.com> wrote:
> Yes, it is. But when a configuration method fails, shouldn't the
> corresponding ITestResult's status be set to FAIL?
>
> On Apr 28, 5:11 pm, Kartik Kumar <krishnan.1...@gmail.com> wrote:
>
>
>

> > It is initialized to SUCCESS in the init method of its implementation. Seehttp://www.google.com/codesearch/p?hl=en#8L6k9hc6tZg/src/org/testng/i....com&sa=N&cd=1&ct=rc&l=55

[Cédric Beust ♔]

Hi Nalin,

Can you send me a patch?  I'll take a look and I'll commit it.

Thanks for looking into this!

-- 

Cédric

--
Cédric

[Nalin]

Done. Thanks Cedric.

> >www.google.com/codesearch/p?hl=en#8L6k9hc6tZg/src/org/testng/i....com...

> > > > > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
> > <testng-users%252Buns...@googlegroups.com<testng-users%25252Bun...@googlegroups.com>

>
> > > > > > > > .
> > > > > > > > For more options, visit this group at
> > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the
> > Google
> > > > > Groups "testng-users" group.
> > > > > > > To post to this group, send email to
> > testng...@googlegroups.com.
> > > > > > > To unsubscribe from this group, send email to
> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>
> > > > > .

> > > > > > > For more options, visit this group athttp://
> > > > > groups.google.com/group/testng-users?hl=en.
>
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > > "testng-users" group.
> > > > > > To post to this group, send email to testng...@googlegroups.com
> > .

> > > > > > To unsubscribe from this group, send email to
> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>
> > > > > .

> Cédric
>
> --
> You received this message because you are subscribed to the Google Groups "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Ryan Fox]

Did this fix ever get incorporated?  I am running TestNG 6.8.8 and am seeing the same issue identified in this thread,

Thanks,

Ryan

> > <testng-users%...@googlegroups.com<testng-users%252Bunsubscribe@googlegroups.com>
>
> > > > > <testng-users%...@googlegroups.com<testng-users%252Bunsubscribe@googlegroups.com>
> > <testng-users%2...@googlegroups.com<testng-users%25252Bunsubscribe@googlegroups.com>

>
> > > > > > > > .
> > > > > > > > For more options, visit this group at
> > > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the
> > Google
> > > > > Groups "testng-users" group.
> > > > > > > To post to this group, send email to
> > testng...@googlegroups.com.
> > > > > > > To unsubscribe from this group, send email to
> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> > <testng-users%...@googlegroups.com<testng-users%252Bunsubscribe@googlegroups.com>

>
> > > > > .
> > > > > > > For more options, visit this group athttp://
> > > > > groups.google.com/group/testng-users?hl=en.
>
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > > "testng-users" group.
> > > > > > To post to this group, send email to testng...@googlegroups.com
> > .
> > > > > > To unsubscribe from this group, send email to
> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> > <testng-users%...@googlegroups.com<testng-users%252Bunsubscribe@googlegroups.com>

>
> > > > > .
> > > > > > For more options, visit this group athttp://
> > > > > groups.google.com/group/testng-users?hl=en.
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > > "testng-users" group.
> > > > > To post to this group, send email to testng...@googlegroups.com.
> > > > > To unsubscribe from this group, send email to
> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> > <testng-users%...@googlegroups.com<testng-users%252Bunsubscribe@googlegroups.com>

[Kevin White]

I second Ryan's question. I'm seeing the same behavior.

A failure in @BeforeMethod results in a test being marked as "SKIPPED" (which I personally don't like but understand, and deal with in my custom test listener by reporting it to my TCM tool as a failure).

A failure in @AfterMethod results in a test being skipped as well, which is not true because the test obviously ran, and then the teardown failed. (I'm encountering this because of in-house soft assert logic that we're using, which I realize should probably be moved into test methods.)

This email communication (including any attachments) contains information from Answers Corporation or its affiliates that is confidential and may be privileged. The information contained herein is intended only for the use of the addressee(s) named above. If you are not the intended recipient (or the agent responsible to deliver it to the intended recipient), you are hereby notified that any dissemination, distribution, use, or copying of this communication is strictly prohibited. If you have received this email in error, please immediately reply to sender, delete the message and destroy all copies of it. If you have questions, please email le...@answers.com. 

If you wish to unsubscribe to commercial emails from Answers and its affiliates, please go to the Answers Subscription Center http://campaigns.answers.com/subscriptions to opt out.  Thank you.