Failure in @AfterMethod and @BeforeMethod

[Ed]

Ik am a bit confused,

What I want: even when a failure occurs in the method marked with
@AfterMethod(alwaysRun = true), he will continue running the tests
that follow.

However, I understand from a forum post in 2007 that this will not
happen by design.
But can't I still run them by setting alwaysRun = true on the test ? I
tried that but it will not work...
I tried something like @Test(alwaysRun = true).

What happens now when an exception occurs in the @AfterMethod method,
it will run all the @AfterMethod methods of the test that follow, but
not their @BeforeMethod of all the tests that follow :(...

I don't really understand that. Why the the after methods but not the
before methods ?
Why does it even bother running any of the test (their after methods)
that follow considering the above design decision?
What is the expected behavior when an exception occurs in the after
method for the tests that follow?

Ed

[Ed]

Can somebody please give me some feedback on this?
I am a bit stuck with it :(

[Ed]

I am currious if this isn't a bug. I mean the after method's are
called but not the before methods... A bit strange not ?

[Cédric Beust ♔]

Hi Ed,

The behavior is a bit subtle.

The documentation says:

   * For after methods (afterSuite, afterClass, ...): 

   *  If set to true, this configuration method will be run

   *  even if one or more test methods invoked previously failed or

   *  was skipped.

This attribute only applies to failures happening in *test* methods.

If a failure happens in a configuration method, TestNG doesn't recover because the results of subsequent tests can no longer be trusted.

If you really want the execution to continue after a failure in a configuration method, you should catch that exception so that TestNG never sees it.

Does this make sense?

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Ed Bras]

Hi Cedirc,
Thanks for the feedback.
It makes sense.
However, I don't understand the  "not recover" behavior of testNG when an exception isn't catched in an after method.
I my case, all after methods of all following tests are run... but not any corresponding before method (observed by setting breakpoints).
Isn't that a bit strange? Shouldn't testNG not simple stop completely after catching an exception in a before/after method?

> If you really want the execution to continue after a failure in a configuration method, you should catch that exception so that TestNG never sees it.

But how should I report this?
I mean: testNG will report that the test is OK, so how can I indicate to testNG that the test failed ?

What I do:
I run several (selenium) tests against a browser. After every test I perform the same check to see if an error occurred, as such that I include it in an after method.
So basically I like to mark a test as failed if a check in the after method fails.
The alternative would be that I include a check method call at the end of every test method (about a few hunderd) which is easy forgotten by a tester (especially if they switch work). The checks are important and always the same as such perfect for in an after method.
I might be able to do this through a listener but they are run the easy way by the eclipse plugin without setting any configuration, which I would like to keep this way.
Maybe an idea: set a kind of "recover option" (fail through)  through annotation to indicate how an exception should be treated by the configuration methods..

Maybe you have some creative idea's ;)
Ed
 

2010/3/22 Cédric Beust ♔ <cbe...@google.com>

[Cédric Beust ♔]

I think your main problem is that you are writing a test in your @AfterMethod.  That's going against the design of TestNG:  configuration methods are meant to configure or tear down your tests, not perform actual tests.

You should move this test logic in a real test method and have it depend on all the other test methods that need to run first.

-- 

Cédric

[Ed]

Thanks for the idea, but the "depends" attribute (either a method or
group) is used to run method(s) -before- a test method.
I want a run a certain method -after- each test method (in a group) .
So I don't want to run it after "all" my hundreds of test are finished
(which I could do with a dependsOnGroup), but I need to run this -
after- method after "each" test method.
How can I do this using the testng annotations?
Sorry, can't find it :(

Ed

On Mar 22, 5:49 pm, Cédric Beust ♔ <cbe...@google.com> wrote:
> I think your main problem is that you are writing a test in your
> @AfterMethod.  That's going against the design of TestNG:  configuration
> methods are meant to configure or tear down your tests, not perform actual
> tests.
>
> You should move this test logic in a real test method and have it depend on
> all the other test methods that need to run first.
>
> --

> *Cédric*

> >> *Cédric*

> >>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/testng-users?hl=en.
>
> >>  --
> >> You received this message because you are subscribed to the Google Groups
> >> "testng-users" group.
> >> To post to this group, send email to testng...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/testng-users?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

[Kartik Kumar]

On Mon, Mar 22, 2010 at 4:17 PM, Ed <post2...@gmail.com> wrote:

Thanks for the idea, but the "depends" attribute (either a method or
group) is used to run method(s) -before- a test method.

I want a run a certain method -after- each test method (in a group) .
So I don't want to run it after "all" my hundreds of test are finished
(which I could do with a dependsOnGroup), but I need to run this -
after- method after "each" test method.

How can I do this using the testng annotations?

Try BeforeGroups and AfterGroups

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

On Mon, Mar 22, 2010 at 6:08 PM, Kartik Kumar <krishn...@gmail.com> wrote:

On Mon, Mar 22, 2010 at 4:17 PM, Ed <post2...@gmail.com> wrote:

Thanks for the idea, but the "depends" attribute (either a method or
group) is used to run method(s) -before- a test method.

I want a run a certain method -after- each test method (in a group) .
So I don't want to run it after "all" my hundreds of test are finished
(which I could do with a dependsOnGroup), but I need to run this -
after- method after "each" test method.

How can I do this using the testng annotations?

Try BeforeGroups and AfterGroups

Same problem:  you shouldn't be writing tests in configuration methods.

I think I have a solution, I'll write it up and I'll post it here.

--

Cédric

[Ed]

Thanks guys.
Ok Cedric, I am looking forward to it.

On Mar 23, 6:55 am, Cédric Beust ♔ <cbe...@google.com> wrote:
> On Mon, Mar 22, 2010 at 6:08 PM, Kartik Kumar <krishnan.1...@gmail.com>wrote:

>
>
>
> > On Mon, Mar 22, 2010 at 4:17 PM, Ed <post2edb...@gmail.com> wrote:
>
> >> Thanks for the idea, but the "depends" attribute (either a method or
> >> group) is used to run method(s) -before- a test method.
> >> I want a run a certain method -after- each test method (in a group) .
> >> So I don't want to run it after "all" my hundreds of test are finished
> >> (which I could do with a dependsOnGroup), but I need to run this -
> >> after- method after "each" test method.
> >> How can I do this using the testng annotations?
>
> > Try BeforeGroups and AfterGroups
>
> Same problem:  you shouldn't be writing tests in configuration methods.
>
> I think I have a solution, I'll write it up and I'll post it here.
>
> --

> *Cédric*

[Cédric Beust ♔]

Hi Ed,

On Tue, Mar 23, 2010 at 1:34 AM, Ed <post2...@gmail.com> wrote:

Thanks guys.
Ok Cedric, I am looking forward to it.

Here it is:   http://beust.com/weblog/2010/03/23/better-mock-testing-with-testng/

Let me know what you think.

--

Cédric

[Ed]

Thanks, I will reply on the blog post directly.

One more thing: Like mentioned above, when an exception occurs in the
after method, all following tests are run with exceptions (all
following after methods are run and not all before methods)...
Like I understand from you, testNG doesn't recover from an exception
in an after method. So I think that testNG should stop testing more
gracefully instead of the running all tests half, something I observe
now. This is a bit strange especially if you have a few hundreds tests
which are run half.
Or not ?
The alternative would be that I check if an exception occurs in the
after method and dont' do anything, because otherwise a after method
of the following test is called of which the pre-conditions aren't met
because the before method isn't called... If testNG simple stops in
case an exception occurs in an configuration method, these problems
will not occur.

On Mar 23, 7:21 pm, Cédric Beust ♔ <cbe...@google.com> wrote:
> Hi Ed,
>

[Ed]

Hi Cedric,

Can you please let me know if I should report the remark about
"graceful stopping testing" as a bug/enchancement? (listed below)