Common test superclass, @BeforeClass fails for one test, and all subclasses skipped

[Wujek Srujek]

Hi Cedril,

I have a test hierarchy:

public class AbstractTest {

    @BeforeClass
    public void beforeAbstractTestClass() {
        System.out.println("<<<<<>>>>> AbstractTest.beforeAbstractTestClass() for class " + getClass().getName());
        throw new RuntimeException("just testing stuff");
    }
}

public class TestOne extends AbstractTest {

    @Test
    public void test() {
        System.out.println("<<<<<>>>>> TestOne.test()");
    }
}

public class TestTwo extends AbstractTest {

    @Test
    public void test() {
        System.out.println("<<<<<>>>>> TestTwo.test()");
    }
}

So TestOne and TestTwo extend a common superclass that defines a @BeforeClass method. When I invoke the tests, I get this output (excerpt from eclipse, latest plugin, the same on the command line):
<<<<<>>>>> AbstractTest.beforeAbstractTestClass() for class com.test.TestOne
FAILED CONFIGURATION: @BeforeClass beforeAbstractTestClass
java.lang.RuntimeException: just testing stuff
    at com.test.AbstractTest.beforeAbstractTestClass(AbstractTest.java:13)
... Removed 22 stack frames
SKIPPED CONFIGURATION: @BeforeClass beforeAbstractTestClass
SKIPPED: test
SKIPPED: test

As you can see, the @BeforeClass is called only once, and it fails, and the other test is ignored not called. So, the result of @BeforeClass of one test subclass impacts whether the other subclass is called.

This might be ok for this simple case, as the method fails always, so it is actually smart to not bother calling it again. Hovewer, you can never be sure if this invariant is true. Consider a possibility that a the @BeforeClass method actually does some stuff and it's result varies depends on some variables:

if (getClass().getName().contains("TestOne")) throw new RuntimeException();
(Please suppose for the purpose of this example that TestOne always precedes TestTwo.)

Sure, it is a stupid example, but what I mean by this is: TestOne is invoked first, and the @BeforeClass method throws an exception, so its tests are skipped - all good. Then, the method would be called for TestTwo, and it would finish normally, so its test should be executed. This doesn't happen.

To make the example more concrete: in the common superclass I am reading annotations on the test class / its members and if it is fine, the tests are executed; if there is some problem, it is a configuration problem and the method throws an exception are skipped. I noticed that some tests were skipped entirely, not even their @BeforeClass was called, as described, and hence I started investigating this, and produced this simplest case.

Is this supposed to work this way? Maybe I simply need to change my @BeforeClass and throw some other kind of exception? I tried TestNGException and SkipException, with the difference that the latter causes the first invocation of the method be classified "skipped" instead of "failed".

Please tell me if I am doing things wrong here.

Best regards,
Wujek

[Cédric Beust ♔]

Hi Wujek,

This looks like quite a contrived example, and it seems to me that if you have such variability in your tests, you should be able to find a way to isolate this part in a lower (in the hierarchy) @BeforeClass so that it won't impact other classes if it fails...

But maybe I didn't understand your example completely, which wouldn't surprise me :-)

--
Cedric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Wujek Srujek]

Hi,

it actually is not that contrived. But I am surprised that TestNG treats such TestOne and TestTwo kind of as they were the same, which is quite different to what Java treats them - they only share a superclass and a common method for them, otherwise they are not related. TestNG has one test influence the other because they share a supeclass, whereas in Java the only thing in common they have is some algorithm (the @BeforeClass). Just think of the getClass() method as the template method pattern. Why would making Tea influence making Coffee (when modeled in a similar hierarchy of common superclass Drink with method prepare() that uses a template method putStuff(), with stuff being coffee for the Coffee subclass, and tea herbs for the Tea subclass.

I find it unnatural behaviour. Well, that's just the way I see it. In the end, if we have an error in one test, the others will be reported skipped whereas they should not be (in my opinion), but as far as the build is concerned, it wouldn't succeed anyways. And it boosts morale when you fix one bug and suddenly your thousend tests all start to work ;d

But maybe I didn't understand your example completely, which wouldn't surprise me :-)

Yes, other people say that too when I try to explain them things, it must be one of my shortcoming or even maybe something genetic - I always had trouble understanding my father when he tried to explain me maths ;d Ah well...

Please just don't treat anything I write as criticism. I was just making sure I understand they what and the why in TesNG. This is also another shortcoming of mine - I say stuff in a way that makes people think I am bashing them, which is not my point at all (in most cases). I guess I am just poor at human relations, but I am working on it!

Thanks and regards,
Wujek

2010/2/21 Cédric Beust ♔ <cbe...@google.com>

[Wujek Srujek]

Hi Cedric,

just one more post in this field. In another conversation ("Test superclass's @BeforeClass called even though @BeforeTest failed") you write:
"@BeforeClass only applies to the class it's declared on, or it would complicate initializations quite a bit, so I think it's normal that when it fails, it only invalidates its own class and not its children."

However, it turns out that not only does it invalidate the class that it is defined on - it actually invalidates _all_ test classes that derive from it as well. I think the correct behaviour should be:
1. for each test class, call the @BeforeClass defined in any of its superclasses
2. if such configuration method call fails at any level, skip the tests only in the concrete test class in question
3. for another class that uses the same superclasses (a sibling test) do all the calls again, as the result might be quite different as my contrived example showed

Does this make any sense?

If it does, I would kindly ask you to point me to the code that actually deals with the skips and superclass' calls, I would like to take a look and see if I can help you with this (I know you are busy, congrats on the twins, BTW ;d) if it is within my capacity. I could find the code myself, but it will save me time as you probably know the package / classes from the top of your head.

Best regards,
Wujek

[brandon zeeb]

Wujek,

When you extend classes containing TestNG annotations, your sub-class
effectively inherits those as well. Cédric is correct, by failing in a
@BeforeClass configuration defined in your super-class, you will by
definition of a @Before_____ method skip all tests.

I would recommend defining your @Before____ methods with a higher
level of granularity using groups / dependsOnGroups to chain your
@BeforeClass methods if you need to. This can even happen across
class bounds (ie: a BeforeClass method in the sub class can depend on
a BeforeClass method in a super class). Thus, you can define more
classes in your hierarchy that have higher or lower precondition
levels for your tests to then inherit.

In you example, all tests are being skipped because the same
@BeforeClass configuration method fails for each test, this is what
you want!

> <wujek.sru...@googlemail.com>wrote:

> >>> testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>

> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/testng-users?hl=en.
>
> >> --

> >> ***Cédric
> >> *

>
> >>  --
> >> You received this message because you are subscribed to the Google Groups
> >> "testng-users" group.
> >> To post to this group, send email to testng...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>

[Cédric Beust ♔]

Hi Wujek,

You made me wonder so I went back to the source and found out that you are right:  if a @BeforeClass fails, its class any test class that extends it will be skipped.

Now I realize that I never exactly specified the behavior in such a case, so it's become a tricky situation.

Intuitively, I would agree that a failing @BeforeClass should only invalidate the tests in the current class and current class only, but I can imagine situations where this method is placed in a base class in order to apply to a broad range of subclasses.  In such a situation, the fact that this method is in a superclass is a simple syntactic convenience to avoid copy/pasting it everywhere, but in effect, that method should be considered as present in each subclass.

By the way, everything I'm saying here applies to @BeforeTest as well:  if such a method fails, all the classes in that <test> tag *and/or* their subclasses will get skipped.

Does this make sense so far?

So now I'm in a situation where I can't change the current behavior because I bet that a lot of users are already taking advantage of it, and to be honest, I want to preserve that behavior as well for the reason above.

What I am thinking is that I could add a boolean flag to @BeforeClass/@BeforeTest:

@BeforeClass(strict = true)  // or some other name

which would tell TestNG that a failure in this method should only impact this class and not its subclasses.

What do you think?

By the way, thanks for raising all these points (in this message and your other ones as well):  you are bringing up important details and you seem to have a very good understanding of how TestNG works, so don't be shy if you have additional feedback :-)

--
Cedric

[Wujek Srujek]

Hi,

I guess I mistook @BeforeClass (which is shared by all classes, so subclasses too) for a test instance specific @BeforeInstance (I know it doesn't exist, but maybe it makes sense?). However, it actually is called before each test class, setting its state, so it behaves like the hypothetical @BeforeInstance in this regard, but like a @BeforeClass in that if it fails, it skipps all subclasses. I beg to have an opinion that it is not intuitive.
I will think about this and also take a look if I could do what I want with groups, but it looks like it could work.
Thanks for your valuable comment and the tip on groups, Brandon.

Regards,
Wujek

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Wujek Srujek]

Hi Cedric,

I didn't even think about @BeforeTest classes. What about @BeforeSuite?
I am not sure if I understand about <tests>:
Suppose we have the XML (I use shortcuts for brevity, but I am sure you will know what I mean):
<test name="one"><package name="com.test.one"> and <class name="com.test.BeforeTestCallback">
<test name="two"><package name="com.test.two"> and <class name="com.test.BeforeTestCallback">
BeforeTestCallback defines a @BeforeTest callback that would be invoked for both tests.
Now suppose that <test "one"> is invoked and the callback fails. Does this mean that <test "two"> will also be skipped?
(I am not mentioning subclasses of BeforeTestCallback here, but I think it is not necessary?)
If so, it is also surprising. Suppose that the @BeforeTest callback uses the ITestContext to get some parameters, that can be <test> scoped, and override <suite> level parameters. There is some parameterA which has some default at <suite> level (or not, doesn't matter), however, <test "one"> overrides it, and causes the @BaforeTest to fail (I don't know, maybe the param is a JDBC driver class name that has a typo or something like this). Now, will <test "two"> be skipped because of this, even though it has the correct parameter value (the said driver class) specified (either the default, or overridden, but correctly)? It gets slightly more complicated with using subclasses, but I think this example is enough?
Or did you mean a different scenario: <test "one"> fails becuase of the faluty parameter used in @BeforeTest callback, and all test classes in  <test "two"> that are the same classes or subclasses of tests in <test "one"> are skipped because of that, but others get executed? Now, with this, it would be really confusing. At least for me, as I would expect that the <tests> are completely separate, but it would turn out that they are correlated in a way.
This could also mean that <test "one"> has some class that fails initialization and is skipped. <test "two"> has this class also, and also subclasses - would they be skipped because of the failed @BeforeClass in <test "one">?
I will test this myself if you don't remember. I think it will be a useful excersise.

My general view is that the callbacks can be parametrized per <test> / test class (as with the example with the driver class name, and the previous contrived example that calls getClass() and does some stuff with it), so it actually is possible that the outcome of their calls with different set of parameters is different and should not influence other <tests> / classes that use the same callbacks via inheritance.

I think your idea with the 'strict' attribute would do fine. Do you think it would be a big task or just a few lines somewhere?

Thanks for your kind words, but I am only getting to know TestNG's inner workings.

Regards,

[neotru...@gmail.com]

Hello Cédric,

 

I have faced the same kind of problem in my scripts. I had my scripts in JUnit previously and they worked fine. Since I wanted to exploit the features of testNG I ported all the tests to testNG only to be suprised to see that all the tests fail because of a failure in the before class method of one Java Test class. This somehow violates our notion of inheritance.

 

You have talked about situations where people used the method in the base class as a syntactic convenience, but imho the easy way would be to have a before group annotation (or maybe a static method in the base class?). However in a situation like this what could be the alternative solution? Could you please describe how I could probably use some before__ methods to solve the problem at hand in an easy way?

 

Regards,

Neo

[Jek]

Hi Cedric,

I'd love to have the strict=true (or whatever it would be called) flag
added to the @BeforeMethod and particularity the @AfterMethod. We've
changed the TestNG code to continue to run tests if an @AfterMethod
method fails. (This was only a few lines of code.) But I hate being
on my own version and having to deal with porting my change to newer
versions. This seems really about scope. The discussion so far has
been about reducing the scope of @BeforeClass to just an instance
(which I am in favor of). I want the scope of @XXXMethod reduced to
just a single method, which seems reasonable and consistent with
TestNG behavior. Since that is not how it currently works, a flag
like you purpose could also apply to @XXXMethod.

Here are links to issues and discussions that a flag like this could
resolve, starting with the one that affects me the most.

Add support for verify methods running after test methods
http://jira.opensymphony.com/browse/TESTNG-145

Single Failure in Inherited BeforeMethod Causes All Subsequent
Invocations to be Skipped
http://groups.google.com/group/testng-users/browse_thread/thread/4b5d81fa2c3af24a

A @BeforeMethod failure skips all remaining tests in sibling instances
generated by an @Factory.
http://jira.opensymphony.com/browse/TESTNG-202

Thanks for your consideration,

-John

> <wujek.sru...@googlemail.com>wrote:

> >>>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >>>> .
> >>>> For more options, visit this group at
> >>>>http://groups.google.com/group/testng-users?hl=en.
>
> >>> --

> >>> ***Cédric
> >>> *

>
> >>>  --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "testng-users" group.
> >>> To post to this group, send email to testng...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >>> .
> >>> For more options, visit this
>

> ...
>
> read more »

[neotru...@gmail.com]

Hello John,

 

Could you be kind enough to elaborate on this?

"We've changed the TestNG code to continue to run tests if an @AfterMethod method fails.  (This was only a few lines of code.)"

 

Thanks,
Neo

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Jek]

Hi Neo,

Sure. You can find the aftermethod.patch patch file attached to
TESTNG-145:

http://jira.opensymphony.com/browse/TESTNG-145

I believe the patch is intended for TestNG 5.7. We're using it with
5.8, though I recall applying it by hand. The patch was supplied by
Dan Fabulich.

-John

On Feb 24, 3:07 am, neotrumat...@gmail.com wrote:
> Hello John,
>
> Could you be kind enough to elaborate on this?
> "We've changed the TestNG code to continue to run tests if an @AfterMethod
> method fails.  (This was only a few lines of code.)"
>
> Thanks,
> Neo
>

> On Wed, Feb 24, 2010 at 03:08, Jek <john.m.sm...@attachmate.com> wrote:
> > Hi Cedric,
>
> > I'd love to have the strict=true (or whatever it would be called) flag
> > added to the @BeforeMethod and particularity the @AfterMethod.  We've
> > changed the TestNG code to continue to run tests if an @AfterMethod
> > method fails.  (This was only a few lines of code.)  But I hate being
> > on my own version and having to deal with porting my change to newer
> > versions.  This seems really about scope.  The discussion so far has
> > been about reducing the scope of @BeforeClass to just an instance
> > (which I am in favor of).  I want the scope of @XXXMethod reduced to
> > just a single method, which seems reasonable and consistent with
> > TestNG behavior.  Since that is not how it currently works, a flag
> > like you purpose could also apply to @XXXMethod.
>
> > Here are links to issues and discussions that a flag like this could
> > resolve, starting with the one that affects me the most.
>
> > Add support for verify methods running after test methods
> >http://jira.opensymphony.com/browse/TESTNG-145
>
> > Single Failure in Inherited BeforeMethod Causes All Subsequent
> > Invocations to be Skipped
>

> >http://groups.google.com/group/testng-users/browse_thread/thread/4b5d...

> ...
>
> read more »

[Ben Douglas]

This thread is very very old I know... but it looks like this was never implemented.  Was there another answer here (besides just having a super "helper" method that is called").  I'm using @BeforeMethod as I need something done before each test method in a class (and have several sub-classes that I don't want to fail either).

Thanks,

Ben

[Krishnan Mahadevan]

Ben,

Have you given this one more try?

I cant seem to reproduce this using the latest version of TestNG 6.11.

If you are still facing this problem, I would suggest that you please help create a sample that can be used to reproduce the issue and share that with us.

 

package com.rationaleemotions.googleforums.common;

 

import org.testng.annotations.BeforeClass;

 

public class AbstractTest {

    @BeforeClass

    public void beforeAbstractTestClass() {

        System.out.println("<<<<<>>>>> AbstractTest.beforeAbstractTestClass() for class " + getClass().getName());

        throw new RuntimeException("just testing stuff");

    }

}

 

import org.testng.annotations.Test;

 

public class TestOne extends AbstractTest {

    @Test

    public void test() {

        System.out.println("<<<<<>>>>> TestOne.test()");

    }

}

 

import org.testng.annotations.Test;

 

public class TestTwo extends AbstractTest {

    @Test

   public void test() {

        System.out.println("<<<<<>>>>> TestTwo.test()");

    }

}

 

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Common_Suite" parallel="classes" verbose="2" >

    <test name="Common_Test">

        <classes>

            <class name="com.rationaleemotions.googleforums.common.TestOne"/>

            <class name="com.rationaleemotions.googleforums.common.TestTwo"/>

        </classes>

    </test>

</suite>

 

T E S T S

-------------------------------------------------------

Running TestSuite

...

... TestNG 6.11 by Cédric Beust (ced...@beust.com)

...

 

<<<<<>>>>> AbstractTest.beforeAbstractTestClass() for class com.rationaleemotions.googleforums.common.TestTwo

<<<<<>>>>> AbstractTest.beforeAbstractTestClass() for class com.rationaleemotions.googleforums.common.TestOne

Tests run: 4, Failures: 2, Errors: 0, Skipped: 2, Time elapsed: 0.338 sec <<< FAILURE! - in TestSuite

beforeAbstractTestClass(com.rationaleemotions.googleforums.common.TestOne)  Time elapsed: 0.231 sec  <<< FAILURE!

java.lang.RuntimeException: just testing stuff

 

beforeAbstractTestClass(com.rationaleemotions.googleforums.common.TestTwo)  Time elapsed: 0.231 sec  <<< FAILURE!

java.lang.RuntimeException: just testing stuff

 

 

Results :

 

Failed tests:

  TestOne>AbstractTest.beforeAbstractTestClass:9 » Runtime just testing stuff

  TestTwo>AbstractTest.beforeAbstractTestClass:9 » Runtime just testing stuff

 

 

 

Thanks & Regards

Krishnan Mahadevan

 

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

You received this message because you are subscribed to the Google Groups "testng-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.