Why aren't the TestNG annotations inherited?

[Anson]

Hello all,

A question for you: why not let TestNG annotations be inherited from
superclasses, so that you can write shared fixtures once (i.e.
@Configuration)? Also, if you've ever heard of the idea of contract
tests, having @Test inherited would be great for them, too. (Contract
tests: you write an abstract test case that tests an abstract class or
interface. Then if someone develops a subclass of your abstract class
or implementation of your interface, they can subclass your test,
implementing a method or two to teach the test how to instantiate the
concrete class, and thereby be easily able to verify that their class
obeys the contract implied by the abstract class or interface.)

In fact, the only TestNG tag I can think of that doesn't make sense to
me as an @Inherited annotation is @Factory. But having a factory on a
superclass doesn't make sense to me anyway.

So...was this a conscious decision, for reasons I cannot yet
appreciate, or just an oversight? Thanks for filling me in,

-aB

[Alexandru Popescu]

It pretty much depends on what you mean by annotation inheritance.

1/

public interface IMyConfiguration {
@Configuration(beforeTestMethod=true)
void beforeTestMethod();
}

public class MyTest implements IMyConfiguration {
[...]
}

this is not supported because it would be against the Java spec.

2/ In terms of inheritance if you class A declares some @Configuration
methods, @Test methods and class B extends A without overridding the
mentioned methods, running B will take into consideration the
inherited methods from A.

hth,

./alex
--
.w( the_mindstorm )p.

InfoQ.com Architect

[Anson]

I am seeing inherited tests working OK, but what I tried to do was
different. So the thread title ought to be: Why aren't superclass
fixtures run before and after subclass tests?

And my question is best expressed as: why aren't inherited methods
annotated with @Configuration(beforeTestMethod = true) and
@Configuration(afterTestMethod = true) run for the tests in the
subclass as if they were declared in the subclass itself? Using
superclasses seems to me to be an extremely valuable way to re-use
common fixtures, and I envisioned having a standard superclass for all
browser-based testing that set up the browser library in a protected
variable a la JUnit 4.0. But if I have to override the methods in
every class to get TestNG to find and run them, they become nearly as
much of an irritant as re-implementing the fixture in every class.

-aB

[Alexandru Popescu]

When you use the word fixture do you mean @Configuration? If so, than
those should be run exactly as you describe. But as I pointed out in
some previous threads at this moment there might be some problems when
computing the graph of execution. I am planning to look into this
quite soon, because I see more and more reports about this.

So, here is what TestNG should do:

public class Parent {
@Configuration(beforeTestClass=true)
public void beforeTestClassInParent() {}

@Configuration(beforeTestMethod=true)
public void beforeTestMethodInParent() {}

@Configuration(afterTestMethod=true)
public void afterTestMethodInParent() {}

@Configuration(afterTestClass=true)
public void afterTestClassInParent() {}
}

public class Child extends Parent {
@Configuration(beforeTestClass=true)
public void beforeTestClassInChild() {}

@Configuration(beforeTestMethod=true)
public void beforeTestMethodInChild() {}

@Configuration(afterTestMethod=true)
public void afterTestMethodInChild() {}

@Configuration(afterTestClass=true)
public void afterTestClassInChild() {}

@Test
public void myTestMethod() {}
}

When running Child test TestNG should run the following methods in order:

beforeTestClassInParent
beforeTestClassInChild
beforeTestMethodInParent
beforeTestMethodInChild
myTestMethod
afterTestMethodInChild
afterTestMethodInParent
afterTestClassInChild
afterTestClassInParent

hth,

./alex
--
:Architect of InfoQ.com:
.w( the_mindstorm )p.

On 5/20/06, Anson <ans...@gmail.com> wrote:
>

> X-Google-Language: ENGLISH,ASCII-7-bit
> Received: by 10.11.53.63 with SMTP id b63mr72901cwa;
> Fri, 19 May 2006 16:42:44 -0700 (PDT)
> X-Google-Token: 4-jdigwAAAC61kBY2dcMEhi1mTwTFGr_
> Received: from 12.162.10.2 by u72g2000cwu.googlegroups.com with HTTP;
> Fri, 19 May 2006 23:42:43 +0000 (UTC)
> From: "Anson" <ans...@gmail.com>
> To: "testng-users" <testng...@googlegroups.com>
> Subject: Re: Why aren't the TestNG annotations inherited?
> Date: Fri, 19 May 2006 16:42:43 -0700
> Message-ID: <1148082163.9...@u72g2000cwu.googlegroups.com>
> In-Reply-To: <c6f400460605191548j6f0...@mail.gmail.com>
> References: <1148074002.0...@g10g2000cwb.googlegroups.com>
> <c6f400460605191548j6f0...@mail.gmail.com>
> User-Agent: G2/0.2
> X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3,gzip(gfe),gzip(gfe)
> Mime-Version: 1.0
> Content-Type: text/plain
> X-Google-Approved: the.mindstor...@gmail.com via web at 2006-05-20 10:02:38

[Cédric Beust ♔]

Hi Anson,

As far as I can tell, TestNG works exactly the way you expect.  I am attaching the ParentTest and ChildTest classes I used with a before/afterTestMethod, and here is the output:

[Parent] PARENT BEFORE TEST
[Parent] CHILD BEFORE TEST
[Parent] TEST CHILD
[Parent] CHILD AFTER TEST
[Parent] PARENT AFTER TEST

Do you have a specific test case to show us where TestNG is not behaving the way you expect?

--
Cédric

[Alexandru Popescu]

Ced, I haven't tried it out yet, but I remember the same problem
reported by Christian (indeed Christian had also some groups)

./alex
--
.w( the_mindstorm )p.

[Anson]

Great, it's good to have a clear statement that it's designed to do
what I want. Now I can throw it in the debugger and see what it's
doing. I'd send you the code, but I didn't create my tests in
isolation, so they're deeply imbedded in the test systems we've already
built--it'd be tough to send you a meaningful fragment. But let me
modify your scenario to more closely emulate what I've set up:

public class Parent {
protected String foo = null;

@Configuration(beforeTestMethod=true, alwaysRun=true)
public void beforeTestMethodInParent() {
foo = "Hi mom!";
}

@Configuration(afterTestMethod=true, alwaysRun=true)
public void afterTestMethodInParent() {
System.out.println(foo);
foo = null;
}

}

@Test(groups={"bar"})

public class Child extends Parent {

@Test(groups={"smoke", "fast"})
public void myTestMethod1() {
assertNotNull(foo);
}

@Test(groups={"smoke", "fast"})
public void myTestMethod2() {
assertNotNull(foo);
}

@Test(groups={"smoke", "slow"}, dependsOnMethods={"myTestMethod2"})
public void myTestMethod3() {
assertNotNull(foo);
}

}

Note that the parent has no tests, and the child no configurations.
Groups are involoved, as are dependencies, and I am running the 'smoke'
group. Can you give that a try in your vanilla environment, and let me
know if it works? Or let me know if my code is flawed somehow?
Hopefully, I've represented all of the TestNG complexity in this
example, but of course, there's more complexity when you factor in the
fact that the protected member of the parent is a Java object that
wraps a browser session, and in my world, Parent extends another
superclass. Anyway, it works if I add the following methods to the
child:

@Configuration(beforeTestMethod=true, alwaysRun=true)
public void beforeTestMethodInParent() {
super.beforeTestMethodInParent();
}

@Configuration(afterTestMethod=true, alwaysRun=true)
public void afterTestMethodInParent() {
super.afterTestMethodInParent();
}

So I'm relatively confident that the code ought to work, and it's
something in TestNG going haywire. I'll throw the full/rich code in
the debugger on Monday if it works for you, and let you know where it's
failing for me. If you get the chance, and this test works smoothly,
letting me know a good place to set a breakpoint in order to watch the
code decide which configurations to call would be very helpful.

Thanks!

-aB

[Eran]

Hi Anson,

There is indeed a problem when combining groups and inheritance (see this thread).
Alexandru, Cedric and myself have discussed this issue off list and hopefully a solution will be found soon.

Regards,
Eran

On 5/20/06, Anson <ans...@gmail.com> wrote:

X-Google-Language: ENGLISH,ASCII-7-bit
Received: by 10.11.53.63 with SMTP id b63mr87909cwa;
        Sat, 20 May 2006 11:13:05 -0700 (PDT)
X-Google-Token: L_sOxQwAAADqe92gRrxbFl2yIev5GSbn
Received: from 63.197.3.248 by j33g2000cwa.googlegroups.com with HTTP;
        Sat, 20 May 2006 18:13:05 +0000 (UTC)

From: "Anson" < ans...@gmail.com>
To: "testng-users" <testng...@googlegroups.com>
Subject: Re: Why aren't the TestNG annotations inherited?

Date: Sat, 20 May 2006 18:13:05 -0000
Message-ID: <1148148785.3...@j33g2000cwa.googlegroups.com>
In-Reply-To: < c6f400460605200310j262...@mail.gmail.com>
References: <1148074002.0...@g10g2000cwb.googlegroups.com>
   <c6f400460605191548j6f0...@mail.gmail.com>
   < 1148082163.9...@u72g2000cwu.googlegroups.com>
   <c6f400460605200310j262...@mail.gmail.com>
User-Agent: G2/0.2
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3,gzip(gfe),gzip(gfe)
Mime-Version: 1.0
Content-Type: text/plain

[Alexandru Popescu]

The quick problem I have spotted with the attached code is that the parent @Configuration methods do
not belong to the group you are running ('smoke').

As Eran pointed out, a long discussion has taken place on and offline about how TestNG would best
behave. By looking at your code, I can confess that seeing all those annotations around I am myself
confused.

For example: you set a @Test(groups={"bar"}) on the Child type, but than you are using the @Test
annotation on all the methods on that type. Is your expectation that the type-level defined group
should be somehow inherit on the method-level?

To me this is really difficult to read and considering other combination of attributes on @Test
annotation it becomes quite dangerous.

./alex
--
.w( the_mindstorm )p.

#: Eran changed the world a bit at a time by saying (astral date: 5/21/2006 12:29 AM) :#

> Hi Anson,
>
> There is indeed a problem when combining groups and inheritance (see this

> thread<http://groups.google.com/group/testng-users/browse_thread/thread/74fa13f39b8a400b/ddaf159c89f60ff1>

[Anson]

Alex-

I took 'alwaysRun=true' to mean that a given annotation would be run
regardless of the groups setting. Quoting the doc: "For before methods
(beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not
beforeGroups): If set to true, this configuration method will be run
regardless of what groups it belongs to."

And actually, my code has a @Test annotation at the class level to show
other people that I consider it a shortcoming that you can't define
groups at the type level and inherit them at the method level. My
vision was basically to give a group to each class at the class
level--that group indicating what component/module the class is
in/for--and a fast/slow smoke/regression group to each method, but of
course that doesn't work in TestNG.

There are reasons why this isn't a big deal at present. Chief among
them is the fact that we test per module, because each module can have
a different classpath. But only being able to specify groups at the
per-method level makes it hard to enforce any sorts of rules about
grouping, so I at present expect that to roll TestNG out to a large
development project will require writing some factories for apt (the
Annotation Processing Tool) so that we can build a validator for tests,
to ensure that each test method is bound to a reasonable set of groups
(i.e. one of [ fast | slow ], but not both, etc.). I might even
encourage the TestNG project to incorporate a basic framework for that,
so that it's easier for people to write such validators as they see the
need.

I agree with you that the @Test annotation at the type level creates
complexities; me personally, I would recommend a different annotation,
@TestSuite, that just has attributes that make sense at the type level.
It and @Test could inherit from an abstract annotation if that made
the most sense. Then it'd be clearer what propagated to the method
level, and what didn't.

But if someone wants to go and write 10,000 tests on TestNG, there's
going to have to be mechanisms in place to keep them well-organized.
That's what's prompting me to explore these issues at present.

-aB

[Anson]

I would definitely agree with the concluding post in that thread:
eliminating @Configuration in favor of @BeforeSuite, @BeforeTest,
@BeforeTestMethod, etc. would enforce a great deal more type-safety
than the current approach. Right now, you can create senseless
@Configuration tags all too easily.

I assume you'd have to support @Configuration in parallel for a while
to handle backwards compatibility, but I believe an enumerated set of
annotations would be quite valuable to new users of TestNG.

-aB