@RunWith annotation

[Putrycz, Erik]

JUnit has a nice @RunWith annotation that enables to specify a test
runner.
I've been doing lots of testing with Guice and TestNG and I have to
extend a base class all the time - where I have a @BeforeClass,
@AfterClass, @BeforeMethod and @AfterMethod.
I'm not certain that specifying a test runner is the most elegant
solution, probably in most cases you may simply want to have
@BeforeClass, @AfterClass, @BeforeMethod and @AfterMethod in a different
class.
What I would really like is something like this

@Test
@RunWith(GuiceInitializer.class) // also be able to specify this
globally for all my tests
class A {
@Inject
private Database db;
...
}

class GuiceInitializer {

@BeforeClass
public void initTestClass(Object testInstance) {
...
injector.injectInstance(testInstance);
}

}

Any thoughts?

Erik.

[Mark Derricutt]

I've often thought this would be good, one way you could implement it already without upstream changes would be having a base test class implement IHookable, and have the associated run(org.testng.IHookCallBack,org.testng.ITestResult iTestResult) method look for a @RunWith annotation on the method, and delegate as necessary.

I've not tried this but it should work...

[Cédric Beust ♔]

The only benefit I can see is that it lets you extend a different class if you so desire, but I'm not really convinced it's really that useful.

Let me ask you this:  can you think of scenarios where the annotations that you need cannot be put on the class you extend?

Besides, @RunWith is also a headache for integration since most IDE's will ignore it and they will therefore not make these symbols available to the plug-in...

-- 

Cédric

[Putrycz, Erik]

I don’t think there is any case where you cannot put the annotations in a parent class, or use several levels of inheritance. It’s more a matter of organization, I’d like to have a generic Guice testing framework and use the inheritance to group functions and my own annotations for each of my modules.

 

Why do you think @RunWith would be a headache for integration? Aren’t they all only detecting the @Test and then hand the execution to the test runner? In my example, you won’t pass a test runner to @RunWith, rather somekind of interceptor – so the IDE shouldn’t need to handle it.

 

Erik.

 

 

 

---

From: testn...@googlegroups.com [mailto:testn...@googlegroups.com] On Behalf Of Cédric Beust ?
Sent: January-20-09 3:43 PM
To: testn...@googlegroups.com
Subject: Re: @RunWith annotation

 

The only benefit I can see is that it lets you extend a different class if you so desire, but I'm not really convinced it's really that useful.

 

Let me ask you this:  can you think of scenarios where the annotations that you need cannot be put on the class you extend?

 

Besides, @RunWith is also a headache for integration since most IDE's will ignore it and they will therefore not make these symbols available to the plug-in...

 

-- 

C餲ic

[Mark Derricutt]

One place I've seen this pattern somewhat useful is when mixing some other test frameworks, such as concordion.  In this scenario, theres common/standard before/after boiler plate code that just initializes an environment for the test.  A more 'testng' like approach would be say an @AroundMethod annotation that sat on a method conforming to IHookable's run() method parameters.  It would just be a more succinct way of expressing BeforeMethod/AfterMethod(runalways=true) patterns.

I could see this working better when using groovy/scala traits/mixins.  But could also complicate things more.

[Cédric Beust ♔]

2009/1/20 Putrycz, Erik <Erik.P...@nrc-cnrc.gc.ca>

I don't think there is any case where you cannot put the annotations in a parent class, or use several levels of inheritance. It's more a matter of organization, I'd like to have a generic Guice testing framework and use the inheritance to group functions and my own annotations for each of my modules.

Ok.

 

Why do you think @RunWith would be a headache for integration? Aren't they all only detecting the @Test and then hand the execution to the test runner? In my example, you won't pass a test runner to @RunWith, rather somekind of interceptor – so the IDE shouldn't need to handle it.

The problem is that @RunWith is basically a way to substitute your own runner to whatever other is currently in place.  IDE's typically use their own runner to keep track of progress, so you usually lose a lot of IDE functionality as soon as you use @RunWith in your tests.

The way I see it, JUnit wasn't designed with enough listening and hooking infrastructure in place, so they had to resort to @RunWith to give at least some flexibility to developers.  "Our engine is a black box, but you can replace it completely".

Fixing this was one of my early goals with TestNG, which I achieved by providing not just more configuration annotations but also more listeners and hooks into the engine so that its behavior can be modified and overridden without compromising tooling.

--

Cédric

[Putrycz, Erik]

So instead of a @RunWith maybe specifying the listeners and hooks with annotations would do the trick (I never use the suite.xml files)?

I haven’t found in the listeners something that would correspond to a @BeforeClass but if there is something equivalent, this might be a better way.

Something such as @Listeners(GuiceInitializer.class)

 

Erik.

 

---

From: testn...@googlegroups.com [mailto:testn...@googlegroups.com] On Behalf Of Cédric Beust ?
Sent: January-20-09 4:11 PM
To: testn...@googlegroups.com
Subject: Re: @RunWith annotation

 

 

2009/1/20 Putrycz, Erik <Erik.P...@nrc-cnrc.gc.ca>

Why do you think @RunWith would be a headache for integration? Aren't they all only detecting the @Test and then hand the execution to the test runner? In my example, you won't pass a test runner to @RunWith, rather somekind of interceptor so the IDE shouldn't need to handle it.

 

The problem is that @RunWith is basically a way to substitute your own runner to whatever other is currently in place.  IDE's typically use their own runner to keep track of progress, so you usually lose a lot of IDE functionality as soon as you use @RunWith in your tests.

 

The way I see it, JUnit wasn't designed with enough listening and hooking infrastructure in place, so they had to resort to @RunWith to give at least some flexibility to developers.  "Our engine is a black box, but you can replace it completely".

 

Fixing this was one of my early goals with TestNG, which I achieved by providing not just more configuration annotations but also more listeners and hooks into the engine so that its behavior can be modified and overridden without compromising tooling.

 

--

C餲ic

[Cédric Beust ♔]

2009/1/20 Putrycz, Erik <Erik.P...@nrc-cnrc.gc.ca>

So instead of a @RunWith maybe specifying the listeners and hooks with annotations would do the trick (I never use the suite.xml files)?

This would be easy to add.  The pro is that it would benefit from static validation, the con is that you need to recompile whenever you want to add or remove a listener.

I'm still not convinced it's a big gain over just specifying these listeners in the base class, though...

--

Cédric

[Peter Niederwieser]

IDE runners don't have to be aware of @RunWith except that they need
to
recognize classes with this annotation as JUnit tests. JUnit will do
the
rest. Sorting and filtering tests should still work, unless the
user-provider runner doesn't support it.

Is it possible to write a custom runner for TestNG that gives me full
control over test execution like JUnit does? Last time I checked, I
couldn't
find a way to do this.

Cheers,
Peter

[Putrycz, Erik]

> IDE runners don't have to be aware of @RunWith except that they need
> to
> recognize classes with this annotation as JUnit tests. JUnit will do
> the rest. Sorting and filtering tests should still work, unless the
> user-provider runner doesn't support it.

I think what Cedric meant is that if something such as @RunWith was
added, all the IDE plugins would need to "recognize" it whereas changes
in the test runner only are much easier for all the IDE plugin
maintainers.

> Is it possible to write a custom runner for TestNG that gives me full
> control over test execution like JUnit does? Last time I checked, I
> couldn't
> find a way to do this.

Could you detail what you need custom runner for?
I looked at the JUnit Runner API and it doesn't look too friendly. There
should be better ways to customize the test execution.

Erik.

[Cédric Beust ♔]

On Wed, Jan 21, 2009 at 7:55 AM, Putrycz, Erik <Erik.P...@nrc-cnrc.gc.ca> wrote:

> IDE runners don't have to be aware of @RunWith except that they need
> to
> recognize classes with this annotation as JUnit tests. JUnit will do
> the rest. Sorting and filtering tests should still work, unless the
> user-provider runner doesn't support it.

I think what Cedric meant is that if something such as @RunWith was
added, all the IDE plugins would need to "recognize" it whereas changes in the test runner only are much easier for all the IDE plugin
maintainers.

Exactly.  With @RunWith, developers writing runner classes now compete with IDE's, and by taking away the IDE's ability to run the tests themselves, they will lose functionalities such as runtime reporting, stack trace displaying, etc...

I would rather explore the possibility where IDE's and runner developers alike can develop their own piece of code and they can all work together instead of one replacing the other.
 

> Is it possible to write a custom runner for TestNG that gives me full
> control over test execution like JUnit does? Last time I checked, I
> couldn't
> find a way to do this.

Could you detail what you need custom runner for?
I looked at the JUnit Runner API and it doesn't look too friendly. There
should be better ways to customize the test execution.

I was going to ask just that to Peter:  please explain what you are trying to do and let's see what we can do.

--

Cédric

[Peter Niederwieser]

Cédric Beust ♔ wrote:
>
> With @RunWith, developers writing runner classes now compete with IDE's
>

Just to make sure that we are on the same page: For IDEs that use JUnit in
the way it is intended to be used, @RunWith is totally transparent: IDE
calls JUnit calls @RunWith runner. As of JUnit 4.5, this should take you
pretty far. I'm wondering if it's still necessary to "abuse" JUnit to offer
good IDE integration, but I'm not an expert here.
From the standpoint of a framework developer, the good thing about @RunWith
is that it gives you a lot of flexibility. The bad thing is that it isn't
obvious how to write a fully conforming runner, and that you have to
reinvent the wheel even if you only need a slight variation of the standard
runner. That's why JUnit 4.5 has introduced an API that facilitates the
writing of runners that, to some extent, follow the JUnit model.

Cédric Beust ♔ wrote:
>
> I was going to ask just that to Peter: please explain what you are trying
> to do and let's see what we can do.
>

I'm writing my own test framework and would like to reuse TestNG's IDE and
build tool integration. Not sure if you consider this a valid use case. :-)
Anyway, @RunWith works pretty well for me, but I haven't found a way to
integrate with TestNG.

Cheers,
Peter
--
View this message in context: http://www.nabble.com/%40RunWith-annotation-tp21570801p21590053.html
Sent from the testng-dev mailing list archive at Nabble.com.

[Peter Niederwieser]

> With @RunWith, developers writing runner classes now compete with IDE's [...]

Just to make sure that we are on the same page: For IDEs that use
JUnit in the way it is intended to be used, @RunWith is totally
transparent: IDE calls JUnit calls @RunWith runner. As of JUnit 4.5,
this should take you pretty far. I'm wondering if it's still necessary
to "abuse" JUnit to offer good IDE integration, but I'm not an expert
here.
From the standpoint of a framework developer, the good thing about
@RunWith is that it gives you a lot of flexibility. The bad thing is
that it isn't obvious how to write a fully conforming runner, and that
you have to reinvent the wheel even if you only need a slight
variation of the standard runner. That's why JUnit 4.5 has introduced
an API that facilitates the writing of runners that, to some extent,
follow the JUnit model.

> I was going to ask just that to Peter: please explain what you are trying to do and let's see what we can do.

[Cédric Beust ♔]

On Wed, Jan 21, 2009 at 11:08 AM, Peter Niederwieser <pnie...@gmail.com> wrote:

Cédric Beust ♔ wrote:
>
> I was going to ask just that to Peter:  please explain what you are trying
> to do and let's see what we can do.
>

I'm writing my own test framework and would like to reuse TestNG's IDE and
build tool integration. Not sure if you consider this a valid use case. :-)
Anyway, @RunWith works pretty well for me, but I haven't found a way to
integrate with TestNG.

Yeah, that's probably a bit out there for a valid use case :-)

Still, I'm curious:  what kind of functionalities do your @RunWith runners accomplish?

--

Cédric

[Putrycz, Erik]

> I'm writing my own test framework and would like to reuse TestNG's IDE
> and build tool integration. Not sure if you consider this a valid use
> case. :-)

So what is specific about your test framework?

Erik.

[Toni Menzel]

okay, i have to warm up this discussion because this is exactly what
we (Support TestNG in Pax Exam).

What we currently can do with Junit is this:
@RunWith(OurInvoker.class)
class Foo
@Test
public void bla()
{
}

@OurSpecialConfigutationAnnotationStuff
public Class configure {
.. bla
}

What we do when in OurInvoker:
- recognize all tests, annotations (junit specifics + our specifics)
- wrap the test classes up, make a jar (osgi bundle to be specific)
for each test do that:
- invoke a totally fresh container in a new VM (container= osgi
runtime, but you can think of any container you might think of, say, a
tomcat instance)
- invoke the execution of test via rmi

There's a lot stuff you don't have to think about now, but just the
use case for having a RunWith interceptor.
And Eclipse,Maven and IntelliJ at least recognize this RunWith in case
of Junit to let us intercept the test invokation.

Sure, it is up to the implementor of OurInvoker to delegate as much
reporting / statistics to the real runner (test ng).
So, @Cédric, is there something you can point to in TestNG to support
this in an extensible, futuresafe way without "hacking testng" ?

[Cédric Beust ♔]

Hi Toni,

I can think of several ways you can do this.

The easiest one is to create your own TestNG object, do all your configuration and annotation finding there, add whatever listeners you need and then run it.

Another way would be to use an ITestListener and do this before the suite begins.

Let me know if you need help with either of these.

--
Cedric

--

Cédric