Custom runner or @RunWith equivalent

[Harald Wellmann]

This seems to be a FAQ on this list, but after reading the threads with a similar subject, I still don't see the answer, so pardon me for trying again...

JUnit is extremely extensible by means of its @RunWith annotation, which lets a test class specify the runner to be used for executing the tests. This mechanism is completely transparent to any JUnit integration like Maven Surefire or the Eclipse JUnit support, because it does not affect the JUnit API.

@RunWith is leveraged by various embedded or remote container testing frameworks like Spring Test, Pax Exam, jeeunit or Arquillian, just to name a few. Using these extensions with Eclipse or Surefire requires zero configuration and zero lines of additional code.

With TestNG, there seems to be a multitude of extension mechanisms, none of which is transparent enough to serve as an equivalent to JUnit's @RunWith:

1) Extend org.testng.TestNG and override some its methods as needed.
2) Invoke org.testng.TestNG with a -testrunfactory option.
3) Define listeners via META-INF/services
4) Define listeners by annotation.

1) or 2) cannot be used because they affect the integration API. There is no way I can tell Eclipse or Maven Surefire to use my derived class or to pass that extra commandline argument to TestNG.

3) would be the most elegant solution IMHO, if it also worked for ITestRunnerFactory and not just for ITestNGListener.

Adding listeners by option 3) or 4) does not work either, as there is no way to intercept and redirect or supersede all configuration and test method invocations.

Am I simply missing an obvious solution? Or otherwise, is there a chance for TestNG to supply the missing bits (like a runner factory via ServiceLoader) in a forthcoming release?

Best regards,
Harald

[Cédric Beust ♔]

Hi Harald,

As you correctly point out, TestNG offers this extensibility through various hooks and listeners instead of a generic runner.

Can you give more details on what you're trying to do exactly? We can see from there if we can help you with the existing mechanisms and if not, explore what we can do.

Thanks.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/jwczosrJfI8J.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Harald Wellmann]

Hi Cedric,

the use case is in-container testing. The container might be a Java EE 6 server or an OSGi framework or whatever, running embedded in the same VM as TestNG or in a remote VM.

1) Users shall be able to run an in-container TestNG from their IDE or their build tool, simply by adding an annotation to the test class or by adding a lib to the classpath.

2) When the suite starts, the suite runner needs to launch the container, deploy the system under test, augmented by a proxy runner that will do the real work within the container.

3) When running a test method, the suite runner shall delegate the method execution to the proxy runner, so that the test method will run in the container context.

4) The same holds for all before and after methods (on suite, test, class or method level): these methods shall be executed by the proxy runner and not by the suite runner.

I think that 2) and 3) can be solved by an ISuiteListener and an IHookable, respectively, but I don't see how to solve 4). With an IInvokedMethodListener, we can forward before and after methods to the proxy runner, but this is not enough, we need to intercept them to prevent them from being executed by the suite runner as well.

Best regards,
Harald

[Harald Wellmann]

Is it true that TestNG does not currently provide any hooks for
intercepting @BeforeXXX and @AfterXXX methods?

Best regards,
Harald

[Cédric Beust ♔]

IConfigurationListener2 allows you to get notified of when configurations start and when (and how) they complete.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Harald Wellmann]

Yes, but that's listening, and not intercepting - is there are way for
the listener to _replace_ the configuration method call by some other
invocation?

Thanks,
Harald

[Cédric Beust ♔]

Not at the moment.

It shouldn't be too hard to provide an IConfigurationInterceptor (similar to the already existing IMethodInterceptor) which would hand you a list of configuration methods it's about to run, let you alter it and then use the list that you return.

-- 

Cédric

[Mark Derricutt]

I've often wanted to do this ( for Concordion, and now Cucumber ).

Having thought many times on this, TestNG's ObjectFactory does what one needs, however - the current behavior of adding @ObjectFactory declares a method to be used for ALL test instantiates.

Would a compromise/adaption here be something like:

@ObjectFactory(MyConcordionRunner.class)

public class MyTest {

….

or another suitably named annotation, this marks that THIS particular test class should look at MyConcordionRunner.class for a method annotated with @ObjectFactory and use that to instantiate the test class.

My scenario is that the test class wants common setup for a third party harness.

I wonder, if theres anyway that IHookable could be extended to affect other instances rather than its own. In the case of having something construct you, that class may wish to also decorate around your test method/run in a more coupled fashion than just a listener.

[Andreas Pieber]

Hey Guys,

Just to push this thread and idea; as Harald already pointed out a possibility to to "fully" hook into test-ng would definitely be of help. Although this feature would be interesting for various test frameworks I speak especially for the ops4j pax exam community here. We're getting more and more requests to fully support test-ng in paxexam 2.x but couldn't because of the missing hook.

Is there any chance of getting this in some future? Or any JIRA we can vote on to give this feature more weight? :-)

Thanks and sorry if beeing annoying.

Kind regards,

Andreas

[Cédric Beust ♔]

Hi Andreas,

Before going down that path, let's see what we can do with the current design.

What do you need exactly to support TestNG?

-- 

Cédric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/icXPoLItNCoJ.

[Harald Wellmann]

Am 26.10.2011 09:33, schrieb Cédric Beust ♔:
>
> What do you need exactly to support TestNG?
>

See further up in this thread: we need an interceptor (not merely a
listener) for configuration methods.

The IConfigurationInterceptor you mentioned on 25 Aug sounds fine to me.

Best regards,
Harald

[Cédric Beust ♔]

The thing is: IMethodInterceptor doesn't really let you replace test methods, just modify the list of test methods (e.g. reordering them, removing some, etc...). There is currently no facility to completely replace one method with another one.

Can you give me an example of the kind of interface/API you'd like to see TestNG support?

-- 

Cédric