How to Programatically Skip Tests

[David Genrich]

I would like to have a list of tests that I want to skip, but the list will be created at runtime.  Due to this, excluding groups will not meet my needs, since those needs to be set before runtime.

Basically, I want to do something like this:

If TestA fails for a SPECIFIC reason, then skip TestB.  If TestA fails for some other reason, run TestB.

 

Is there a way in onStart, that I could cause a test to be skipped?

 

Here is kind of what I was thinking….

 

public class TestListener extends TestListenerAdapter {

 

      public void onTestStart(ITestResult arg0) {

            super.onTestStart(arg0);

                 

            if (skipTests.contains(arg0.getMethod().getDescription()))  {

                  // skip the test, not sure how?

            }

           

      }

}

 

 

With this, when a test starts, I read the description attribute.  If that matches an element in skipTests, then the test is skipped.

 

Is this possible?  Any advice?

 

Thanks,
David Genrich

 

[Ivan ✪]

what about throwing a SkipException? (org.testng.SkipException)

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>

[Marcus Döring]

you can use the IAnnotationTransformer interface.
you would need to have the @Test annotation on every Method though.
But with that you could do exactly what you descriped (Or something
along those lines).

On 16 Feb., 19:21, David Genrich <david.genr...@gmail.com> wrote:
> I would like to have a list of tests that I want to skip, but the list will
> be created at runtime.  Due to this, excluding groups will not meet my
> needs, since those needs to be set before runtime.
>
> Basically, I want to do something like this:
>
> If TestA fails for a SPECIFIC reason, then skip TestB.  If TestA fails for
> some other reason, run TestB.
>
> Is there a way in

> onStart<http://testng.org/javadocs/org/testng/TestListenerAdapter.html#onStar...)>,

> that I could cause a test to be skipped?
>
> Here is kind of what I was thinking….
>

> *public* *class* TestListener *extends* TestListenerAdapter {
>
>       *public* *void* onTestStart(ITestResult arg0) {
>
>             *super*.onTestStart(arg0);
>
>             *if* (skipTests.contains(arg0.getMethod().getDescription()))  {

[Marcus Döring]

Ah damn...never mind. I'm probaply too tired...
The IAnnotationTransformer runs only once on startup. So it wouldn't
be possible to change something at runtime with that.

Some Time ago I had the same idea as you do now, but I had to give up
on it. It just seems to be not possible.

The things I tried were: having an TestListener who has some methods
for failing / skipped... tests, then I wanted to change the @Test
annotation via reflections.
Sadly they dont seem to have any setter methods.
It could maybe work if you wrapped the TestNG Annotation and used
this, not shure, as I said, I had to give up.

On 17 Feb., 21:03, Marcus Döring

[Cédric Beust ♔]

Hi David,

One way of achieving this would be to maintain a globally available map of what methods need to be skipped. You'd update this map as you find that tests fail (with a listener). Then each test could check this list and automatically throw a SkippedException if they find their name in it.

It's not exactly lightweight, but there is no way you can preemptively mark a method skipped or fail before it's actually run.

-- 

Cédric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Cédric Beust ♔]

On Thu, Feb 17, 2011 at 2:08 PM, Marcus Döring <wtf.my.name.is...@googlemail.com> wrote:

Ah damn...never mind. I'm probaply too tired...
The IAnnotationTransformer runs only once on startup. So it wouldn't
be possible to change something at runtime with that.

Correct.

 

Some Time ago I had the same idea as you do now, but I had to give up
on it. It just seems to be not possible.

I just suggested one way to do this, there might be more.

 

The things I tried were: having an TestListener who has some methods
for failing / skipped... tests, then I wanted to change the @Test
annotation  via reflections.
Sadly they dont seem to have any setter methods.

Correct, the reflection API is read-only.

 

It could maybe work if you wrapped the TestNG Annotation and used
this, not shure, as I said, I had to give up.

That's exactly how TestNG does it: wrap each annotation in a mutable class (they are called ITestAnnotation, IFactoryAnnotation, etc...). These are the classes you receive in the annotation transformer, and since they are mutable, you can change their value (which is the point of the annotation transformer).

--
Cédric

[David Genrich]

I did get this to work:

public class TestListener extends TestListenerAdapter {
public void onTestStart(ITestResult arg0) {
super.onTestStart(arg0);

if (skipTests.contains(arg0.getMethod().getDescription())) {

throw new SkipException ("Skipping Test: " +
arg0.getMethod().getDescription());
}
}
}

It does exactly what I need it do - skips the tests that are in the
ArrayList skipTests.

Thanks,
David Genrich

[Cédric Beust ♔]

Ah, I wasn't sure that skipping from a listener would work. Glad to hear it does, and it saves you from having to put this logic in every single method.

Nice.

-- 

Cédric

Thanks,
David Genrich

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Jeff Russell]

I'm not sure that it does. When I try to throw a SkipException in my
listener, it throws the skip exception, but seems to just keep going.
The test does not get skipped.
public class LoggingTestListener implements IInvokedMethodListener,
ITestListener {
public void onTestStart(ITestResult result) {

String autouuid = getAutoUuid(result);

if ((autouuid == null) || (autouuid.equals(""))) {
throw new SkipException("Auto UUID ID is Missing.");
}
}
...
}

In my output I see...
ERROR 2011-03-08 16:57:50,587 -
com.my.framework.testng.LoggingTestListener - Auto UUID ID is Missing.
ERROR 2011-03-08 16:57:50,595 -
com.my.framework.testng.NPFTestListener - [NPF Framework] Exception in
TestNG Listener
org.testng.SkipException: Auto UUID ID is Missing.
at
com.my.framework.testng.LoggingTestListener.onTestStart(LoggingTestListener.java:
134)
at
com.my.framework.testng.NPFTestListener.onTestStart(NPFTestListener.java:
230)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1484)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1459)

Am I missing something?

On Feb 17, 7:00 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Ah, I wasn't sure that skipping from a listener would work. Glad to hear it
> does, and it saves you from having to put this logic in every single method.
>
> Nice.
>
> --
> Cédric
>

[jweiss]

We recently upgraded from TestNG 5.9 to the latest version (5.14.10).

Throwing SkipException in onTestStart in a listener *used* to work.
Now it doesn't. Now TestNG will swallow the SkipException and report

java.lang.NullPointerException: Method should not be null
at org.testng.SuiteRunner.afterInvocation(SuiteRunner.java:602)
at org.testng.internal.Invoker.runInvokedMethodListeners(Invoker.java:
543)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:799)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1103)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:
137)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:
121)
at org.testng.TestRunner.runWorkers(TestRunner.java:1098)
at org.testng.TestRunner.privateRun(TestRunner.java:727)
at org.testng.TestRunner.run(TestRunner.java:581)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:315)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:310)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:272)
at org.testng.SuiteRunner.run(SuiteRunner.java:221)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:40)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:83)
at org.testng.internal.thread.ThreadUtil
$CountDownLatchedRunnable.run(ThreadUtil.java:151)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

I tried each version of TestNG to find where this broke. It was
between 5.12.1 and 5.13.

I would be grateful if this functionality could be restored. Here's
our use case -

If a test is failing and we've already opened a bug on it, we add a
group "blockedByBug-123456" where 123456 is the bug number in our bug
tracker. We have a BugListener that checks the groups in onTestStart,
and looks up the bug in the tracker (via xmlrpc). If the bug is still
not fixed, it throws SkipException to skip the test (why bother
running it when we already know it's going to fail?).

Without this functionality our bug tracker integration does not work
at all.

Is there a workaround? Some way of forcibly setting that
IInvokedMethod so that it's not null when the SuiteListener looks at
it? I don't see any way of accessing it from the listener.

Thanks,
Jeff

[jweiss]

I created issue http://jira.opensymphony.com/browse/TESTNG-470 to
track this.

[Cédric Beust ♔]

Hi Jeff,

I just posted a comment on the issue, let's move the discussion there.

-- 

Cédric

On Mon, Mar 14, 2011 at 12:47 PM, jweiss <jeffrey...@gmail.com> wrote:

I created issue http://jira.opensymphony.com/browse/TESTNG-470 to
track this.

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric