Skipping tests at runtime using Annotation Transformer
Mike Willekes
condition that's only known at runtime.
http://groups.google.com/group/testng-users/browse_thread/thread/9d85a1c49dcde2e7
I'd a newbie java developer so to better grasp what TestNG's actually
doing when skipping a test, I loaded the souce into Eclispe and
started debugging... I was experimenting with the code this afternoon
and came up with a way to do what I wanted and would appreciate some
feedback (apologies in advance if I'm way off-base here).
I added a new boolean field called "skipped" to the @Test annotation
(and a few corresponding changes to the related classes) I then made
a small change to Invoker.invokeTestMethods().
>From this (Invoker.java):
boolean okToProceed = checkDependencies(testMethod, testClass,
allTestMethods);
if (okToProceed) {
//
// Invoke the test method if it's enabled
To this (Invoker.java):
// Check to see if test is being force-skipped
boolean forceSkipped =
MethodHelper.isSkipped(testMethod.getMethod(), m_annotationFinder);
// Check to see if all dependencies (groups, methods) are met
boolean okToProceed = checkDependencies(testMethod, testClass,
allTestMethods);
if (okToProceed && !forceSkipped) {
//
// Invoke the test method if it's enabled
To experiment with a real scenario, I created a test with a custom
annotation called "KnownBug". The idea is I want to skip the test
because of a known bug (that's some other component's fault) - but I
don't want to silently disable the test - because I'll probably forget
to reenable it in 3 months when the bug gets fixed. Here's my test
case:
@Test(groups = { "sanity" })
public class MyTest {
@KnownBug(id="BUG12345")
@Test
public void deviceInitialization() {
assertTrue(false);
}
@Test(dependsOnMethods={"deviceInitialization"})
public void queryDeviceState() {
assertTrue(true);
}
@Test
public void resetDevice() {
// there is no error right now.
assertNull(null);
}
}
I then use an annotation transform to search for all methods that have
the @KnownBug annotation and set those tests to be skipped:
public class AnnotationTransformer implements IAnnotationTransformer
{
public void transform(ITest annotation, Class testClass,
Constructor testConstructor, Method testMethod)
{
// Process annotations on the test method
if (testMethod != null){
if (testMethod.isAnnotationPresent(KnownBug.class))
doKnownBugs(annotation, testMethod);
}
}
private void doKnownBugs(ITest annotation, Method testMethod)
{
// If the current test method has a @KnownBug annotation skip
that test
((TestOrConfiguration)annotation).setSkipped(true);
// Get the old description and append to it
String currentDesc = annotation.getDescription();
// Get the Data from the KnownBug annotation
KnownBug bug = (KnownBug)
testMethod.getAnnotation(KnownBug.class);
String newDesc = "Skipped due to known bug: " + bug.id();
((TestOrConfiguration)annotation).setDescription(currentDesc + "
- " + newDesc);
}
}
The end result is that the one test runs successfully (as expected),
one test is skipped due to a known bug and one test is skipped to to a
dependency on the other skipped test.
Thoughts or Opinions?
Cheers!
Mike
P.s. Hmmm... I had thought I could attach a file to a message. :-(
Cédric Beust ♔
Looks all good except... Alexandru checked in a new "SkipException" recently that pretty much does everything you describe below, including the timed aspect (see the subclass TimebombSkipException).
It's only available in SVN right now, though, so feel free to sync and see if it addresses all your needs.
On a similar note, I'd be curious to hear what your experience was about hacking TestNG. Did you find it difficult? Reasonable? Very easy? Anything we could improve in our codebase to make it easier for developers interested in working on TestNG?
Keep us posted!
--
Cedric
(by the way Alex, sorry to be a nag, but it should be called TimeBombException, capital T and B)
Cédric
Mike Willekes
> hacking TestNG. Did you find it difficult? Reasonable? Very easy?
I had no major problems importing the code into Eclipse to try
"hacking" around with it (using the code that's bundled with the 5.5
release). It did take a bit of trial and error to figure out which
JARs had to be added to the project classpath (tools.jar, ant.jar,
junit.jar and the three JARS that ship with TestNG in \3rdParty) to
get everything to compile. It also took me a while to realize that to
test/debug my changes to TestNG while still using the eclipse plugin I
had to export a TestNG JAR to <eclipse\plugins
\org.testng.eclipse_5.5.0.0\lib> (or I could have exported a JAR file
into my test project and selected the "Use project TestNG jar"
option). I have since installed SVN and had no problems fetching a
copy of the latest sources...
> Looks all good except... Alexandru checked in a new "SkipException"
> recently that pretty much does everything you describe below, including the
> timed aspect (see the subclass TimebombSkipException).
> It's only available in SVN right now, though, so feel free to sync and see
> if it addresses all your needs.
We'll be writing tests to test the Java layer that interacts between a
host PC and a hardware device. We already have JUnit tests that are
true "unit" tests (i.e. don't require a real connection to external
hardware) - but now we need to write real system-level tests and JUnit
is woefully inadequate.
I'm looking for a potential solution that implements something simlar
to TCLTest's (which we use to test a similar C++/COM system)
"constraints".
In many cases, the differences between devices is subtle and the only
way to not have to manually maintain a monsterous matrix of which
tests run on which devices is to use these constraints. With TCL, the
test author tags each individual test method with one (or more)
constraints. During the test suite's "setup" routine, we make the
connection to the hardware and then set/clear constraints depending on
which device happens to be currently connected.
I looked into SkipException but:
- I'd like to use annotations to keep track of constraints (as this
will be a familiar concept for test authors):
@Constraints(
is={c1,c2,c3},
isnot={c4}
)
@Test()
public void myTest() throws Exception
{
...
}
However I can't find an elegant way to do this by throwing a
SkipException. Since most (maybe all) of our tests will need this
functionality it makes sense to have a some sort of helper function.
I don't really want the test authors to have to call some static
method (or worse - inherit from a base class):
@Test()
public void myTest() throws Exception
{
if (!(evalConstraints()))
throw new SkipException();
}
I don't think I can centralize the code in a @BeforeTest method -
because I don't think the @BeforeTest method actually knows which
method it's running before... That's why the annotation transformer
seems so useful/powerful.
Thanks,
Mike
Alexandru Popescu ☀
>
> > On a similar note, I'd be curious to hear what your experience was about
> > hacking TestNG. Did you find it difficult? Reasonable? Very easy?
>
> I had no major problems importing the code into Eclipse to try
> "hacking" around with it (using the code that's bundled with the 5.5
> release). It did take a bit of trial and error to figure out which
> JARs had to be added to the project classpath (tools.jar, ant.jar,
> junit.jar and the three JARS that ship with TestNG in \3rdParty) to
> get everything to compile. It also took me a while to realize that to
> test/debug my changes to TestNG while still using the eclipse plugin I
> had to export a TestNG JAR to <eclipse\plugins
> \org.testng.eclipse_5.5.0.0\lib> (or I could have exported a JAR file
> into my test project and selected the "Use project TestNG jar"
> option). I have since installed SVN and had no problems fetching a
> copy of the latest sources...
>
Happy to hear! And in SVN you can find the Eclipse project
configuration that would also answer to the question of what jars must
be on the classpath so that the project is correctly compiled.
Indeed, there may be completely different scenarios for which the
@Skip annotation and SkipException should be used and I think both fit
well in TestNG. However, I do feel that what you are trying to achieve
needs more than just a @Skip annotation (and you are showing some
custom annotations). Also, I am not sure why you would like to mark a
test as skipped if it is not supposed to be run considering the
current test constraints and not just disabled (which can be achieved
by setting invocationCount to 0 in your AnnotationTransformer).
> I don't think I can centralize the code in a @BeforeTest method -
> because I don't think the @BeforeTest method actually knows which
> method it's running before... That's why the annotation transformer
> seems so useful/powerful.
>
I think you mean @BeforeMethod (a method that is invoked before each
@Test). If you declare on a @BeforeMethod a parameter of type Method
then your @BeforeMethod will be aware of what method it is invoked
before.
bests,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> Thanks,
> Mike
>
>
> >
>
Mike Willekes
> configuration that would also answer to the question of what jars must
> be on the classpath so that the project is correctly compiled.
Thanks - I'll take a look at that also...
> Also, I am not sure why you would like to mark a
> test as skipped if it is not supposed to be run considering the
> current test constraints and not just disabled (which can be achieved
> by setting invocationCount to 0 in your AnnotationTransformer).
This origrinally stemmed from the fact that sometimes due to complex
constraints on a test - the test would never actually get executed -
so there was a need to flag tests that were never run.
I just noticed that the TestNG HTML report lists these tests, although
not if invocationCount is set to zero (The test does show up in this
list if it's not run due to a group being excluded, or the test being
disabled it shows up in this list).
> I think you mean @BeforeMethod (a method that is invoked before each
> @Test). If you declare on a @BeforeMethod a parameter of type Method
> then your @BeforeMethod will be aware of what method it is invoked
> before.
This sounds promising - I'll look into it.
Thanks,
Mike