capture failed tests
steve.ebersole
TestNG? In JUnit, this was achieved via overriding runTest() with a
try/catch block. Is something similar available in TestNG?
I would not mind something like the runTest() from JUnit, but I was
thinking something like the ability to annotate a method with @Failure
(or similar) would be awesome.
Alexandru Popescu ☀
It is not very clear what you would like to achieve. If your question is:
- how can I ignore failing tests:
a) you can disable them
b) you can add them to a special "failures" group that is excluded
- how can you deal with exceptions in @Test
use @Test(expectedExceptions={})
Can you please add some more context to your question?
bests,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
>
> >
>
steve.ebersole
Absolutely. I was trying to keep it short and sweet ;)
Essentially, as part of a "before class" process I would like to build
a Hibernate SessionFactory. The SessionFactory should get closed
"after class". However, in between, if a test fails, I need to close
the current SessionFactory and create a new one (essentially
rebuilding the database schema) so that the next test after the
failing test can start fresh.
Alexandru Popescu ☀
This is an interesting one!
In TestNG if a test fails the other will be skipped. However this
doesn't seem to fit your scenario. On the other hand, if you declare
the @Test(expectedException="HibernateException") then the @Test will
be considered as passed, so you will not know it.... Hmmm, I don't
think I have a solution for this. Must think more about it.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
PS: there may be an ugly one, though: considering the test instance is
statefull (as opposed to JUnit where it is stateless, meaning that for
each test method the class is re-instantiated), you can go with a
trick like:
private boolean m_failed;
@BeforeMethod
public void validateSessionFactory() {
if(m_failed) {
// m_sessionFactory =
m_failed = false
}
}
@Test
public void someTest() {
try {
//
}
catch(HibernateException hibex) {
m_failed = true;
//
Reporter.log("someTest failed: " ....
}
}
Unfortunately this has a few drawbacks as you can probably notice :-(.
>
> >
>
Cédric Beust ♔
How about setting a flag if a test fails and checking for this flag in an @AfterMethod? (and if it's true, create a new session)
--
Cédric
Alexandru Popescu ☀
That's exactly what I've suggested too (unfortunately it has a couple
of drawbacks: mainly you are loosing the fact that the test failed).
This scenario is a bit against of TestNG principles: it wants to
continue after a failure :-).
bests,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> --
> Cédric
>
> >
>
Cédric Beust ♔
As Alexandru pointed out, this would work but would be pretty tedious since you'd have to repeat this in every test method.
I can imagine introducing a new kind of annotated method that could be called every time a test method throws an exception:
@OnFailure
public void onFailure(Throwable e, ITestNGMethod method) {
}
If this method throws an exception, then the method failure is considered a "real" failure and all the other methods on the class are skipped (as is the case today), but if the @OnFailure method returns without throwing, the test method is still logged as a failure but the other methods will be invoked anyway.
Seems a bit contrived and I'd like to be convinced that there are other situations where such a feature would be useful...
--
Cédric
Alexandru Popescu ☀
I am thinking that having something like JUnit runTest() protocol may
be very intersting. It will probably partially solve this problem, it
will allow easier integration with mocking frameworks.
bests,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> --
> Cédric
> >
>
steve.ebersole
Unfortunately, that will not work for me :(
Is there not a way to tell TestNG to "keep going" on failure?
steve.ebersole
> public void onFailure(Throwable e, ITestNGMethod method) {
>
> }
>
> If this method throws an exception, then the method failure is considered a
> "real" failure and all the other methods on the class are skipped (as is the
> case today), but if the @OnFailure method returns without throwing, the test
> method is still logged as a failure but the other methods will be invoked
> anyway.
>
part" in here.
Well not sure what I can do to convince you about the overall
usefulness of this. I can tell you that it is absolutely needed for
me to move Hibernate over to TestNG. We have what some would call
poorly designed test classes where lots of different types of behavior
and different types of assertions are tested in the same test
classes. The one thing in common for these test classes is that all
the test methods operate against a common database schema. Because
they test different pieces of functionality, I don't want the tests to
stop when a particular method fails. However, because they use the
same schema and I don't know if the schema was properly cleaned, I
want to recreate that schema before the next test begins.
If there is another way to achieve this given existing TestNG
features, I'm happy to learn about it and try out that approach. But
I did not see anything.
steve.ebersole
is specific to me and my use-case) would be the ability to know which
test failed in terms of its overall position within the test class
methods. Why? So I know whether to even bother rebuilding the
SessionFactory (if it was the last test method which failed, there is
no need).
Cédric Beust ♔
The Javadoc can be found here.
What do you think?
--
Cédric
Alexandru Popescu ☀
> IHookable.
>
> The Javadoc can be found here.
>
> What do you think?
>
He he... we are having so many features that even us can forget about :).
I think this may be a partial solution, but I am not 100% sure it will
solve the fail - continue scenario (maybe we should add a test level
parameter for dealing with this).
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> --
steve.ebersole
me from the javadocs whether that was the case, or if it was just
called once for the class. In the case of b4 each method, then this
certainly would fit the bill I think; otherwise I don't see how that
could work unless I can keep calling callback() after each failure.