Can TearDown detect if the last test failed?
AndrewA
Basically the subject of the post says most of it. I just wanted to
know if I could write a TearDown that would react differently if the
last test failed/passed. Is there any way to do that?
I'm currently coding in C# using Nunit 2.4.8.
Thanks,
Andrew
Charlie Poole
NUnit doesn't have any way to indicate this, although you
could do it by setting a flag at the end of every test.
If you really need different teardown depending on the
success or failure of your tests, I'd suggest taking
another look at the design of the tests.
OTOH, if you are wanting to do something else in the
teardown method - like reporting, for example - please
tell us what it is and we'll see if there is an easier
solution.
Charlie
Brad Stiles
We once thought we had a need to do this as well, and the way we did it
was to have a private struct in the test class with two properties:
TestName and Result. As each test started, it would set the name to the
test name, and the result to false. If the test made it to the end, it
would set the Result to true. Teardown would then examine it and do its
thing.
Later, we came to understand that this was...less than optimal, and that
we didn't actually need it while the tests were running, and just
started using the XML result file that NUnit produces.
Brad
Kenneth Xu
I have a use case. I'm writing a Multithread lib. In each of my test
cases, I starts multiple threads, registered it with
TestThreadManager, asserts threads are interacting as expected. Any
failure in the thread will be captured by TestThreadManager and later
reported on the main thread when TestThreadManager.JoinAndVerify is
called.
In the tear down, I loop through them and expect the are threads are
terminated. If there is any thread still alive, I need to do two react
differently based on the status of the test.
- If the test is not failed, that means the main thread completed
fine, but one other thread is hanging or taking longer then expected.
I must abort the thread and fail the test.
- If the test is failed, that means mostly likely the hanging thread
is the result of the failure already identified. All I need is just
abort the hanging thread.
I understand that I can set a flag in the end of every each test, but
that sounds very tedious and error prone to me (I have hundreds of
test cases). IMHO, this is something that a test frame can help.
How about accept a TearDown method takes AssertionException as
parameter? In 2.5, test cases are taking parameters already :)
Thanks,
Kenneth
Charlie Poole
> I have a use case. I'm writing a Multithread lib. In each of
> my test cases, I starts multiple threads, registered it with
> TestThreadManager, asserts threads are interacting as
> expected. Any failure in the thread will be captured by
> TestThreadManager and later reported on the main thread when
> TestThreadManager.JoinAndVerify is called.
>
> In the tear down, I loop through them and expect the are
> threads are terminated. If there is any thread still alive, I
> need to do two react differently based on the status of the test.
> - If the test is not failed, that means the main thread
> completed fine, but one other thread is hanging or taking
> longer then expected.
> I must abort the thread and fail the test.
> - If the test is failed, that means mostly likely the
> hanging thread is the result of the failure already
> identified. All I need is just abort the hanging thread.
>
> I understand that I can set a flag in the end of every each
> test, but that sounds very tedious and error prone to me (I
> have hundreds of test cases). IMHO, this is something that a
> test frame can help.
>
> How about accept a TearDown method takes AssertionException
> as parameter? In 2.5, test cases are taking parameters already :)
This has been requested by others as well and it seems like
a reasonable idea. We'll provide it in a future release,
most likely 3.0.
Charlie
Jeff Brown
This feature has been in MbUnit v3 since the beginning and it has made
possible all sorts of things like capturing screenshots on test
failure or recording results to a test database (by way of a test
decorator).
Jeff
Kenneth Xu
> We'll provide it in a future release, most likely 3.0.
it if I can provide (without any promise:)) a patch to accept a tear
down method like below?
[TearDown] public void MyTearDownNeedsStatus(Exception e)
{
}
On Thu, Aug 6, 2009 at 4:29 PM, Jeff Brown<jeff....@gmail.com> wrote:
> This feature has been in MbUnit v3 since the beginning
Jeff, thanks for the info about MbUnit. How easy for me to switch from
NUnit 2.5 (I used the generic/parametered test a lot) to MbUnit 3.0?
Will it be a drop in replacement?
Cheers,
Kenneth
Charlie Poole
Absolutely. It could even go into 2.5.2 if we had it relatively
quickly. A few points you'll need to deal with...
1. Teardowns with arguments are currently not valid and cause
the test to be non-runnable. You would have to make an
exception <g> of those taking a System.Exception as an
argument.
2. The logic in TestMethod should probably call those teardowns
taking an exception argument with the just-thrown exception
or null if none had been thrown.
3. If the excpetion was expected via ExpectedExceptionAttribute,
I don't think you should pass it to the teardown.
And more stuff that will come up, I'm sure...
In 3.0, I don't expect to continue this approach but to use some
sort of context that contains information about the test that
your methods can use. It may be possible for you to do a prototype
of that approach as opposed to passing arguments.
Let me know if you run into issues.
Charlie
> -----Original Message-----
> From: nunit-...@googlegroups.com
> [mailto:nunit-...@googlegroups.com] On Behalf Of Kenneth Xu
> Sent: Thursday, August 06, 2009 2:10 PM
> To: nunit-...@googlegroups.com
> Subject: [nunit-discuss] Re: Can TearDown detect if the last
> test failed?
>
>
Kenneth Xu
That sounds great! I like your idea of using context. ThreadStatic or CallContext? It seems that thread static will not work because there is no reference between the framework dll and core dll.
Also what would be the data structure for this? Again, due to no reference, we'll need to use all buildin types or maybe xml!?
Cheers,
Kenneth
Sent from my Verizon Wireless BlackBerry
-----Original Message-----
From: "Charlie Poole" <cha...@nunit.com>
Date: Thu, 6 Aug 2009 15:00:43
Charlie Poole
> That sounds great! I like your idea of using context.
> ThreadStatic or CallContext? It seems that thread static will
> not work because there is no reference between the framework
> dll and core dll.
Either of those would mess up people testing applications that
make use of them - this is a general problem in test frameworks.
So I was thinking of something of our own, like a TestContext.
> Also what would be the data structure for this? Again, due to
> no reference, we'll need to use all buildin types or maybe xml!?
Probably some sort of property dictionary, indexed by well-known
names. Common values could also have a property. The current
NUnit.Core.TestPackage does something like this. The key would
be to make the core do as little work as possible... For example,
it could cache a reference to the prperty bag the first time it
needed it. This would not change during the test run.
It's definitely doable if someone wants to take the time. It's
just a question of whether it's better to do in the 3.0 project
or the 2.5 project. It's small enough that I don't mind putting
it into 2.5, but it will be simpler when the test runners are
in the same assembly as the framework.
Charlie
Kenneth Xu
> Either of those would mess up people testing applications that
> make use of them - this is a general problem in test frameworks.
While anything in CallContext will be seen by others, Thread static
should probably not impact as they are invisible to others.
> So I was thinking of something of our own, like a TestContext.
We'll certainly have TestContext, this class will be in the framework
dll, there is no way for core to communicate to it unless use some
sort of reflection.
> Probably some sort of property dictionary,
Yeap, a dictionary will work. It'll be fine as long as it is a .Net
framework type.
> but it will be simpler when the test runners are
> in the same assembly as the framework.
Absolutely right, having them in same assembly solves all those
problems mentioned. But I heard that there is advantage of separating
the runner and framework. Don't remember what exactly it is, maybe the
runner can run different test frameworks and/or framework can be run
by different runners?
Charlie Poole
> Absolutely right, having them in same assembly solves all
> those problems mentioned. But I heard that there is advantage
> of separating the runner and framework. Don't remember what
> exactly it is, maybe the runner can run different test
> frameworks and/or framework can be run by different runners?
Well, runner is an overloaded term. I am using it here to mean
the compnents used to load and execute NUnit framework tests
Other meanings - also used in NUnit - are
* component that know how to set up an appdomain and arrange
for a test to run in it.
* component that can launch a process and arrange for tests
to run in it
* executable that interacts with a user to run tests
These should be separate from the framework, because they
may deal with separate framewors. But the component that
can run the nunit framework only can easily be placed in
the same assembly with the framework and versioned
along with it.
There will be more discussion about this design change
later - both here and on the dev list.
Charlie
Kenneth Xu
I have implemented your context idea, but couldn't find a easy way
other then using CallContext. Data in CallContext is identified by
string "NUnit.Framework.TestContext" so hopefully nobody else will use
the same name to cause a conflict. Also, the data is dictionary
inherited from Hashtable so it doesn't implement
ILogicalThreadAffinative, hence, it won't gets propagated out of the
AppDomain, which is a good thing.
Attached is the patch to latest cvs trunk (sorry, no test cases at the
moment). Tested with below and I was able to get the state
successfully. The implementation is extensible as it uses IDictionary
to communicate between core and framework dll.
Cheers,
Kenneth
[TestFixture] public class TestContextTest
{
private TestState _state;
[TearDown] public void TearDown()
{
TestState state = TestContext.State;
Console.WriteLine(state);
Assert.That(state, Is.EqualTo(_state));
}
[Test] public void Ignored()
{
_state = TestState.Ignored;
Assert.Ignore();
}
[Test] public void Success()
{
_state = TestState.Success;
Kenneth Xu
I didn't here from you one this. Did you get a chance to take a look
at it or you think it is just evil :) BTW, with my little (2.5 years)
experience with .Net, I don't see the problem of using CallContext
here. So please enlighten me if I'm missing something.
Thanks,
Kenneth
Charlie Poole
I see I didn't respond. Sorry. It's on my list of things to
address for the 2.5.2 release, but I haven't gotten to it yet.
Charlie
> -----Original Message-----
> From: nunit-...@googlegroups.com
> [mailto:nunit-...@googlegroups.com] On Behalf Of Kenneth Xu
> Sent: Monday, August 10, 2009 12:15 PM
> To: nunit-...@googlegroups.com
> Subject: [nunit-discuss] Re: Can TearDown detect if the last
> test failed?
>
>
Charlie Poole
A little more my suspicions of CallContext...
It's not for any specific technical reason. Rather for this...
NUnit tries to run your tests and uses a number of "tricks"
to do it. Each different "trick" it uses places an added
limitation on the program being tested.
For example, it runs your test in an AppDomain that
it creates. As a result, any code that can only run
in a primary AppDomain - yes, there is such code -
cannot be tested by NUnit.
So whenever a new "trick" comes up, I get suspicious. :-)
In the past, btw, applications that made certain kinds
of changes to the CallContext would crash NUnit. we had
to create a sort of context "firewall" to prevent any
leakage of the context across the AppDomain boundary.
If you look in the source, you'll see an event queue
that serves only one purpose: to make sure that
events are not sent across the appdomain boundary
on the same thread that is running the tests.
So it *seems* like CallContext will work for this,
but I can't help but being suspicions. OTOH, there
are other - maybe harder - techniques that I absolutely
know will not interfere with the user application.
I'll get back to you on this later today or tomorrow.
Charlie
> -----Original Message-----
> From: nunit-...@googlegroups.com
> [mailto:nunit-...@googlegroups.com] On Behalf Of Kenneth Xu
> Sent: Monday, August 10, 2009 12:15 PM
> To: nunit-...@googlegroups.com
> Subject: [nunit-discuss] Re: Can TearDown detect if the last
> test failed?
>
>
Kenneth Xu
No problem at all. I understand the situation. I see you are online
almost 24 hours a day, you guys are making a lot of progress.
Thanks for the good work :)
Kenneth
Charlie Poole
OK, I looked it over and it isn't evil, but it seems we need to
spend more time working on this and I'd like to get 2.5.2 out
tonight. Let's come back to this topic after the release.
Charlie
> -----Original Message-----
> From: nunit-...@googlegroups.com
> [mailto:nunit-...@googlegroups.com] On Behalf Of Kenneth Xu
> Sent: Monday, August 10, 2009 12:15 PM
> To: nunit-...@googlegroups.com
> Subject: [nunit-discuss] Re: Can TearDown detect if the last
> test failed?
>
>
Kenneth Xu
> For example, it runs your test in an AppDomain that
> it creates. As a result, any code that can only run
> in a primary AppDomain - yes, there is such code -
> cannot be tested by NUnit.
I guess this is important for NUnit-GUI for two reasons:
1. NUnit-GUI doesn't keep the test assembly open so it can be rebuild
without closing down the GUI.
2. And still, debugger can easily attach to the NUnit-GUI to debug the test.
But for the console runner, using AppDomain is probably unnecessary,
at least can be optional.
> In the past, btw, applications that made certain kinds
> of changes to the CallContext would crash NUnit. we had
> to create a sort of context "firewall" to prevent any
> leakage of the context across the AppDomain boundary.
> If you look in the source, you'll see an event queue
> that serves only one purpose: to make sure that
> events are not sent across the appdomain boundary
> on the same thread that is running the tests.
I guess this problem is probably related to LogicalCallContext and
ILogicalThreadAffinative, which I purposely avoided here. If not, I
would like to learn how exactly it is, so that I can avoid the problem
in future, where should I start if I want to read the code? Is there a
class name or keyword to search for?
> OTOH, there
> are other - maybe harder - techniques that I absolutely
> know will not interfere with the user application.
Would like to learn too :)
Best wish to 2.5.2 release,
Kenneth
Charlie Poole
> > For example, it runs your test in an AppDomain that it
> creates. As a
> > result, any code that can only run in a primary AppDomain -
> yes, there
> > is such code - cannot be tested by NUnit.
>
> I guess this is important for NUnit-GUI for two reasons:
>
> 1. NUnit-GUI doesn't keep the test assembly open so it can be
> rebuild without closing down the GUI.
> 2. And still, debugger can easily attach to the NUnit-GUI to
> debug the test.
>
> But for the console runner, using AppDomain is probably
> unnecessary, at least can be optional.
application under test their own ApplicationBase, probing path,
configuration file, etc. All the .NET test frameworks I know
of work that way in fact.
In .NET, you don't just load assemblies from wherever you want
to - not normally. And if you used an abnormal (read LoadFrom)
methd of loading test assemblies, you would be seriously
interfering with the normal functioning of the application.
> > In the past, btw, applications that made certain kinds of
> changes to
> > the CallContext would crash NUnit. we had to create a sort
> of context
> > "firewall" to prevent any leakage of the context across the
> AppDomain
> > boundary.
> > If you look in the source, you'll see an event queue that
> serves only
> > one purpose: to make sure that events are not sent across the
> > appdomain boundary on the same thread that is running the tests.
>
> I guess this problem is probably related to
> LogicalCallContext and ILogicalThreadAffinative, which I
> purposely avoided here. If not, I would like to learn how
> exactly it is, so that I can avoid the problem in future,
> where should I start if I want to read the code? Is there a
> class name or keyword to search for?
> > OTOH, there
> > are other - maybe harder - techniques that I absolutely
> know will not
> > interfere with the user application.
>
> Would like to learn too :)
Charlie
Uday
I am a fresher and trying to learn to use NUnit...Here is my question:
In a class with TestFixture attribute, does the method with setup
attribute always be executed before running the method with Test
attribute?
If yes, is there any way to avoid the execution of Setup method before
a particular Test method?
Charlie Poole
> I am a fresher and trying to learn to use NUnit...Here is my question:
> In a class with TestFixture attribute, does the method with
> setup attribute always be executed before running the method
> with Test attribute?
Run TestFixtureSetUp
For each Test method
Run SetUp
Run the Test
Run TearDown
Run TestFixtureTearDown
> If yes, is there any way to avoid the execution of Setup
> method before a particular Test method?
put it in separate TestFixture class.
In some cases, a test method will not use *every* item
in the SetUp and that's OK so long as it's limited. But
any substantive difference in setup really calls for
a new class.
Charlie
> >
>
Kenneth Xu
> We have to create an AppDomain in order to give the test and
> application under test their own ApplicationBase, probing path,
> configuration file, etc. All the .NET test frameworks I know
> of work that way in fact.
I see, that make sense. Very different from Java, thanks for sharing in patient.
> Read the NUnit code. :-)
Yes, I am. It would be great if you point me where the particular
feature is located, otherwise I'll get there eventually :)
Thanks,
Kenneth
Kenneth Xu
> In some cases, a test method will not use *every* item
> in the SetUp and that's OK so long as it's limited. But
> any substantive difference in setup really calls for
> a new class.
Agree! Although this is generally correct but there are times that
different parameters can be used in setup depends on the test cases.
I believe the context we were discussing would solve this problem.
Maybe the test name or a particular attribute can be exposed to SetUp
via context.
Cheers,
Kenneth