NUnit Failures that Stop the Test Run
Charlie Poole
I've had some discussions with the submitter of this bug...
https://bugs.launchpad.net/nunit-3.0/+bug/730891
about various ways of stopping the test run on first failure.
Rather than discussing different implementations and syntaxes
for doing it, I thought we should discuss the following question:
"Is it reasonable to permit a test to signal that the test run
should be completely abandoned?"
I can see several reasonable ways of designating certain
failures as "Fatal" provided we think it's
1) Useful to at least some people
2) Not dangerous to the general user
Charlie
Rafael Teixeira
---------------------------------------
"The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' (I found it!) but 'That's funny ...'"
Isaac Asimov
US science fiction novelist & scholar (1920 - 1992)
--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To post to this group, send email to nunit-...@googlegroups.com.
To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
Charlie Poole
Any setup (i.e. SetUp, TestFixtureSetUp or SetUpFixture SetUp method) will
cause the tests under it's control to be abandoned. They are not run, although
they may appear to because they show up as red in the Gui and are counted
as failing test cases in the summary report.
I'm not sure if the requestor's use case can be translated into some sort
of setup check. That might be worth exploring.
Charlie
nr4christ
Not sure how I would go about doing that. Do you mean writing several
unit tests inside Setup? I doubt that's what you mean, but I honestly
don't understand.
Let me do a quick overview of my scenario, to bring anyone in this
thread up to speed.
I am writing subsystem level tests -- basically integration tests (or
sometimes called "component" tests). They are not what the TDD folks
would normally call "unit" tests, as I am not testing a method in
isolation.
In fact, as I am testing unmanaged code written to talk to cellular
devices, the unit tests excercises: our code, native SDKs, drivers,
and cellular devices. Cellular network conditions can also come into
consideration.
In such a scenario it is very possible to have certain type of errors
that are not easily reproducible by just running one unit test (e.g.,
race conditions) but can be reproduced by running a batch of unit
tests (failing at different unit tests, but in the same section of the
code under test).
In one bug that I am working on currently, I found it very useful to
use a TestContext in the TearDown method to find out if the test
failed, and then hang the TearDown method upon failure (that's not
quite what I ended up doing, but the intend was basically the same
thing). It allowed me to get a memory dump *at the time the failure
was detected* (and *without* cleanup), analyze it in WinDbg, and send
the dump to the native SDK OEM for further analysis.
My scenario is not common. But I would think the desire of being able
to stop running a group of unit tests upon failure, and even better
programmatically, would be something applicable to "integration" tests
in general, where it is not as easy to get deterministic behavior as
in one stand alone method.
Anyhow, that's the context. My hack does pretty well for now, but I
just thought some other people may like to have such feature.
Nick
Greg Young
to get taken as opposed to pausing the test run in a teardown?
> --
> You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
>
>
--
Le doute n'est pas une condition agréable, mais la certitude est absurde.
Greg Young
Take a look at ADPlus configs here
http://msdn.microsoft.com/en-us/magazine/cc163833.aspx
Would this be a better way of handling this use case?
nr4christ
-- the code under tests resides in an out of process COM server, so I
send a message to it from TearDown so that it can call DebugBreak().
And since that does stop every thread, it does stop logging and does
the trick.
When I originally requested this feature (as a GUI-only setting) I was
thinking more of logs and not of memory dumps. I wanted to stop the
logging at the time of the failure. So maybe I muddied the waters by
talking about memory dumps.
And it is doable today -- just put a sleep when the TestContext says
the test failed.
> Take a look at ADPlus configs herehttp://msdn.microsoft.com/en-us/magazine/cc163833.aspx
> Would this be a better way of handling this use case?
>
>
>
>
>
>
>
>
>
> On Wed, Jul 27, 2011 at 10:52 AM, Greg Young <gregoryyou...@gmail.com> wrote:
> > Aren't there easier ways of doing this, like just triggering the dump
> > to get taken as opposed to pausing the test run in a teardown?
>
Charlie Poole
not actually help you. Any facility to stop the remaining tests would
have to run all the pending teardowns first, else their wouldn't be a
proper shutdown.
Charlie
Nicolas Rivera
I do want to stop when a unit test fails, and in this case, I don't even want to run the TearDown upon failure (I don't want to cleanup). You are correct that in my original request the TearDown would have been executed for the unit test that failed, so it wouldn't have been as ideal. But given that my original request was in the context of analyzing a log, it still would have been better stopping after TearDown upon failure, than not stopping at all. It is difficult to analyze a verbose log and try to find in it where the test failed, when 20 other tests have already been executed after the failure.
I am aware there are workarounds. I wrote some of them in the link you posted. And I can come up with other ways to make my task easier.
I don't know what makes a feature worth having in your mind, hence I am not sure I can be of much more help to you. It depends on what you think the level of effort should be for a developer to use NUnit (should a developer be looking for workarounds for these type of requests?), who are your target users (TDD users or a wider group?), how difficult would it be to implement the feature, what is your vision for NUnit, etc.
Thanks,
Nick
Charlie Poole
On Wed, Jul 27, 2011 at 9:39 PM, Nicolas Rivera <nick4...@hotmail.com> wrote:
> Not sure I follow what you mean. I am not running tests in parallel, thus
> there is only one pending TearDown at a time (at least from my point of
> view, I don't know how the tool is implemented).
Let's say you have a test case that fails. There are potentially multiple
TearDown methods that need to be run in the same class or in its base
classes. In addition, the class and base classes could have multiple
TestFixtureTearDowns and any of the enclosing namespaces might
have SetUpFixtures with TearDowns.
That doesn't even account for pending After actions needing to be completed
due to action attributes in NUnit 2.6.
For a clean shutdown, they must all run.
> I do want to stop when a unit test fails, and in this case, I don't even
> want to run the TearDown upon failure (I don't want to cleanup). You are
> correct that in my original request the TearDown would have been executed
> for the unit test that failed, so it wouldn't have been as ideal. But given
> that my original request was in the context of analyzing a log, it still
> would have been better stopping after TearDown upon failure, than not
> stopping at all. It is difficult to analyze a verbose log and try to find
> in it where the test failed, when 20 other tests have already been executed
> after the failure.
Perhaps we are using the word "stop" differently. I'm taking it to me "end the
test run gracefully, performing any needed cleanup" which is what it has
meant in some other pending requests. I see that you want to freeze
the action in a way that allows you to debug things. Perhaps we need
another word to communicate that.
>
> I am aware there are workarounds. I wrote some of them in the link you
> posted. And I can come up with other ways to make my task easier.
>
> I don't know what makes a feature worth having in your mind, hence I am not
> sure I can be of much more help to you. It depends on what you think the
> level of effort should be for a developer to use NUnit (should a developer
> be looking for workarounds for these type of requests?), who are your target
> users (TDD users or a wider group?), how difficult would it be to implement
> the feature, what is your vision for NUnit, etc.
It's more a question of what the feature needs to do and what might be
the best way for the program to express it. We need a clear way to talk
about what it is first off: "stop" was confusing me at any rate. Secondly,
do we want to have NUnit initiate dumps, etc. for you or merely give
you a way to do those things manually by pausing before terminating
the app.
Charlie
> Thanks,
>
> Nick
>
> On 7/27/2011 9:22 PM, Charlie Poole wrote:
>
> So really, with regard to your original request to _stop_ the gui, that
> would
> not actually help you. Any facility to stop the remaining tests would
> have to run all the pending teardowns first, else their wouldn't be a
> proper shutdown.
>
> Charlie
>
Nicolas Rivera
> It's more a question of what the feature needs to do and what might be
> the best way for the program to express it. We need a clear way to
> talk about what it is first off: "stop" was confusing me at any rate.
> Secondly, do we want to have NUnit initiate dumps, etc. for you or
> merely give you a way to do those things manually by pausing before
> terminating the app.
Personally I wasn't thinking of NUnit initiating dumps, etc. It seems
out of scope. At any rate, I wasn't expecting it. What I was expecting
was for the tool to make it easy for me to do those things if I wanted
to -- but not to actually do them for me.
Being a developer, what I would find most flexible/extensible, would be
if the tool provided a programmatic way to abort/terminate the execution
of a batch of tests at any point. That, plus good support of
TestContext (which allows one to know if the test actually failed),
would probably be enough to deal with just about any similar type of
scenario without having to come up with custom workarounds -- and
without having to go back to you with another variation of the same request.
Just my two cents for what is worth.
Thanks,
Nick
Charlie Poole
What if we had a method like 'Wait' or 'Pause'?
I do something like this myself when I need to debug nunit-agent.exe. I insert
a MessageBox call that says "Attach to nunit-agent and press OK"
A more general implementation of this might work like this...
1. Test or Setup or Teardown issues TestContext.Wait("message"). A variant
would be to use Assert.Wait("message");
2. NUnit uses it's normal event/progress channel to send a Wait event.
3. The runner takes action on the wait.
3.1 For the Gui a message box is used
3.2 For the console runner pause message is used
Of course, you would have to avoid using Wait() if running unattended.
We could give nunit-console an option to indicate that no pausing is
allowed and it would then be ignored.
Another possibility is to allow various action choices like
* Continue the run
* Stop the run after this test
* Stop immediately without any further cleanup.
But that could be added later.
How does that sound?
Charlie
Nicolas Rivera
Sounds pretty good to me.
I also like your idea of having a way to tell the console or GUI whether
pausing is allowed or not. Right now I am using the existence of a file
with a particular name to accomplish the same thing. But if the tool
itself had that setting, it would be one less workaround (albeit a
rather trivial one).
Thanks,
Nick
Charlie Poole
Charlie
On Thu, Jul 28, 2011 at 11:28 AM, Nicolas Rivera