Expecting exceptions from another step

[Brandon Bonds]

I cannot find a way to do this, so unless I missed something, consider
this a suggestion.

I occasionally need to write code like this:

[When(@"...")]
public void When___()
{
try
{
// do something...
}
catch (Exception ex)
{
ScenarioContext.Current.Set(ex);
}
}

And then this:

[Then(@"an exception should be thrown with the message ""(.
+)""")]
public void ThenAnExceptionShouldBeThrownWithTheMessage(string
message)
{
Assert.AreEqual(message,
ScenarioContext.Current.Get<Exception>().Message);
}

Since SpecFlow is already catching any exception that is thrown, it
would be nice if I did not need to write the try/catch block every
time. And instead, my 'Then' method could look more like this:

[HandlesScenarioTestError]
[Then(@"an exception should be thrown with the message ""(.
+)""")]
public void ThenAnExceptionShouldBeThrownWithTheMessage(string
message)
{
Assert.AreEqual(message,
ScenarioContext.Current.TestError.Message);
}

If this is possible already, please let me know. Thanks!

(I am writing an API, so yes, I actually want to throw exceptions
instead of displaying friendly error messages to the user... in my
case, the user is another developer.)

[Darren Cauthon]

Hi Brandon,

I kinda like how you're doing it already. :)

My two main concerns with what I think you're asking are:

1.) The try/catch portion of your code should probably be specific to
the exact portion of your code that needs to check for the exception,
and nothing more. This means that any setup, including instantiating
the object that will be tested, needs to be outside of the try catch.

For example, let's say that I'm testing an MVC application, and I want
to write a test around an action call. My current step would look
like:

[When("I go to the home page")]
public void x()
{
var dependencyResolver =
ScenarioContext.Current.Get<IDependencyResolver>();
var controller =
dependencyResolver.GetService(typeof(HomeController));
try {
var result = controller.Index();
ScenarioContext.Current.Set(result);
} catch (Exception ex)
{
ScenarioContext.Current.Set(ex);
}
}

Putting an attribute around the entire step definition would behave
differently in that the dependency resolver retrieval and the
controller instantiation will be wrapped in what was meant to be
exception handling *for the application.* If an exception happens in
one of those two things, I need SpecFlow to fail the spec due to the
fact that either my SpecFlow setup or my entire application is not
functioning -- not because of a "Then" clause.

I know, it's not a problem that's probably going to affect anything.
If I'm writing good steps, the exceptions returned will be different
than the ones I'm testing for and the spec will fail anyway. But
still, I think it's crossing a dangerous line. What's even worse is,
crossing that line will then become baked into the framework. By
adding a property to retrieve this one exception and a special
attribute, it would almost be like a suggested practice in SpecFlow.

2.) I think the attribute hides important meaning. Whenever someone
new starts on my projects, one of my first suggestions to them is to
put a debugger on the first step in the spec and to watch how my words
match with my code. Even people who are new to SpecFlow are usually
able to pick it up pretty quickly, since the only SpecFlow-specific
thing they have to know about is ScenarioContext.... and everybody
understands what a string-object dictionary is.

I know it doesn't seem as elegant, but if another developer sees your
try/catch statement they're going to know instantly how your words
match with your code -- regardless whether they've read the SpecFlow
documentation.

--

I don't mean to shoot down an idea. Perhaps there's another way?
Maybe like passing a lambda statement to a method that wraps the
Call() in a try catch and does what you want? Then it could be
something like: HelperThingy.SaveTheExceptionFromThisCall(() =>
controller.Index()) ? I don't know, it's all I can think of right
now.


Darren Cauthon

[Darren Cauthon]

I do need to make this clear: I am *NOT* saying anybody should wrap
their MVC actions in a try/catch like that. It was just an
example. :)

[Brandon Bonds]

I see your point. I sort of like what you have near the bottom, but
by that point it's almost the same amount of typing to use try/catch
blocks... and it is easier to comprehend what the try/catch is
actually doing. I attempted to use an extension method to do the same
thing, but alas, C# will not allow you to invoke an extension method
on a lambda. :)

It's a hard call. It would seem useful for SpecFlow itself to be able
to judge whether an exception fails a test, or if it gets handled in a
"Then" clause. Using my original code, if I needed to use the same
When phrase in another scenario that does not expect an exception to
be thrown, the try/catch would eat any exception and the scenario
could finish green. I guess I could specify in each scenario a new
Then clause that ensures that no exception was thrown, but I'm not
sure that's the best way to handle it either considering that SpecFlow
handles that case normally.

Thanks for the idea... I'll consider it. Any other suggestions are
welcome too.

Brandon Bonds

On May 21, 2:22 pm, Darren Cauthon <darrencaut...@gmail.com> wrote:

[Gáspár Nagy]

Interestingly I had a (mail) discussion with someone else having the
same idea. I'll re-post here the relevant parts.

There I came to the same conclusion as Darren, but for partly
different reasons (while I also agree with Darren's notes as well):
1. I fear that such extension would increase the confusion about
whether SpecFlow should be used for unit testing or functional
testing. Currently we try to describe it as a functional testing tool
(inside the BDD story of course), because that’s the area where the
benefits of business readability more clearly overweight the
additional effort of the loose (free-text, regex, etc.) bindings. Many
SpecFlow users have realized that it can be also used for testing
technical interfaces (external APIs, web services, units, etc.), but
this is not yet a mainstream and I’m not sure that it is always
efficient. So I would first try find a good story, why this feature
would be also helpful for functional testing, before adding it to the
product.
2. The general workflow of swallowing / propagating and handling the
exceptions is quite complex and error prone in general.
E.g. if a “then” step is not prepared to run after a failing when
(some inconsistent state, for instance), in case of a real error (not
an expected one), the original error will be hidden by the error in
the unprepared “then” step. Also handling the case when there are
multiple “when” steps is hard as you would not be able to distinguish
in the assertion step, which when’s exception you want to assert.
I a scope of a concrete project it is easy to make rules to avoid this
confusion (only one “when”, etc.), but for making it as a generic
feature, I think this is too complex.

My conclusion was that this is something what SpecFlow can support
with allowing good extension points, but the concrete implementation
should not be part of the project (at least now).

I also did a little bit of experiments for "solve" this problem in
SpecFlow 1.6.1. I have two possible solutions.

In the first solution you have to use a “that results in an exception”
postfix for the steps in order to catch the exception. You can see a
working sample here: http://dl.dropbox.com/u/51289/BlogAttachments/AssertException.zip.
I had to apply some tricks to work and also the config setting
“<runtime stopAtFirstError="true" />” was necessary. It also collects
the unhandled errors and re-throws them at the end of the scenario
(see the second scenario in the feature file). The benefit of this
solution that you can decide about the exception handling per scenario
and not in the step binding itself (so you can safely use it in normal
and expected-exception scenarios as well). The disadvantage is, that
it makes the "go to binding" unusable, as it will always point to the
generic step definition.

The other solution is more closer to what you have imagined. Basically
this is a unit-test provider plugin, that replaces the specflow test
runner to a custom one (AssertExceptionTestRunner), where you can do
the exception manipulation as necessary. You can download this one too
from here: http://dl.dropbox.com/u/51289/BlogAttachments/AssertExceptionPlugin.zip.
To have this working, you need to do one extra steps. The “generator”
part of the customization (AssertExceptionPlugin.Generator.dll) has to
be placed into the “C:\Program Files (x86)\TechTalk\SpecFlow” folder
(or wherever specflow VS integration was installed to) and the
generator has to be configured in the app.config of the test project.
<specFlow>
<unitTestProvider name="NUnit"
generatorProvider="AssertExceptionPlugin.Generator.AssertExceptionTestGenerator,
AssertExceptionPlugin.Generator" />
</specFlow>
You also need stopAtFirstError="true" for this solution.

You should have a look at the AssertExceptionTestRunner class, where
there is an commented-out part, that limits this behavior only for
scenarios having a special tag. Also I have now separated the
generator/runtime part of the plugin into two different assemblies to
show more clearly what is used where. Since this plugin is quite
simple and it does not have any additional dependencies, these can be
safely merged into one for better maintainability.

These solutions work with SpecFlow 1.6.1. They are a little bit of
hacks, so there is a chance that you need to alter them for future
versions.

Have fun!

Br,
Gaspar

[Brandon Bonds]

Those solutions are interesting. Thanks for taking the time to come
up with them and to implement examples.

I'm most interested in what you commented out that allows limiting the
behavior to specific tags.

Thanks!
Brandon

> > > >            ...
>
> read more »