Some things I don't like about SpecFlow

[Kevin]

Hi everyone,

three things I wanted to point out about SpecFlow (which let me say, I
have been trying yesterday for the first time seriously, and there a
couple of things that are really neat... some others I am still not
convinced about):

* The extreme verbosity in how SpecFlow reports the outcome of the
tests, either in the NUnit console or in the Output window console
when using TestDriven.NET. Sometimes it becomes a bit difficult to
realize what the problem is when a test fails.

* The way SpecFlow suggests to group the specs for a class.. all
in one single "Step" file. I believe that the Step file can grow very
quickly, and it may become a bit difficult to detect which test failed
or where.
I think I prefer the way MSpec suggests to organize the files: one
file per scenario, so you would have when_user_edits_profile.cs,
when_user_logs_in.cs ...etc

* The text that needs to be written in the specs it is also quite
verbose. Usually, in a "Then" I had to write no less than 3 lines just
to do one Assert. I don't quite like the variables being sent from
here to there within the collection in the Scenario.Context.
Maybe there are some helpers out there somewhere that could
alleviate that a bit.. although I don't think it is possible, because
of the way SpecFlow is built, you always need to store/recover the
variables from the ScenarioContext.

Look at a very simple When and Then I wrote, how much text they
contain... am I doing it the right way?
(disregard the 'technical' language used in my specs rather than
being more user-driven specs... I was just trying at the smallest unit
level, just to give it a try)


[When(@"I create an apartment")]
public void WhenICreateAnApartment()
{
Address address =
ScenarioContext.Current.Get<Address>();
int floor = (int)ScenarioContext.Current["floor"];
string apt = (string)ScenarioContext.Current["apt"];

Apartment apartment = new Apartment(address, floor,
apt);
ScenarioContext.Current.Set<Apartment>(apartment);
}

[Then(@"it should set the address")]
public void ThenItShouldSetTheAddress()
{
Address address =
ScenarioContext.Current.Get<Address>();
Apartment apartment =
ScenarioContext.Current.Get<Apartment>();

address.ShouldEqual(apartment.Address);
}


* Lastly (I promise), is there some helpers to handle exceptions
more elegantly? I know that since SpecFlow is for more user-oriented
specifications one shouldn't talk about exceptions, right?
But what if we want to test lower level functionality (ie: unit
tests)? Or do you suggest not to use SpecFlow for that purpose?
In unit level tests, you may have to write some expectations for
Exceptions for sure.
These are 2 very simple helpers I wrote, but maybe there is a
better way:

public static class Expect
{
public static void Exception(Action throwingAction)
{
Exception("Exception", throwingAction);
}

public static void Exception(string id, Action
throwingAction)
{
try
{
throwingAction();
}
catch (Exception ex)
{
ScenarioContext.Current.Set(ex, id);
}
}
}

public static class Catch
{
public static Exception Exception()
{
Exception ex;


ScenarioContext.Current.TryGetValue<Exception>("Exception", out ex);

if (ex != null)
{
return
ScenarioContext.Current.Get<Exception>("Exception") as Exception;
}
else
{
return null;
}
}
}

[Jimmy Li]

Hi Kevin,

I am using SpecFlow for unit specs. I may have two answers for the
problems you described.


1. The way I try to shorted the text I have to type in each Given,
When, Then is by creating a container that contains short hands for
all the objects (mostly domain objects) that are used in the specs.
E.g. so that the example you gave would become

[When(@"I create an apartment")]
public void WhenICreateAnApartment()
{

Sc.Apartment = new Apartment(Sc.Address, Sc.Floor,
Sc.ApartmentNumber);

}

[Then(@"it should set the address")]
public void ThenItShouldSetTheAddress()
{

Sc.Address.ShouldEqual(Sc.Apartment.Address);
}

// Here is the shorthands for the current ScenarioContext to make the
above work
public static class Sc
{
public static Address Address
{
get { return (Address)ScenarioContext.Current["Address"]; }
set { ScenarioContext.Current["Address"] = value; }
}

public static Apartment Apartment
{
get { return (Apartment)ScenarioContext.Current["Apartment"]; }
set { ScenarioContext.Current["Apartment"] = value; }
}

public static int Floor
{
get { return (int)ScenarioContext.Current["Floor"]; }
set { ScenarioContext.Current["Floor"] = value; }
}

public static string ApartmentNumber
{
get { return
(Apartment)ScenarioContext.Current["ApartmentNumber"]; }
set { ScenarioContext.Current["ApartmentNumber"] = value; }
}

public static Exception Exception
{
get { return (Exception)ScenarioContext.Current["Exception"]; }
set { ScenarioContext.Current["Exception"] = value; }
}
}

Note that I name the shorthands as though they are unique in the whole
system (mostly using the ubiquitous language) to try to avoid
conflicts.


2. For exception, I use the Catch.Exception from MSpec.
[When(@"I create an apartment with invalid address")]
public void WhenICreateAnApartment()
{
Sc.Exception = Catch.Exception(() => Sc.Apartment = new
Apartment(Sc.Address, Sc.Floor, Sc.ApartmentNumber););
}

and I have steps that verify different exceptions. E.g.:
public virtual void
ThenShouldGetAPreconditionExceptionWithMessage(string message)
{
var preconditionException = Sc.Exception;
preconditionException.ShouldBeOfType(typeof(PreconditionException));
preconditionException.Message.ShouldEqual(message);
}

Hope this help. Please share if anyone have a better way. :)

[Kevin Krac]

I'm not convinced by your implementation, sorry.

With your approach, for each business class you have in you application you need to have an equivalent helper class.
You alleviate the verbosity on one side (tests), but add it on another (the helper classes)

A more generic implementation to store/retrieve objects to and from the ScenarioContext would be better.

Maybe it's not possible to reduce those 3 or 4 lines in excess after all. Perhaps that's the price we have to pay by using a scenario context in opposition to, for example MSpec, which declares the variables globally in the scope of the scenario.

[Jonas Bandi]

There are four different options for sharing state between steps:

1. Instance variables
2. static members
3. ScenarioContext
4. ContextInjection

Explanations:

1. Only possible for sharing state between steps implemented in the
same class hierarchy

2. static members are very pragmatic and in this case not so evil as
we as developers might first think (there is no threading or need
formocking/replacing in step-definitions)

3. You already discussed that ...

4. If the constructor of a step definition class needs arguments,
Specflow tries to inject these arguments. This can be used to inject
the same context into several step-definitions. See an example here:
http://github.com/techtalk/SpecFlow/tree/master/Tests/FeatureTests/ContextInjection/

--
mail: jonas...@gmail.com
web: www.jonasbandi.net
blog: blog.jonasbandi.net
twitter: twitter.com/jbandi

[Kevin Krac]

Thanks Jonas, will look more in detail option 4.

As for Option 2 (static members)... so you're saying that there is no problem with different scenarios sharing static variables in their When(s) or Then(s)?

[Jimmy Li]

"With your approach, for each business class you have in you
application you
need to have an equivalent helper class."

I only have one Sc (shorthand) class which contains one property per
each domain class. The Sc class is used to share state between all
unit specs. So, Sc.Apartment, Sc.Address, etc will be used by other
unit specs that interacts with Apartment, Address, etc...

MSpec was good. I used it for about a year. But I like SpecFlow better
after using it for about a month because I can write scenarios in
plain English in the feature file... save more brain processing cycle
to translate from code to meaning... The only thing missing is that it
is missing a way to link the scenarios (unit specs of domain classes)
to the domain classes' documentation.

[Kevin Krac]

@jonas

you said that option 2 (static members) is a good option.
But if you use a static member how do you take advantage of reusability through regular expressions?

I mean... if you use static members, you must declare them in a global class so that they can be accessed from within your Steps definitions AND from other files/classes where you might have defined reusable specs (ie: ThenItShouldDisplayTheMessage() )

On Tue, Oct 19, 2010 at 1:33 PM, Jonas Bandi <jonas...@gmail.com> wrote:

[Kevin Krac]

I don't think option 2 (static members) will work well.

This is a video I found that explains why: http://bit.ly/cbMcDg


In short, you may have the "Then" defined in a different class as the one for your Steps (to take advantage of reusability via regular expression) and whatever variable you defined locally in your Steps file will not be accessible from the "Then" file...

So you either have to:

1.- Define that shared variable in a sort of global class accessible from both, the Steps file and the Then file.
Or
2.- Put the Then clause within the Steps file... not taking advantage of reusability, and potentially doing a bad design of your tests, by putting everything in a single file.


That said, I think that static variables won't work, as instance variables won't work either. And since I don't like ScenarioContext verbosity.... that leaves me with the ContextInjection option.

Not the very best option I would have chosen since you still have to write additional code (the contexts for each of the scenarios) but not very bad.


Suggestions?

[Kevin Krac]

@Jimmy: ah, I thought you had one "shorthand" class per each class under test. So you're doing a kind of context yourself, but typed.


Still not convinced by any of the approaches. I would like to be able to use variables and not have to store them in a context.

[Jonas Bandi]

I was mainly thinking about "Patterns" like Singletons that use global
static variables...
These can be accessed by all Step definitions ...