Complex feature-level setup

[Kevin Dente]

I'm evaluating using SpecFlow as a possible alternative for functional
testing that we've traditionally done entirely with testing platforms
like SilkTest and TestComplete. Perhaps this might be a bit of a
stretch beyond SpecFlow's primary purpose, but the idea of writing
test specs in a human readable DSL like Gherkin is very appealing. The
product being tested is a fairly complex legacy enterprise software
product, which makes things a little trickier to test than a
greenfield app.

In particular, I'm concerned that we may to do some fairly complex
setup steps for each feature - configuration that take enough time
that running them for each scenario probably isn't feasible. I'm
envisioning this sort of like Background type step definitions, but
one that would run once for the feature (or for a group of features)
rather than before each scenario.

Is such a thing possible?

[rakesh loki]

You can do this in specflow either by creating SpecFlowEventDefination like BeforeFeature/AfterFeature or by using a Background per feature which runs once per feature

If you try adding a new SpecFlowEventDefinition file in Visual Studio then you can see list of Event definitions supported by SpecFlow.

If you would like to create a background per feature then you can find an example here... http://github.com/techtalk/SpecFlow-Examples

[Xerxes Battiwalla]

You can have a background step, but that only makes your gherkin more readable. When implementing the step, you would need to use a [BeforeFeature] or [BeforeScenario] to do your set-up just once.

In fact we do exactly this, and keep a static instance of hardware devices we need to spin up for test purposes. If it was to spin the hardware device each test, our suite would take far too long. Upon test-run, ([BeforeTestRun]) we do our initialisation and cache the instance.

-xerx

[Jonas Bandi]

And just for completion:

EventHooks [ie. BeforeScenario] can be bound to tags, so that they
only get executed when the tag is present on the respective element
(i.e. scenario).

Example from SpecFlow-Examples:
http://github.com/techtalk/SpecFlow-Examples/tree/master/ASP.NET-MVC/BookShop/

feature:

@web
Scenario: Title should be matched
When I perform a simple search on 'Domain'
Then the book list should exactly contain book 'Domain Driven Design'

feature:

[BeforeScenario("web")]
        public static void BeforeWebScenario()
        {
            StartSelenium();
        }

[AfterScenario("web")]
        public static void AfterWebScenario()
        {
            if (ReuseWebSession)
                StopSelenium();
            Assert.AreEqual("",
ScenarioContext.Current.SeleniumErrors().ToString(), "Selenium
verification errors");
        }

--
mail: jonas...@gmail.com
web: www.jonasbandi.net
blog: blog.jonasbandi.net
twitter: twitter.com/jbandi

[Kevin Dente]

Sorry, are you saying there's an example that shows Background-per-
feature? Because I'm not seeing anything like that. Can you point me
in the right direction?

On Jun 4, 12:18 am, rakesh loki <rreddy.l...@gmail.com> wrote:
> You can do this in specflow either by creating SpecFlowEventDefination like
> BeforeFeature/AfterFeature or by using a Background per feature which runs
> once per feature
>
> If you try adding a new SpecFlowEventDefinition file in Visual Studio then
> you can see list of Event definitions supported by SpecFlow.
>
> If you would like to create a background per feature then you can find an

> example here...http://github.com/techtalk/SpecFlow-Examples

[Kevin Dente]

Thanks for all the feedback, folks. Just to be clear, ideally I'd like
to be able to express the feature-level background spec *in the
gherkin*. Unless I'm missing something, the event based stuff as shown
in the samples requires the setup be written entirely in code. This
makes sense for things like external hardware configuration and
database prep, but again I'd like to be able specify the
configuration through the feature description.

From my reading it appears Gherkin itself doesn't really have a way to
express this - background is assumed to be run for each scenario.
Really I want both - feature-level background and scenario-level
background.

Any other thoughts on a reasonable way to express this? Would
extending SpecFlow to support this make sense? I'd be willing to take
a shot at if it people though it would be useful.

[Ricardo Stuven Garrido]

What I miss in Gherkin is a way to define a sort of "Feature Outline", ie. running all scenarios using a sequence of common parameters. Something like:

Feature Outline:

Background:

    Given I have opened the application using <browser>

# Scenario 1...

# Scenario 2...

Feature Examples:
| browser      |
| firefox      |
| iexplore     |
| googlechrome |

Regards,
Ricardo Stuven.

[Jonas Bandi]

Hi Kevin,

a possible approach to your problem (possible not satisfaying for you but I still want to mention it):

You can tag your features like this:

@needs_extensive_setup @needs_special_setup
Feature: My feature
  In order to run a feature requiring a complex setup
  As a stakeholder
  I want to signal the required setup with a tag
 
Scenario: Relying of a common feature setup
  Given bla bla
  When bli bli bli
  Then blu blu blu

The tags are kind of a business readable specification of the required background.

Then in the step-definitions you can implement hooks that execute the required setup:

        [BeforeFeature("needs_extensive_setup")]
        public static void Hook()
        {
            int test = 42;
        } 

        [BeforeFeature("needs_special_setup")]
        public static void Hook2()
        {
            int test = 42;
        }

But of course this has probably not the expressiveness you like ...

We made quite some effort to be compatible with Gherkin. We should not leave that path to lightheaded.
In my opinion we should first move over to the Cucumber community and ask how they solve your problem, respectively their opinion.
Then we should also ask them what they think of extending Gherkin in this direction and also the suggestion of Ricardo.

I imagine that extending the syntax for SpecFlow without changing Gherkin is currently a bit messy, the cleanest way would be to incorporate any changes into Gherkin itself, but that needs some skills beyond .NET :-)
The other possibility would be to provide another parser. This shouldn't be too hard since the current parser has quite a clear interface ...

[Kevin Dente]

You're right, not quite what I'm looking for.

I appreciate the tight compatibility with standard Gherkin, and I
agree that's important. Simply extending SpecFlow unilaterally and
diverging from Cucumber probably doesn't make sense.

Another idea that a friend had is to overload Scenarios for extended
setup stuff. I'm going to evaluate that option and see if it's
workable. I'm envisioning something like:

Feature: My feature
  In order to run a feature requiring a complex setup
  As a stakeholder
  I want to signal the required setup with a tag

@feature_setup
Scenario: Do some complicated setup
  Given bla bla

I welcome any thoughts on this approach.

> mail: jonas.ba...@gmail.com

[Gáspár Nagy]

Hi,

I was playing a little bit with this idea. This is the code that I
finally created, maybe it gives you some other ideas for such tricks.
(Sorry for pasting longer code parts.)

Basically I have created a BeforeFeature event handler, that tries to
find the scenarios in the generated unit test class that are marked
with the "feature_setup" tag and executes that. It will also ignore
automatically the scenario if it is executed directly and not through
the feature setup. This all implemented in the following class, that
you can include to your project (please note, that this works
currently only with NUnit):

[Binding]
public class SpecialFeatureSetupTools
{
private const string FEATURE_SETUP_TAG = "feature_setup";

[BeforeScenario(FEATURE_SETUP_TAG)]
public static void IgnoreNormalExecution()
{
// ignore the scenario if it was executed through the
"normal" execution
bool value;
if
(FeatureContext.Current.TryGetValue("BeforeFeatureExecution", out
value) && value)
return;

Assert.Ignore("ignoring normal execution of the feature
setup");
}

[BeforeFeature]
public static void BeforeFeature()
{
var fixtureType = GetFixtureType();
if (fixtureType == null)
return;

object fixtureInstance = null;
var testRunner = TestRunnerManager.GetTestRunner();

FeatureContext.Current["BeforeFeatureExecution"] = true;

foreach (var featureSetupMethod in
fixtureType.GetMethods().Where(mi => IsTaggedWith(mi,
FEATURE_SETUP_TAG)))
{
if (fixtureInstance == null)
{
fixtureInstance =
Activator.CreateInstance(fixtureType);
}

featureSetupMethod.Invoke(fixtureInstance, null);
testRunner.OnScenarioEnd();
}

FeatureContext.Current["BeforeFeatureExecution"] = false;
}

private static Type GetFixtureType()
{
var stackTrace = new StackTrace(false).GetFrames();
if (stackTrace == null)
return null;

for (int frame = 0; frame < stackTrace.Length; frame++)
{
if (stackTrace[frame].GetMethod().Name ==
"OnFeatureStart")
{
return stackTrace[frame +
1].GetMethod().DeclaringType;
}
}

return null;
}

private static bool IsTaggedWith(MethodInfo mi, string
tagName)
{
var attributes = mi.GetCustomAttributes(typeof
(CategoryAttribute), false);

return attributes != null &&
attributes.Cast<CategoryAttribute>().Any(a => a.Name == tagName);
}
}

For testing, I have created a feature file like this:

Feature: Complex Feature Setup

In order to run a feature requiring a complex setup
As a stakeholder
I want to signal the required setup with a tag

@feature_setup
Scenario: Do some complicated setup

Given the complicated setup has executed 0 times
When I do some complicated setup

Scenario: A scenario requires complicated setup
Given the complicated setup has executed 1 times

Scenario: Another scenario requires complicated setup
Given the complicated setup has executed 1 times


and some steps for the test...

[Binding]
public class SpecialFeatureSetupSteps
{
[Given(@"the complicated setup has executed (\d+) times")]
public void GivenTheComplicatedSetupHasExecutedTimes(int
times)
{
int counter = GetCounter();
Assert.AreEqual(times, counter);
}

[When(@"I do some complicated setup")]
public void WhenIDoSomeComplicatedSetup()
{
int counter = GetCounter();
Console.WriteLine("doing some complex setup");
FeatureContext.Current["complex_setup_counter"] = +
+counter;
}

private int GetCounter()
{
int counter;
if (!
FeatureContext.Current.TryGetValue("complex_setup_counter", out
counter))
counter = 0;
return counter;

[Kevin Dente]

Awesome! I'll give it a try.