SpecFlow

[jcmeyrignac]

Hi !

I'm currently searching for the ideal BDD framework for starting a new
project (the project will start next week).

I tried:
- Fitnesse.Net (but there is no BDD layer in .NET)
- Cucumber (but the process of running tests does not seem easy)
- MSpec (but this is too oriented towards the developers)

And I just discovered your great tool: SpecFlow, which corresponds
exactly to our needs (.Net, VS integrated, nUnit).

Since we are french, I think we'll use a version with french wording.
Also, we are using Selenium since a long time, so I think I'll work on
adding Selenium into this framework, for UI BDD.

I'll keep you informed.

JC

[Gáspár Nagy]

Hi,

Maybe you have realized in the meanwhile we don't have French support
yet, but English only. We are working on the language support, so we
will have that soon. As the first step, we were thinking on a non-
extensible fast solution for a couple of selected languages (was
German planned, but we could have French too), and only later
implement a fully extensible solution. This could be ready even next
week, so you don't have to wait too much.

We also have a project that uses Selenium together with SpecFlow, so
it should work. Maybe I'll post some snippets how we did the
integration.

Br,
Gaspar

[jcmeyrignac]

On Nov 5, 2:16 pm, Gáspár Nagy <gaspar.n...@gmail.com> wrote:
>
> Maybe you have realized in the meanwhile we don't have French support
> yet, but English only. We are working on the language support, so we
> will have that soon. As the first step, we were thinking on a non-
> extensible fast solution for a couple of selected languages (was
> German planned, but we could have French too), and only later
> implement a fully extensible solution. This could be ready even next
> week, so you don't have to wait too much.
>

That would be great, but it's not so urgent.
I'll do a presentation about SpecFlow to my colleagues tomorrow, so
that they start using it at the first iteration of our product (which
will start at the end of the next week).
The Bowling is not a realistic example, so I'm trying to build a demo
about our business logic (it's an application to manage human
contacts).

> We also have a project that uses Selenium together with SpecFlow, so
> it should work. Maybe I'll post some snippets how we did the
> integration.
>

I'm still not sure if I should expose a Selenium interface in
SpecFlow.

I want that our tests take only a few seconds.

Over the last years, we built 51 test suites (composed of a total of
more than 550 cases), and running them took more than 5 hours.
I'd like to keep these tests separate from the UnitTests (TDD & BDD).

In my opinion, UI testing is not too much related to BDD.

Also, I'm not sure if TDD's unit tests are useful when you use BDD
(except for coverage).
TDD and BDD push towards two different directions, and I'd like to
reduce the amount of constraints on our programmers.

From your experience, what tests are really important ?
And what frameworks do you recommend ? (we need to query a database,
if possible with an ORM).

JC

[chassa]

Hi,

I am not sure about on which level Dan North originally intended BDD.
However, nowadays, I see the concept of BDD applied on the level of
units (=TDD) as well as on the level of features (user stories, things
that have tangible value to business = ATDD).

- BDD on TDD level helps developers to drive their technical design,
by letting them phrase behavior expectations against the unit under
development.
- BDD on ATDD level helps Product Owners to drive features, by letting
them phrase acceptance criteria expectations against the feature (user
story) under development.

While the distinction might be blurred, and a given tool might be
usable for both levels, I think there are categories of tools that are
more feasible for one or the other level (TDD or ATTD).

Tools that use plain text descriptions and/or tables of values to be
filled in by business (tester, product owner), which are then bound
somehow to the actual implementation, are IMHO more suitable for the
ATDD level. Tools of that kind are Cucumber, SpecFlow, Fit/Fitnesse
(there are surely more of that kind, just to name a few). If I
understand Martin Fowler right, he categorizes languages of these
tools as "external DSLs". I think they are more suitable to ATDD
level, because the language can be better customized to be understood
(and potentially also written) by business. And the primary goal of
ATDD is to facilitate the trust and communication between business and
development team (including testers, if you split those roles), where
this understandability helps.

Tools that use a programming language for specifying the expectations,
are IMHO more suitable for the TDD level. Tools I would count into
this category are RSpec, NBehave or MSpec (just to name a few). We
have also build a tool for such a BDD for TDD approach, which we plan
to add to the SpecFlow project later, but which is completely
independent of the current SpecFlow Cucumber style approach (see:
http://gasparnagy.blogspot.com/2009/10/devcamp09-behavior-driven-development.html).
Martin Fowler categorizes languages of these tools as "internal DSLs".
I think they are more suitable to TDD, as the primary goal of TDD is
to drive the design of the developer and secure and document the
expectations a developer made against a given unit. And a developer
usually has no problem to read those languages. Applying BDD to TDD
raises the abstraction level just a tiny bit, so that the developer
can better focus on the expected behavior of a given unit.


Having said this, we see the primary purpose of SpecFlow to be used
for specifying feature behaviors (user story acceptance criteria)
through examples, that are then bound with step definitions to gain
automated acceptance tests - in the sense of Acceptance Test Driven
Development and Agile Specifications (specification by example).

You can vary the layer on which you do the binding (step definitions):
directly to the user interface (e.g. through Selenium or other UI
automation tools) or only at the controller of the UI, and this has
impact on efforts and complexity for binding. However, the test then
goes through all the layers of the system under development (including
the database) to verify the specified behavior of a feature (user
story).

We use SpecFlow integrated with our Scrum development process: for
each acceptance criteria of story specified in sprint planning, we
implement one or more SpecFlow scenarios. Regarding performance, we
select subsets of tests to be run continuously by developers vs. all
tests that run in a nightly verification build (controlled through
attributes on specflow features). We also use an O/RM and switch to in-
memory database (SQL-Lite) for speeding up the continuously run tests,
and the nightly build can again use the real database.

I have the impression you currently plan to use SpecFlow on the BDD
for TDD level, and I am not sure how feasible it is for that purpose.
As I mentioned, we rather use an internal DSL to apply BDD for TDD in
our projects, and we use this concept in parallel to SpecFlow.

For doing BDD on ATDD, however, I think SpecFlow perfectly fits.
Compared to pure user interface automation scripts (like Selenium),
SpecFlow scripts are (ideally) ambiguous to UI details, as the
specifics of the UI are expressed in the binding (step definition).
This brings business readability (and potentially business can also
write/extend/change scripts) and improves maintainability. However,
the benefit is not only having test automation. SpecFlow scripts add
value already before implementation, as they are part of the agile
specification process and facilitate the communication between
business and development.

Greetings,
Christian

[jcmeyrignac]

Thank you very much for your in-depth reply !

It's really interesting, and answers some of my own questions.

I thought that the BDD process could replace the TDD part, but now, I
see them as complementary.

About the UI part, I read somewhere that validating UI can only be
done by a human, and it makes sense.

Until recently, I have been a developer, but a few weeks ago, I was
moved to the QA part (as the dev support, since I wrote all the
Selenium tools here).

Due to our previous poorly testable architecture, we could only add
very high level tests, mostly running with Selenium.

Our new shift to BDD should speed up a lot the process of coding/
testing.

As a QA member, I'll concentrate on working on anti-regression tests
with Selenium, which is why I don't think BDD should be allowed to
test the interface.
Anyway, I'll be glad to see how you added some Selenium into Specflow.

A last point of concern for me is the lack of documentation about
SpecFlow.
There are only a few set of examples, so I can prepare a smalll
presentation, but I don't see how we could build large sets of BDD.

Here are some questions:

- how can we separate the scenarios ? In TDD, I can use a Setup for
every scenario, are all the setups placed in the
SpecFlowStepDefinition ?
- what's the use of @mytag ?
- what is a SpecFlowEventDefinition ?

BTW, I very much liked the why/how pyramid from Gaspar's document (I
shamelessly used it for presenting BDD to our product managers ;-)).

JC

> independent of the current SpecFlow Cucumber style approach (see:http://gasparnagy.blogspot.com/2009/10/devcamp09-behavior-driven-deve...).

[jcmeyrignac]

Ok, by doing some research, I answered most of my questions:

>  - how can we separate the scenarios ? In TDD, I can use a Setup for
> every scenario, are all the setups placed in the
> SpecFlowStepDefinition ?

It seems the tag mechanism is here to solve this problem, along with
the Event Definitions.
However, I fear that when you have a massive set of behaviors (let's
speak about thousands), the event definitions will get very large, and
it will be difficult to maintain.
Is there a mechanism to make these things lightweight and easy to
maintain ?
I'll check how Cucumber solves this problem.

>  - what's the use of @mytag ?
>  - what is a SpecFlowEventDefinition ?

Answered above.
I like the fact that adding a @tag can define a way to setup classes
of behaviors.
For example:
@logged
Scenario: ...
allows to start a scenario with a connected user.

JC

[chassa]

> I thought that the BDD process could replace the TDD part, but now, I
> see them as complementary.

Yes, IMHO BDD can be used to extend TDD as well as ATDD.

> About the UI part, I read somewhere that validating UI can only be
> done by a human, and it makes sense.

I agree. Some things you cannot validate by automated tests, like
usability feedback. It is also hard to write automated tests to
validate usability requirements - at least I have not seen this in
practice. Some projects automatically generate screenshots, which are
later validated by a human.

Anyway, the main purpose I see for automated acceptance tests with
Specflow is to automatically confirm acceptance criteria of a feature
(user story). Either including the UI, or some level below (not
testing defined acceptance criteria for the UI). Note that these kind
of tests are usually strictly structured after the Triple-A principle:
Arrange - Act - Assert. "Arrange" sets up all necessary state in the
app (and mocks of excluded systems), "Act" performs the isolated test
on the specific acceptance criteria to be validated, and "Assert"
performs all kinds of assertions for validating the "Act" was
successful. In "Arrange" and "Assert" you do short cuts, like setting
application state through the system API (e.g. setting login state
instead of automating login through the UI) or querying database state
for an assert (instead of navigating to a UI where you would see the
results). The "Act", however, should be isolated for the acceptance
criteria to cover and go through the number of layers you agreed to
test in your acceptance tests. Shortcuts in "Arrange" and "Assert" and
strict isolation of the "Act" for a single acceptance criteria
improves maintainability of the tests and can help with performance.

This is much in contrast to other kinds of UI automation tests, where
previous "Acts" are the "Arrange" for later "Acts" - so you have
basically a series of "Arrange" - "Act" - "Assert" - "Act" - "Assert"
- "Act"..... to test a whole user workflow through the application.

Having covered all your acceptance criteria with isolated SpecFlow
tests does not fully replace this kind of workflow testing. In
addition to doing automated SpecFlow acceptance tests, we suggest to
define typical user workflows through the application, which can be
also used to demo to the user (e.g. in the sprint review) or to
collect usability feedbacks. Having automated acceptance tests,
however, significanlty reduces the number of required workflow tests.
Whether you automate those workflow tests as well is another decision,
they are definitely more brittle to maintain and you always want to
assert them against a human user anyway :-).

> Anyway, I'll be glad to see how you added some Selenium into Specflow.

Integration with automated test runnners like Selenium, Webrat (or
others) works the same like in Cucumber.

> A last point of concern for me is the lack of documentation about
> SpecFlow.
> There are only a few set of examples, so I can prepare a smalll
> presentation, but I don't see how we could build large sets of BDD.

We strive to be fully compatible with Cucumber (on the Gherkin level)
- which means you can apply knowledge of the concept in both worlds
and you can also use examples and documentation of both worlds. There
is plenty if documentation for Cucumber, and it might be one of the
most interesting areas to discuss for us with the Cucumber team, how
we can share/extend their efforts in documentation and examples.

>  - how can we separate the scenarios ? In TDD, I can use a Setup for
> every scenario, are all the setups placed in the
> SpecFlowStepDefinition ?

Setups, which I described previously as "Arrange" should be put in
"Given" parts. For every setup you require, you should write another
given fragment. You can chain up multiple givens with "and", and you
can also reuse given fragments in different scenarios.
Same is true for the "Act" which corresponds to "When" in Gherkin.
And the "Assert" corresponds to "Then".

What basically happens is that you build up a DSL for your project,
which can be used to express behavioural expectations against the
system (in terms of Arrange, Act and Assert).

Another interesting aspect is, that with this DSL, the testers can use
the Arrange and Assert parts also for manual testing, to setup initial
state in the system or Assert for things that are not visible on the
UI during the manual test. The DSL is basically also a "testing
interface" to the system.

>  - what's the use of @mytag ?

As you figured out already, you can tag scenarios for all kinds of
stuff. We generate unit test categories out of it (so you can
selectively run certain scenarios in your automated build) and you can
also filter for specifically tagged scenarios in extension events.

>  - what is a SpecFlowEventDefinition ?

The events are used to extend step definitions - in Cucumber these are
called "Hooks". It allows to extend step definitions or whole
scenarios for (groups of scenarios) with specific cross-cutting
aspects. Like starting a new selenium session for scenarios which are
marked with @seleniumtag.

> BTW, I very much liked the why/how pyramid from Gaspar's document (I
> shamelessly used it for presenting BDD to our product managers ;-)).

Thanks, Gaspar liked it also and shamelessly used it from one of my
talks about Requirements Management :-).

Greetings,
Christian

[Dan Ryan]

Hi Christian,

I am interested to know why you think SpecFlow is not suitable for BDD
at the unit level? The screencast on the website already shows how
you can use SF in a red-green-refactor cycle. My view is that
particularly in teams that are struggling with the design aspect of
TDD or unit testing in general, defining the behaviour first separate
to hooking up the tests could aid understanding.

The only issue I have currently with using SpecFlow at the unit level
is how to organise the tests. Currently SpecFlow scans for steps
across all files and doesn't allow steps with the same name. This
promotes non granular steps files (I read that the recommendation is a
steps file per problem domain). I think this is fine at the ATDD but
doesn't work at the unit level.

Thanks,

Dan

[Jean-Charles Meyrignac]

Since Christian didn't reply, I'll try to give my own explanation:

TDD is more suited for how you write the code, and how the code should work.
BDD is more suited for why you write the code, and how the application should behave.

BDD is for all the team, and it helps define a common vocabulary (thus the need to match a ton of strings in the code).
TDD is for programmers, and tools like MSpec and XUnit should be more suitable for them.

In my opinion, writing TDD with SpecFlow is just a giant waste of time.

JC

[Gáspár Nagy]

Hi,

I think I agree with your conclusion. You can use SpecFlow for TDD,
but it is probably not efficient enough.

On the other hand I think the ideas of BDD can be very well applied
also for the unit testing (just not in the form, how cucumber/specflow
does it). Maybe you have seen, that we already have a mini-framework,
that we use for BDD-style unit testing. You can read about it (and
download) here: http://gasparnagy.blogspot.com/2009/10/devcamp09-behavior-driven-development.html.

Don't be confused, this library also uses the "SpecFlow" namespace,
because we wanted to join this two thinks somehow. But these are two
different frameworks, probably we will call this other one as
"SpecFlow for unit testing" or something like that.

Br,
Gaspar

On Nov 19, 10:18 am, Jean-Charles Meyrignac <jcmeyrig...@gmail.com>
wrote:

> Since Christian didn't reply, I'll try to give my own explanation:
>

> TDD is more suited for *how* you write the code, and how the code should *
> work*.
> BDD is more suited for *why* you write the code, and how the application
> should *behave*.

>
> BDD is for all the team, and it helps define a common vocabulary (thus the
> need to match a ton of strings in the code).
> TDD is for programmers, and tools like MSpec and XUnit should be more
> suitable for them.
>
> In my opinion, writing TDD with SpecFlow is just a giant waste of time.
>
> JC
>