SpecFlow/BDD vs Unit Testing

1,363 views
Skip to first unread message

timhardy

unread,
Feb 26, 2010, 2:36:31 PM2/26/10
to SpecFlow
I'm relatively new to TDD, and I'm having trouble understanding
exaclty where Unit Testing and BDD/AcceptanceTesting/SpecFlow meet.

I love the idea of writing textual feature descriptions, but these
will most likely be at a higher abstraction level than most unit
tests. They will probably be written by a business analyst or product
owner. They just want to describe what the feature should do without
caring about the details. These will almost always be integration
tests, won't they? They will probably need to hit the db to test for
things like "A Product is created" or something like that.

Unit tests, on the other hand, are very detailed and deeper than a
product owner or user story acceptance test needs to go. They also
cannot cross module boundaries and therefore shouldn't hit the
database, for instance.

What I'd like to know is what the workflow looks like for you guys in
when you write unit tests and when you simply satisfy the acceptance
tests in your feature files.

I see it going one of three ways:
- Either you guys write very specific acceptance tests and these
textual tests ARE your unit tests (not real product owner friendly,
but could maybe work if a developer expanded upon what the owner
wrote)
- All your acceptance tests are basically integration tests, and
that's as granular as your testing gets (seems like you'd lose
something important from TDD here)
- You write granular unit tests that help you accomplish or move
towards your acceptance tests, which primarily act as a guide (sounds
the best, but I'd love to see an example because I've seen zero :))

Any help or examples you can provide would be great. For a newcomer
to TDD and BDD, it's difficult to see where they all fit together in
the land of unit and integration tests.

Thanks,
Tim

sztupi

unread,
Feb 26, 2010, 3:47:11 PM2/26/10
to SpecFlow
hi,

On Feb 26, 8:36 pm, timhardy <hardy....@gmail.com> wrote:
> I love the idea of writing textual feature descriptions, but these
> will most likely be at a higher abstraction level than most unit
> tests.  They will probably be written by a business analyst or product

yes, specflow is to be used on the 'analysis' level (although IMHO the
technical personnel should be part of the analysis, so it'll be
probably forged out together), not on the unit test level.

> owner.  They just want to describe what the feature should do without
> caring about the details.  These will almost always be integration
> tests, won't they?  They will probably need to hit the db to test for
> things like "A Product is created" or something like that.

It depends on what do you call integration tests. I'd say anything
that tests more components working together (even if there's no
'external' system like a DB is involved) is an integration test, so in
this sense, the SF test will definitely be integration tests in almost
all of the cases.
As for whether you'll really touch the DB or just mock it; use the
real UI frontend through a browser or only testing the MVC controllers
- I'd say it's up to how you implement the bindings and of course
should depend your application. On larger projects you may have
multiple strategies for different kinds of features: to use the DB for
a reporting feature that's mostly implemented in custom SQL queries;
to automate the browser testing you're new fancy UI with AJAX/Comet;
and maybe keep away from both for the features that are about business
calculations implemented fully in the middle layer.

> Unit tests, on the other hand, are very detailed and deeper than a
> product owner or user story acceptance test needs to go.  They also
> cannot cross module boundaries and therefore shouldn't hit the
> database, for instance.

a unit test, at least how I see it, should not cross _any_ boundaries,
only the interface of the particular unit, otherwise it's more than a
unit you're testing.

> What I'd like to know is what the workflow looks like for you guys in
> when you write unit tests and when you simply satisfy the acceptance
> tests in your feature files.
>
> I see it going one of three ways:
>     - Either you guys write very specific acceptance tests and these
> textual tests ARE your unit tests (not real product owner friendly,
> but could maybe work if a developer expanded upon what the owner
> wrote)
>     - All your acceptance tests are basically integration tests, and
> that's as granular as your testing gets (seems like you'd lose
> something important from TDD here)
>     - You write granular unit tests that help you accomplish or move
> towards your acceptance tests, which primarily act as a guide (sounds
> the best, but I'd love to see an example because I've seen zero :))

Well, I'd say it's the 4th approach I try to follow, basically the
opposite of you're last one:
as the features (the 'what') we are going to implement are - in the
ideal case :) - defined when thinking about the code (the 'how')
itself can start, so can the features be written in specflow.
then writing the code can start (and writing the appropriate unit test
should be inseparable of that, personally I recommend doing it really
the test-first way), with the aim to fill in the 'gaps' of the
features (the bindings to make the SF tests be 'really' red, then
their implementation to make them green). Originally we imagined this
to be almost always outside-in (so starting from the uppermost layer
the binding connects to and going down to the other layers until all
the tests are fulfilled), although in practice it's not always like
that (whether it's because of lack of discipline, lack of experience
or just because oversimplified theories rarely work in reality, I
don't have an opinion yet).

> Any help or examples you can provide would be great.  For a newcomer
> to TDD and BDD, it's difficult to see where they all fit together in
> the land of unit and integration tests.
>
> Thanks,
> Tim

br,
sztupi

Jonas Bandi

unread,
Feb 27, 2010, 11:24:20 AM2/27/10
to spec...@googlegroups.com
Hi Tim,

as you mention that you are relatively new to BDD I would like to point you to some other resources about the topic.
Remember SpecFlow is heavily inspired by Cucumber (www.cukes.info). The Cucumber community is much bigger and more mature than the SpecFlow community (and any .Net-BDD community). So we can learn a lot from their insights and experiences.

The Cucumber Google Group offers great resources about the questions you asked:
http://groups.google.com/group/cukes

On my wiki I am collecting general BDD resources. Especially among the presentations there are some resources that might be very interesting for necomers to BDD:
http://jonasbandi.net/wiki/index.php/BDD_resources

Especially the RSepc Book is treating your questions:
http://www.pragprog.com/titles/achbd/the-rspec-book

I would also point you to the concept of "outside-in development":

It takes the concepts of TDD to a higher level. In TDD unit-tests are more about "design" and "driving the coding" than about "test".
In BDD (automated) scenarios are more are more about "working out and communicating specifications" and "driving the development" than about "test".

Unit-tests are developer artifacts that care about the "how" and are concerned with producing good code.
BDD-Scenarios are shared artifacts between stakeholders, developers and testers that care about the "what" and are concerned with building a shared understanding and a usable system.

So in the ideal world we have two nested red-green cycles:
You start with writing a scenario (outer cycle)
This scenario fails.
You drop down to the inner cycle and start writing unit-tests that drive the implementation. Always with the goal to make the scenario pass.
Those unit-tests fail.
You build code to make the unit tests-pass.
You repeat the inner cycle by adding more unit-tests. You do that until the scenario from the outer cyle passes.
You continue on the outer cycle by writing a new scenario.

See the following picture from the RSpec book:
http://stove.s3.amazonaws.com/techdoer.com/images/bdd_answer_to_testers_dilemma/rspec_cumber_outside_in.jpg

Or the following presentation:
http://www.infoq.com/presentations/tdd-ten-years-later

Have I seen that in practice? Not to the full extent yet, but we are working on it ...

Regards
jonas
--
mail: jonas...@gmail.com
web: www.jonasbandi.net
blog: blog.jonasbandi.net
twitter: twitter.com/jbandi

bcw

unread,
Apr 15, 2010, 11:10:53 PM4/15/10
to SpecFlow
Hi,

On Feb 27, 7:36 am, timhardy <hardy....@gmail.com> wrote:
> I'm relatively new to TDD, and I'm having trouble understanding
> exaclty where Unit Testing and BDD/AcceptanceTesting/SpecFlow meet.

I'm new to BDD also, but I'll answer in light of my process
experience.

There are always two designs in computer programs - the product
design, and the implementation design.

The product design is the user experience I want to cause. "These
sounds will occur under these circumstances, and these calculations
will be done, and this information will be saved, and it will be in
this color, etc".. It doesn't matter whether I cause this to occur by
writing a program in Java or use Magic Pixies to do it - the user gets
a certain experience.

Specifying this would be the realm of BDD specifications. BDD 'tests'
- automations actually - are a bonus - they are robot guidelines
always ready to tell me how close I've come to fufilling those
specifications at any time. The automations are a convenience.

The implementation design is how I go about telling a computer to
fulfill the user expectations I decided on in the product design.
Here is where I decide between using Java and Magic Pixies, and
arrange for the details to happen.

Unit Testing is a coder's tool. In Test First Development, the unit
tests provide guides to how much of the implementation design I have
fulfilled, and the unit automations are robots I leave behind to warn
me if I stray from my earlier implementation solutions ("You broke the
tests!").

Unit Tests guarantee implementation decisions. BDD guarantees user/
design expectations, quite distinct from how they are provided.

If a technical requirement comes from outside ("the customer requires
we use this Oracle API"), then that's legitimately part of the BDD
(but could be specified otherwise). In this case, it's part of the
user (customer) experience - they insisted, we have no leeway, and
can't decide on a different implemetation design. It's a product
specification, not an implementation decision - it's exposed to the
outside world.

Cheers,
bcw


--
Subscription settings: http://groups.google.com/group/specflow/subscribe?hl=en
Reply all
Reply to author
Forward
0 new messages