GoSpec - a BDD framework

[Esko Luontola]

I have started writing a BDD framework for Go, called GoSpec:
http://github.com/orfjackal/gospec

Right now I have done some experiments about the syntax and though
about the execution model. Here are the project goals:

- Organize the tests/specs in BDD style. Although usually two levels
of nesting is enough for that, GoSpec supports unlimitedly nested
specs.

- Write the names of the specs as strings. You are not restricted to
using only those characters that are allowed in method names.

- Write the specs using a fluent API, so that the code is be easily
readable.

- Execute the specs concurrently on multiple CPU cores. Running the
specs quickly (i.e. less than 10-20 seconds) is a must for using TDD,
so being able to take advantage of all processing power is important.

A draft of the GoSpec syntax and an explanation of its execution model
is at http://github.com/orfjackal/gospec/blob/master/src/experiment.go#L34

(The specs/tests will be executed similarly to this proof of concept
that I did some time ago in Scala: http://gist.github.com/123025 - I
wrote it with Specs in mind and Specs 1.6.0 incorporated this
execution model.)

I haven't yet thought about the exact syntax for assertions. Maybe
something like c.Then("expected").ShouldEqual(actual); and there will
be assertions for collections, similar to what JDave has.

Feedback is welcome!

I see some others have also started writing BDD frameworks, so maybe
we can together make something useful. Personally I'm not familiar
with Makefiles (I come from Java background), so maybe some friendly
person would help in setting up the project, so that building and
running all tests can be done with one command.

[Esko Luontola]

Status update:

Right now GoSpec's execution model is ready. This means that it
executes specs multi-threadedly in isolation.

So the hardest part has now been implemented, but there are still some
things that are needed before GoSpec is ready for use. Here is the
current TODO list:

- Assertions. Syntax is still open. Maybe something like: c.Then
(x).Should.Be(y); c.Then(a).ShouldNot.Be(b); c.Then
(list).Should.Contain(z);

- Printing a report of the executed and failing specs.

- A tool which finds and runs the tests automatically, similar to
gotest. But GoSpec can be used already before that, if you write some
lines of boilerplate code (not much).

[Esko Luontola]

One more status update, possibly the last one before GoSpec is ready
for use.

I have now added some basic assertions (Equal, Be, Contain). Here is
some example code of how they can be used:

value := 42
c.Then(value).Should.Equal(42)
c.Then(value).Should.Be(value > 40)

values := [...]string{"one", "two", "three"}
c.Then(values).Should.Contain("two")
c.Then(values).ShouldNot.Contain("four")

"Equal" works for all primitives which support ==, and for all objects
with an Equals(interface{}) method. "Contain" works for arrays, slices
and iterators. I'll add more containment assertions later. I'll also
try adding support for pointer equality (either as part of "Equal", or
then I'll rename "Be" to "Satisfy" and use "Be" for pointer equality).

I have also added a distinction of "Should" and "Must". In the above
code samples, if the word "Must" is used instead of "Should", then it
will stop the execution of the specs (not immediately, but it will
skip child specs/examples). This can be used in complex test setup
code to verify that after the setup the system under test is in a
known state. If a "Must" fails, then GoSpec will not even try
executing the rest of the test. This makes the test results easier to
read, because a failure in the setup will not be reported multiple
times.

Printing of test run results is under development. I'll add line
numbers to the error messages and improve the formatting of the test
results. That will be the last thing to do before GoSpec is ready for
use.

[Russ Cox]

> value := 42
> c.Then(value).Should.Equal(42)
> c.Then(value).Should.Be(value > 40)

I am not familiar with BDD.
Can someone explain the rationale to me?

I don't see any advantage of this over

assert(value == 42)
assert(value > 40)

(I'm not advocating asserts in tests; the original
presumably has the same problems that the asserts do.)

In fact, the BDD version strikes me as much harder
to read and write, since it requires learning a second
language. And it's more fragile: value appears twice
in the value > 40 line, and there's nothing keeping them
in sync.

What's wrong with using ordinary Go expressions and
code in tests? What compels the creation of a parallel,
verbose language?

Russ

[Bob Cunningham]

I had similar questions, and after following the supplied link and
searching for more, I realized BDD was something I've needed for ages:

Executable Requirements.

The translation of Use Cases and Standards to Requirements then to Tests
has, in my own experience, been fraught with error, largely due to
misinterpretation or miscommunication. We rely on many iterative
reviews and inspections to ferret out such errors, but the process has
gaps. BDD is *just* readable enough to be understood by non-programmer
shareholders.

I have a nasty horror story where executable requirements would very
likely have prevented the slight misreading of one sentence in an FAA
standard from causing a suite of flight instruments to nearly ship with
a major flaw, despite many careful reviews, independent inspections, an
exhaustive FAA certification process, many test flights, and countless
hours of simulation.

The flaw happened when the requirement was read from the standard and
slightly reworded when being entered into DOORS by a systems architect
who was also a requirements expert. The standards experts inspected the
DOORS entry several times, and saw no error. The test author wrote code
to match the DOORS entry. But the standards experts never reviewed the
test code: They weren't expected to comprehend it, so they could only
verify that the requirement was present as a comment in the test code.

Had our standards experts and other domain experts been able to read and
understand our test code, they would have had a very good chance of
catching the error. But our tests were written in C++, not exactly
known for its inherent readability (even by seasoned programmers).

I believe programmers will much more easily learn another simple
language, than a non-programmer domain expert can be taught to correctly
read and fully comprehend tests written in Go (or C++).

BDD is not a cure-all, but it is a huge step in the right direction, at
least for application domains where such visibility is necessary,
allowing a key aspect of product development (program testing) to be
intelligently examined by non-programmers.

I love this list! Tons of brain food here. Lots of Kool-Aid and
Bike-Sheds too!

-BobC

[SnakE]

2010/1/3 Russ Cox <r...@golang.org>

> value := 42
> c.Then(value).Should.Equal(42)
> c.Then(value).Should.Be(value > 40)

I am not familiar with BDD.
Can someone explain the rationale to me?

I don't see any advantage of this over

   assert(value == 42)
   assert(value > 40)

Plain asserts won't tell you what the value actually was when the test failed.  This alone warrants existence of assertEquals() and friends.

Also, while the BDD code looks like simple sequential checks, internally every single test is converted into complete initialize-test-teardown sequences which guarantee correct execution of all tests even if previous tests failed.

What's wrong with using ordinary Go expressions and
code in tests?  What compels the creation of a parallel,
verbose language?

You'll end up writing some sort of testing framework anyway.  You actually wrote gotest already.

[Esko Luontola]

> Plain asserts won't tell you what the value actually was when the test
> failed.  This alone warrants existence of assertEquals() and friends.

Also, BDD frameworks typically use such a syntax that the order of
parameters is easy to remember. With assertEquals() a common mistake
is to put the parameters in the wrong order: is assertEquals(expected,
actual) or assertEquals(actual, expected) the right order?

And then there is the philosophical side of BDD, that TDD is not
really about testing. BDD replaces the words "test" and "assert" with
words such as "spec" and "should". This helps in getting the
programmer in the right frame of mind, especially when the programmer
is new to TDD. Some links to read:

http://dannorth.net/introducing-bdd
http://techblog.daveastels.com/2005/07/05/a-new-look-at-test-driven-development/
http://techblog.daveastels.com/files/BDD_Intro.pdf
http://behaviour-driven.org/Introduction
http://www.infoq.com/presentations/bdd-dan-north
http://www.infoq.com/interviews/Dave-Astels-and-Steven-Baker

> What's wrong with using ordinary Go expressions and
> code in tests?  What compels the creation of a parallel,
> verbose language?

Even though it requires typing some more, typing had never been the
bottleneck in writing code. In my case most of the time when writing a
test goes into choosing an expressive name for the test. In other
words, thinking about what is the small piece of behaviour that should
be implemented next, and how to describe that behaviour explicitly and
detailedly in just a few words.

Other reasons are documented in GoSpec's README (http://github.com/
orfjackal/gospec#readme), under "Project Goals". For me, being able to
nest the specs has the highest priority, because that allows me to be
very detailed in writing the specs (for my preferred style, see
http://github.com/orfjackal/tdd-tetris-tutorial/tree/beyond/src/test/java/tetris/).
Closely related, on about as high priority, is the isolation of the
specs. Without isolation, organizing the specs would be harder, they
would be buggier, and would require more duplicated code. GoSpec is
about 1000 lines of production code, out of which maybe one third
exists because of how it isolates the side effects of the specs.

> You'll end up writing some sort of testing framework anyway.  You actually
> wrote gotest already.

Gotest is useful in bootstrapping other testing frameworks. :)

[Esko Luontola]

> In fact, the BDD version strikes me as much harder
> to read and write, since it requires learning a second
> language.

Every API is a language that needs to be learned.

> And it's more fragile: value appears twice
> in the value > 40 line, and there's nothing keeping them
> in sync.

Typing the value twice is needed so that the framework would produce
better error messages: print the state of the actual object when the
expression is false. Given Go's syntax, that was the best I could come
up with. The "c.Then(actual).Should.Be(bool)" syntax is for
situations, where the simple equality comparison is not enough, and
for checking boolean properties. For example in
http://github.com/orfjackal/gospec/blob/gospec-1.0.0/examples/stack_test.go
the line "c.Then(stack).Should.Be(stack.Empty())" reads somewhat
fluently "stack should be empty" when you skip reading the noise
words. Also http://github.com/orfjackal/gospec/blob/gospec-1.0.0/examples/assert_examples_test.go
has the example "c.Then(s).Should.Be(mediumSized)" about how you can
use local variables to give names to problem domain concepts.

It's not true that "there's nothing keeping them in sync". The tests
and production code double-check each other, similar to double-entry
bookkeeping. If there is a bug in either of them, then it is often
found out by being in mismatch with the other. And in TDD, every test
is checked by running them in the "Red" state (http://
agileinaflash.blogspot.com/2009/02/red-green-refactor.html), after
which we make them pass and get to "Green". Seeing the test to fail is
an important check to make sure that the test is written correctly, as
is seeing the test to pass right after writing the necessary code to
pass it.

[Samuel Tesla]

Another good resource on BDD is Dave Astels' Google talk:

http://video.google.com/videoplay?docid=8135690990081075324#

-- Samuel