Improving Given/When/Then for BDD style acceptance testing
97 views
Skip to first unread message
Graham Allan
Mar 28, 2014, 11:10:50 AM3/28/14
to scalate...@googlegroups.com
Following on a discussion from here[0], where Bill and I get into what ScalaTest provides for BDD-style acceptance testing. Just skip the blog post and go straight to the comments if you don't want to get sucked in to a different debate :)
I think it is a constraint, and I have seen some odd phrasing in places because a method used to make a sentence in one test case is not quite the right name to use in another place. I have no doubt that using Strings will provide the most readable format, at the expense of encouraging reuse and keeping in sync with the test code. Mostly it's not been a concern, and the amount of structure is Just Enough. Since specific types with methods are passed to Given/When/Then methods, it makes it easier to compose test steps together.
I agree. I don't know if, after that initial writing, QA or product managers would be able to modify the spec. It's too fragile to allow without being able to run the test case and fix errors. Using Narrative, we collaborate for new tests and existing tests, but there's always developer-time necessary to make sure it can compile and execute correctly. Sometimes we end the collaboration with non-compiling code, and the developer works without QA/product to make the same text compile, and fill in any stubs.
Since it seems unpractical to expect QA/product to maintain executable test cases without a developer, I'd prefer to use the tools I'm used to, which have added benefits. Would like to hear the experience of anyone using e.g. Cucumber, who have non-devs modify existing test cases.
Yes, that would be cool. Under Narrative we basically have to look at source code. We had an unfinished attempt to parse the source and write out the test case, stripping out all the language syntax. It never really got to a point where it was functional and well integrated, but it's a kind of validation that it would be useful to read tests with all the code stripped away.
So, I think that the linked gist is an improvement over existing Given/When/Then in that it doesn't intersperse the business-readable spec with low-level test code, but it goes in the opposite direction from what I would be keen to see. That is, moving the business-readable spec *as* the test code, as a 'facade' over the low-level stuff, rather than as a mapping from the (albeit brilliantly descriptive) strings to test methods. I *think* this is what encouraged a library of reusable, discoverable test components that you can reuse for new test cases.
Yes, this is something we've observed, and remains a real concern. It's showed up in several different approaches that use single text to code mapping, not just ScalaTest's Given/When/Then, but also specs2's immutable style. Perhaps we are just undisciplined, but I think it's a problem that's widely reported in reference to code comments, and I still see this as analagous. Of course it's still possible for implementation to differ from declaration in code, but I feel it's less likely to have a method name that's inconsistent with the code it contains.
Btw, we also created a bare-bones port of our Java Narrative to Scala[1]. There's next to no JavaDoc, and the examples[2] don't take advantage of some Scala syntax features that could help, like omitting parens, but the background code (to perform actions, extract results) can be much cleaner in Scala. In Java, the test code that QA/product to see is condensed as much as possible, but there's a lot of boilerplate in the rest of the iceberg below that surface.
Do you feel that allowing BDD/acceptance test approach is something that ScalaTest has to support out of the box? When we debated internally what framework to use, ScalaTest made a perfectly capable "engine" for 3 different BDD frameworks[3]. Perhaps it's something that you could just delegate?
Kind regards,
Graham A
[0] http://grundlefleck.github.io/2013/06/23/using-scala-will-make-you-less-productive.html#comment-1301326465
[1] https://github.com/youdevise/narrative-scala
[2] https://github.com/youdevise/narrative-scala/tree/master/examples/src/test/scala/calculator
[3] Cucumber, JBehave and our own Narrative-scala
> I am curious to what extent people may end up feeling constrained by the available syntax
I think it is a constraint, and I have seen some odd phrasing in places because a method used to make a sentence in one test case is not quite the right name to use in another place. I have no doubt that using Strings will provide the most readable format, at the expense of encouraging reuse and keeping in sync with the test code. Mostly it's not been a concern, and the amount of structure is Just Enough. Since specific types with methods are passed to Given/When/Then methods, it makes it easier to compose test steps together.
> Cucumber tries to do this by having separate files for specification text and test code (the "steps"). But I find that this separates the text and test code a bit too much
I agree. I don't know if, after that initial writing, QA or product managers would be able to modify the spec. It's too fragile to allow without being able to run the test case and fix errors. Using Narrative, we collaborate for new tests and existing tests, but there's always developer-time necessary to make sure it can compile and execute correctly. Sometimes we end the collaboration with non-compiling code, and the developer works without QA/product to make the same text compile, and fill in any stubs.
Since it seems unpractical to expect QA/product to maintain executable test cases without a developer, I'd prefer to use the tools I'm used to, which have added benefits. Would like to hear the experience of anyone using e.g. Cucumber, who have non-devs modify existing test cases.
> I would like to be able to allow people to just see the structure of the specification without actually running the test code
Yes, that would be cool. Under Narrative we basically have to look at source code. We had an unfinished attempt to parse the source and write out the test case, stripping out all the language syntax. It never really got to a point where it was functional and well integrated, but it's a kind of validation that it would be useful to read tests with all the code stripped away.
> Both of those have been at the back of my mind and yesterday after reading your reply above I came up with a new possible way to do this in ScalaTest.
So, I think that the linked gist is an improvement over existing Given/When/Then in that it doesn't intersperse the business-readable spec with low-level test code, but it goes in the opposite direction from what I would be keen to see. That is, moving the business-readable spec *as* the test code, as a 'facade' over the low-level stuff, rather than as a mapping from the (albeit brilliantly descriptive) strings to test methods. I *think* this is what encouraged a library of reusable, discoverable test components that you can reuse for new test cases.
> I think that won't address all of your concerns. The specification text is still independent from the test code and can get out of sync.
Yes, this is something we've observed, and remains a real concern. It's showed up in several different approaches that use single text to code mapping, not just ScalaTest's Given/When/Then, but also specs2's immutable style. Perhaps we are just undisciplined, but I think it's a problem that's widely reported in reference to code comments, and I still see this as analagous. Of course it's still possible for implementation to differ from declaration in code, but I feel it's less likely to have a method name that's inconsistent with the code it contains.
Btw, we also created a bare-bones port of our Java Narrative to Scala[1]. There's next to no JavaDoc, and the examples[2] don't take advantage of some Scala syntax features that could help, like omitting parens, but the background code (to perform actions, extract results) can be much cleaner in Scala. In Java, the test code that QA/product to see is condensed as much as possible, but there's a lot of boilerplate in the rest of the iceberg below that surface.
Do you feel that allowing BDD/acceptance test approach is something that ScalaTest has to support out of the box? When we debated internally what framework to use, ScalaTest made a perfectly capable "engine" for 3 different BDD frameworks[3]. Perhaps it's something that you could just delegate?
Kind regards,
Graham A
[0] http://grundlefleck.github.io/2013/06/23/using-scala-will-make-you-less-productive.html#comment-1301326465
[1] https://github.com/youdevise/narrative-scala
[2] https://github.com/youdevise/narrative-scala/tree/master/examples/src/test/scala/calculator
[3] Cucumber, JBehave and our own Narrative-scala
Reply all
Reply to author
Forward
0 new messages