My thoughts on specification by example for OTS products

[Josh Male]

Hello,

I've blogged about automated testing on customised OTS products:

http://www.joshmale.com/blog/2013/02/05/automated-testing-for-an-ots-product/

I think that specification by example is the best way to avoid vendor lockin.

Thanks

Josh

[Rob Park]

Good post. 

I'm not seeing the "Spec by Example" part though other than by name. To me, Selenium != "Spec by Example". I'm not saying you're not doing "Spec by Example" ... just that I'd love to see some "examples" :-)

I like the idea of you having some acceptance tests at the GUI level to validate (smoke test) their release.  But I'd also suggest for you to consider putting another layer in the middle of your triangle for direct XML integration tests. Not necessarily to avoid duplicating their tests, but to isolate problems. That is if I'm using the XML interface directly, I'd want to know that my expectations of that interface have or have not changed.

#rob

[Michael Leung]

Hi Josh,

Your title “Automated Acceptance Testing for an OTS Product” has attracted me as not many people talk about applying Specification by Examples (SBE) to OTS customizations. I can only find a few related posts even thru Googling. 

I am only a beginner exploring this topic thru trial and error. Please correct me if I am wrong. I agree with your post but would like to discuss the issue of duplicating test with your vendor. To me, SBE is not primarily about testing but rather, about requirements specification. It is great if your vendor will eventually ships you the full set of SBE, which lets you understand the requirements as well as accept the application by executing them. Otherwise, you still need the requirements in SBE format for upcoming customization and future maintenance work. After all, code without test is Legacy code, according to Michael Feathers. 

In my case, my ERP vendor won’t ship SBE to us and you understand that re-creating user story level SBE for an ERP is out of question because automated SBE is usually costing more than the application itself. 

Hence in my blog post, I suggest moving up one level and apply SBE in manually-executable Business Process Scenarios. It is not in the conventional Gherkin “Given, When, Then” syntax but in form of Business Process Flowchart and step-by-step UAT plan format. Nevertheless, they are still executable (although manually) requirement specifications of business processes (although not user stories). This is also an effective way to avoid vendor lock-in. 

I plan to use these Business Processes and step-by-step execution plan to discuss customization with users. Following TDD, I will first change the processes, steps and make the test “red”. The test will turn “Green” again after finishing the customization. 

Of course, the customization itself is another story. We can apply agile practices, BDD and user story level SBE like any other bespoke-built solutions.

Michael Leung