Manual testing of gherkin-specified features

[Chris Whatley]

No, not a troll...

I'm in a situation where I have the luxury of product owners that have bought into expressing their needs via gherkin, a solid dev team who is making great progress with test automation, AND availability of manual testers that are effectively zero-cost to me for now.

There are certain aspects of what we are doing that are either not easily automatable or simply don't seem worth the trouble at this time. In our work, aesthetics are extremely important and that's a big challenge. Reviewing reports with screenshots doesn't cover everything we care about either.

Rather than insisting on 100% automated coverage of the features, I am going to use manual testing to close the gap and track the body of manually tested features as technical debt.

Does anyone have any experience with this approach? Any suggestions on tooling and reporting?

[Andy Waite]

This is a valid approach which we have used. I blogged about it a
while ago:

http://andywaite.com/2011/4/24/designing-cucumber-scenarios-for-manual-verification

The act of simply writing specs in Gherkin is hugely beneficial, even
if you never automate them, because it triggers the discussions and
deep analysis to determine what the correct behaviour should be in
tricky scenarios.

You can tag features or individual scenarios with @manual - that will
allow you to create a report listing all the manual tests required for
a release, and also to track the number of manual tests over time.

Andy

[Matt Wynne]

Yeah, +1

Using the odd manual step where you have something that's not cost effective to automate is a good idea, especially while you're getting up to speed with test automation.

Don't forget that Cucumber ships with an #ask method that you can use to prompt the user for input at the console.

cheers,

Matt

--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne

[Chris Whatley]

And I expected a flame over this question. :)

Thanks for the link!

[Chris Whatley]

Thanks Matt.

[Tim Walker]

Hi Chris,

Thank you for doing what I did not have the courage to do (ask about manual execution of gherkin). 

Please take a look at:

https://github.com/TimWalker/cucumbumbler/wiki

Where I've taken a stab at this (it's all just readme at this point). 

Now I see Matt's response about the ask("question", 3000) .... so I need to go understand a little more too.

So....Cucumbumbler....the "Cucumber Bumbler", the manual "Cucumber-er"...what are your thoughts? Think an OSS tool like that would be useful? 

Thank you, 

Tim

On Tue, Mar 6, 2012 at 8:37 AM, Chris Whatley <what...@gmail.com> wrote:

Thanks Matt.

--
You received this message because you are subscribed to the Google Groups "Cukes" group.

To view this discussion on the web visit https://groups.google.com/d/msg/cukes/-/50yTcrMR5qYJ.

To post to this group, send email to cu...@googlegroups.com.
To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cukes?hl=en.

[Jon Kern]

cucumber is great at BDD, but it doesn't mean it is the only test technique (preaching to choir) we should use.

it's critical to understand where automated tests shine, and where human testing is critical -- and to not confuse the two.

as far as cuking manual tests, keeping the tests in one place seems like a good advantage (as described in Tim's cucum-bumbler wiki <g>). Maybe you could use the output to the console as-is, or (re-)write your own method to store the results.

the "ask" method looks interesting

      # Suspends execution and prompts +question+ to the console (STDOUT).
      # An operator (manual tester) can then enter a line of text and hit
      # <ENTER>. The entered text is returned, and both +question+ and
      # the result is added to the output using #puts.
      # ...
      def ask(question, timeout_seconds)
      ...

    ...
  Scenario: View Users Listing
    Given I login as "Admin"
    When I view the list of users
    Then I should check the aesthetics

  Steps:
  Then /^I should check the aesthetics$/ do
    ask("#{7.chr}Does the UI have that awesome look? [Yes/No]", 10)
  end

  Scenario: View Users Listing         # features/user.feature:11
    Given I login as "Admin"           # features/step_definitions/user_steps.rb:1
    When I view the list of users      # features/step_definitions/user_steps.rb:6
Does the UI have that awesome look? [Yes/No]
Yes
    Then I should check the aesthetics # features/step_definitions/user_steps.rb:10
      Does the UI have that awesome look? [Yes/No]
      Yes

Thanks for the pointer, Matt!

jon

blog: http://technicaldebt.com
twitter: http://twitter.com/JonKernPA

Tim Walker said the following on 3/6/12 11:34 AM:

[Matt Wynne]

On 6 Mar 2012, at 19:24, Jon Kern wrote:

cucumber is great at BDD, but it doesn't mean it is the only test technique (preaching to choir) we should use.

it's critical to understand where automated tests shine, and where human testing is critical -- and to not confuse the two.

as far as cuking manual tests, keeping the tests in one place seems like a good advantage (as described in Tim's cucum-bumbler wiki <g>). Maybe you could use the output to the console as-is, or (re-)write your own method to store the results.

the "ask" method looks interesting

      # Suspends execution and prompts +question+ to the console (STDOUT).
      # An operator (manual tester) can then enter a line of text and hit
      # <ENTER>. The entered text is returned, and both +question+ and
      # the result is added to the output using #puts.
      # ...
      def ask(question, timeout_seconds)
      ...

    ...
  Scenario: View Users Listing
    Given I login as "Admin"
    When I view the list of users
    Then I should check the aesthetics

  Steps:
  Then /^I should check the aesthetics$/ do
    ask("#{7.chr}Does the UI have that awesome look? [Yes/No]", 10)
  end

or even

    ask("#{7.chr}Does the UI have that awesome look? [Yes/No]", 10).should == "Yes"

[Jon Kern]

nice tip... I ended up doing this (as there is a trailing /n that you have to chomp/ignore):

    ask("#{7.chr}Does the UI have that awesome look? [Yes/No]", 10).should =~ /yes/i

And I noticed it doesn't play well with running guard/spork. The question pops up over in the guard terminal :-(:

Spork server for RSpec, Cucumber successfully started
Running tests with args ["features/user.feature", "--tags", "@wip:3", "--wip", "--no-profile"]...

Does the UI have that awesome look? [Yes/No]
Yes

ERROR: Unknown command Yes
Done.

<shrugs>

jon

blog: http://technicaldebt.com
twitter: http://twitter.com/JonKernPA

Matt Wynne said the following on 3/6/12 3:29 PM:

[Jon Kern]

oh, i also added to my blog post on this topic, that I suppose manual tests using "ask" not working on spork probably doesn't matter... since they are manual and speed is not of the essence :-P

jon

blog: http://technicaldebt.com
twitter: http://twitter.com/JonKernPA

Matt Wynne said the following on 3/6/12 3:29 PM: