Do you believe BDD scenarios' quality matter?

[Gabriel Oliveira]

Hi folks,

Have you been practicing BDD (Behavior-Driven Development) lately and believe scenarios' quality matter?

I'm writing my graduate thesis on that subject and would like to invite you all to answer a (really) small 2 questions survey about it in here: https://goo.gl/forms/dheA8XK0asiNUVze2

If you're willing to spend a little more time to talk to me about that topic, please remember to fill out your contact information there - or comment on this thread here - and I will promptly get in touch :) !

Best regards,

-- 

Gabriel P.A. de Oliveira
https://gpaoliveirablog.wordpress.com

[George Dinwiddie]

Gabriel,

As short as it is, I find your form impossible for me to answer. The
statement "It is important to have a standard set of criteria to
evaluate written BDD scenarios' quality" implies that there can be such
a set of criteria. I have not yet found it possible, though I have been
giving talks and counseling people on the topic for years.

What qualities do you seek in scenarios? I want them to
- Communicate between roles and constituencies in an organization
- Communicate across time and space
- Describe the intended functionality
- Provide a target for development
- Check the system for unintended regressions
- Provide clear diagnostics when things go wrong
There's probably more, but these come readily to mind.

In any event, I cannot imagine a standard set of criteria to evaluate
scenarios for these qualities without regard to the context. It's like a
standard set of criteria to evaluate Shakespeare's sonnets--which is the
best sonnet?

- George

On 9/3/17 1:55 PM, Gabriel Oliveira wrote:
> Hi folks,
>
> Have you been practicing BDD (Behavior-Driven Development) lately and
> believe scenarios' quality matter?
>
> I'm writing my graduate thesis on that subject and would like to invite
> you all to answer a (really) small 2 questions survey about it in here:
> https://goo.gl/forms/dheA8XK0asiNUVze2
>
> If you're willing to spend a little more time to talk to me about that
> topic, please remember to fill out your contact information there - or
> comment on this thread here - and I will promptly get in touch :) !
>
> Best regards,
>
> --
>

> *Gabriel P.A. de Oliveira*
> /https://gpaoliveirablog.wordpress.com
> <https://gpaoliveirablog.wordpress.com>/
>
>
>
>
>
>
>
>
>
> --
> Posting rules: http://cukes.info/posting-rules.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Gabriel Oliveira]

Hi George,

It's interesting to hear that you found the questions impossible to answer - provoking reactions from people was part of their goal.

We started looking at BDD written scenarios quality inspired by characteristics of good requirements. While even User Stories have their INVEST criteria to help, BDD scenarios lack such a common list of words to guide the reviewer/writer in their job of handling BDD scenarios and what a "good" BDD scenario is. I agree that a well-written requirement is different from a good poem, but I find it interesting to clarify what are the characteristics that make each one of them "good".

We seek to acquire this understanding (or at least, to reduce our lack of understanding) by summarizing multiple people opinions taken from interviews - the survey was just the beginning of it. 

Would you be available to further talk about the subject in a skype call? 

Of course, this invitation is extendable to anyone who found the matter interesting and would like to provide their opinion :)

Best regards,

-- 

Gabriel P.A. de Oliveira
https://gpaoliveirablog.wordpress.com

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com <mailto:cukes+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
 ----------------------------------------------------------------------
  * George Dinwiddie *                      http://blog.gdinwiddie.com
  Software Development                    http://www.idiacomputing.com
  Consultant and Coach                    http://www.agilemaryland.org
 ----------------------------------------------------------------------

--
Posting rules: http://cukes.info/posting-rules.html
--- You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.

[Aslak Hellesøy]

Hi Gabriel,

Your form states:

"Since it is well known that executable solution requirements are one of many potential causes of a project failure, we come to ask some questions about this subject."

1) Are you saying that project failure can be partly caused by the fact that specifications are executable?

2) After 12 years in this field I've never heard this - on what basis are you claiming this is well known?

Cheers,

Aslak

[Gabriel Oliveira]

Hi Aslak,

Thanks for pointing this out - this was a mistake on our part. The intent was to state that fact that bad requirements are one of the many causes of project failures. 

Already fixed on bit.ly/bdd_quality

The Cucumber logo is the intellectual property of Cucumber Ltd, a limited company registered in Scotland, number 456793.

UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead, Cairndow, Argyll, PA24 8AN UK.

CONFIDENTIALITY NOTICE: The information in this e-mail is confidential and privileged; it is intended for use solely by the individual or entity named as the recipient hereof. Disclosure, copying, distribution, or use of the contents of this e-mail by persons other than the intended recipient is strictly prohibited and may violate applicable laws. If you have received this e-mail in error, please delete the original message and notify us by email immediately. Thank you. Cucumber Ltd.

[aslak hellesoy]

Thanks for the clarification Gabriel.

In the past few years, after helping dozens of organisations and hundreds of people adopt BDD I have come to the conclusion that it is counter-productive to have 3 amigos (BAs, testers and programmers) write executable specifications (Gherkin) together.

What I encourage instead is that they create *examples* together. Examples are not executable specifications - they are simply conversation topics, often jotted down on a card as a sketch or a few words. I recommend Example Mapping (https://cucumber.io/blog/2015/12/08/example-mapping-introduction) as a technique to create examples together.

When conversations happen around lo-fi examples like this in an example mapping session, stakeholders tend to uncover a lot of misunderstandings and gain new insight. In my experience, this does not happen when the conversations happen around executable specifications (Gherkin). Instead, conversations tend to revolve around grammar and formulation, diverting the attention from the problem domain and the examples themselves. Writing specifications together is also slow and tedious, and people don't enjoy these sessions as much. When it's not enjoyable, people stop doing it pretty quickly. Example Mapping on the other hand is energising and fun.

BDD is a variant of TDD, where programmers write code guided by failing tests. In my experience, this workflow does not work well when the specifications were written collaboratively. When the programmers starts the red-green-refactor cycle with collaboratively written scenarios, they quickly realise they need to reformulate those scenarios, for a number of reasons. They're incomplete, they're ambiguous etc. When scenarios start passing, the scenarios are often very different from what the diverse group created together. This often causes frustration: "You've completely rewritten it - this is not what we wrote together - I don't understand this".

So instead I recommend programmers write the executable specifications, perhaps with assistance from a tester. They use the example map as input when they do this, and do their best to capture the spirit of the examples and the conversations. When they're done (or better - while they're doing it), they show the scenarios to the BAs/POs and ask for feedback: Do you understand this? Does it reflect what we discussed during example mapping? Then they tweak it, trying to strike a good balance between 3 qualities: Specification, Test and Documentation.

BDD isn't a process where you can write the specification up-front - even when it happens collaboratively. They take shape as you go, in parallel with development. The only thing you can do up-front (before coding) is to reduce the misunderstandings, but executable specifications isn't a good medium for that I think.

Cheers,

Aslak

[Andrew Premdas]

This ties in quite nicely with my idea/opinion that any Cucumber scenario that has examples in it, particularly tables is in some way incomplete.

You complete the scenario by extracting the spirit/meaning of the examples and in the process name things. When you have done this all the tables of examples just disappear and you are left with named concepts specific to your context.

All best

Andrew

------------------------

Andrew Premdas

blog.andrew.premdas.org

[George Dinwiddie]

Andrew,

I'm not exactly sure what you mean by this. Can you give an example of
extracting the spirit/meaning of the examples and naming them?

- George

On 9/14/17 10:15 AM, Andrew Premdas wrote:
> This ties in quite nicely with my idea/opinion that any Cucumber
> scenario that has examples in it, particularly tables is in some way
> incomplete.
>
> You complete the scenario by extracting the spirit/meaning of the
> examples and in the process name things. When you have done this all the
> tables of examples just disappear and you are left with named concepts
> specific to your context.
>
> All best
>
> Andrew
>
> On 14 September 2017 at 10:32, aslak hellesoy <aslak.h...@gmail.com

> <mailto:aslak.h...@gmail.com>> wrote:
>
> Thanks for the clarification Gabriel.
>
> In the past few years, after helping dozens of organisations and
> hundreds of people adopt BDD I have come to the conclusion that it
> is counter-productive to have 3 amigos (BAs, testers and
> programmers) write executable specifications (Gherkin) together.
>
> What I encourage instead is that they create *examples* together.
> Examples are not executable specifications - they are simply
> conversation topics, often jotted down on a card as a sketch or a
> few words. I recommend Example Mapping
> (https://cucumber.io/blog/2015/12/08/example-mapping-introduction

> <https://cucumber.io/blog/2015/12/08/example-mapping-introduction>)

> <mailto:gabriel.p...@gmail.com>> wrote:
>
> Hi Aslak,
>
> Thanks for pointing this out - this was a mistake on our part.
> The intent was to state that fact that bad requirements are one
> of the many causes of project failures.
>

> Already fixed on bit.ly/bdd_quality <http://bit.ly/bdd_quality>

> *Gabriel P.A. de Oliveira*
> /https://gpaoliveirablog.wordpress.com
> <https://gpaoliveirablog.wordpress.com>/
>
>
>
>
>
>
>
>
>
>
>

> The Cucumber logo is the intellectual property of Cucumber
> Ltd, a limited company registered in Scotland, number 456793.
>
>
> UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead,
> Cairndow, Argyll, PA24 8AN UK.
>
>
> CONFIDENTIALITY NOTICE: The information in this e-mail is
> confidential and privileged; it is intended for use solely
> by the individual or entity named as the recipient hereof.
> Disclosure, copying, distribution, or use of the contents of
> this e-mail by persons other than the intended recipient is
> strictly prohibited and may violate applicable laws. If you
> have received this e-mail in error, please delete the
> original message and notify us by email immediately. Thank
> you. Cucumber Ltd.
>
> --
> Posting rules: http://cukes.info/posting-rules.html
> <http://cukes.info/posting-rules.html>
> ---
> You received this message because you are subscribed to the
> Google Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails

> from it, send an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> Posting rules: http://cukes.info/posting-rules.html
> <http://cukes.info/posting-rules.html>
> ---
> You received this message because you are subscribed to the
> Google Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> Posting rules: http://cukes.info/posting-rules.html
> <http://cukes.info/posting-rules.html>
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>
>
>
> --
> ------------------------
> Andrew Premdas
> blog.andrew.premdas.org <http://blog.andrew.premdas.org>

>
> --
> Posting rules: http://cukes.info/posting-rules.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

[Andrew Premdas]

On 14 September 2017 at 16:57, George Dinwiddie <li...@idiacomputing.com> wrote:

Andrew,

I'm not exactly sure what you mean by this. Can you give an example of extracting the spirit/meaning of the examples and naming them?

 - George

Sure, lets take something everyone knows about registrations (excuse bad syntax I don't use outlines examples or tables anymore).

Scenario Outline: Bad registrations

  When I register with <account> <password>

  Then I should not be register

   Examples:

   Account | Password

   bad_account | good_password

   good_account | 1234

   good_account |  sdfdsflsdfsdf3313

   good_account | wieikjkeiiow

OK so our business peeps have come up with some examples of account|password combinations that should fail. Now we have to take each pair and try and extract the spirit/meaning. I'll start with the easy one

good_account | 1234

This is probably to do with password length. (note all examples are by their nature ambiguous, this is why scenario outlines and examples are bad in my book, they don't have information to be clear about their intention)

So I'd extract a scenario

Scenario: Short password

  When I register with a short password

  Then I should see my password is to short

The next one is a bit more confusing. Some possibilities are

Need a capital letter

Need a special character

So here I might have to check with the business, and then I'd rinse and repeat

From this process we end up extracting and making concrete several important concepts

- short passwords

- passwords with special chars

- passwords without capitals

etc.

Now it might seem a stretch but you can apply this concept to every scenario outline, and to every example group or table of examples. And if you do, not only will you get simpler features that are easier to implement and work with, you'll also get most of the 'names' you need to write the code to implement the particular behaviour you are specifiying (yay!)

HTH

Andrew

 

On 9/14/17 10:15 AM, Andrew Premdas wrote:

This ties in quite nicely with my idea/opinion that any Cucumber scenario that has examples in it, particularly tables is in some way incomplete.

You complete the scenario by extracting the spirit/meaning of the examples and in the process name things. When you have done this all the tables of examples just disappear and you are left with named concepts specific to your context.

All best

Andrew

            from it, send an email to cukes+unsubscribe@googlegroups.com
            <mailto:cukes+unsubscribe@googlegroups.com>.

            For more options, visit https://groups.google.com/d/optout
            <https://groups.google.com/d/optout>.

        --         Posting rules: http://cukes.info/posting-rules.html
        <http://cukes.info/posting-rules.html>
        ---
        You received this message because you are subscribed to the
        Google Groups "Cukes" group.
        To unsubscribe from this group and stop receiving emails from

        it, send an email to cukes+unsubscribe@googlegroups.com
        <mailto:cukes+unsubscribe@googlegroups.com>.

        For more options, visit https://groups.google.com/d/optout
        <https://groups.google.com/d/optout>.

    --     Posting rules: http://cukes.info/posting-rules.html
    <http://cukes.info/posting-rules.html>
    ---
    You received this message because you are subscribed to the Google
    Groups "Cukes" group.
    To unsubscribe from this group and stop receiving emails from it,

    send an email to cukes+unsubscribe@googlegroups.com
    <mailto:cukes+unsubscribe@googlegroups.com>.

    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.




--
------------------------
Andrew Premdas
blog.andrew.premdas.org <http://blog.andrew.premdas.org>

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com <mailto:cukes+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
 ----------------------------------------------------------------------
  * George Dinwiddie *                      http://blog.gdinwiddie.com
  Software Development                    http://www.idiacomputing.com
  Consultant and Coach                    http://www.agilemaryland.org
 ----------------------------------------------------------------------

--
Posting rules: http://cukes.info/posting-rules.html
--- You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[George Dinwiddie]

Nice example, Andrew. I have found that when using tables for password
rules I needed a <reason> column to explain which rule was being tested.
That's a sure sign to me that a different scenario is beneficial.

I was at a client recently and they were delighted that they could use
Scenario Outlines rather than repeating the scenario. The issue had to
do with different legal requirements in different states. I don't
remember the details, so these made up examples will have to suffice.

Scenario Outline: State requirements for Spouse as SNI
Given the Primary Named Insured resides in <state>
And the Secondary Named Insured has a relationship of "Spouse"
When the Agent views the policy
Then there is no option to remove the Secondary Named Insured

Examples:
| state |
| AL |
| AK |

Scenario Outline: No state requirements for Spouse as SNI
Given the Primary Named Insured resides in <state>
And the Secondary Named Insured has a relationship of "Spouse"
When the Agent views the policy
Then the Agent can remove the Secondary Named Insured

Examples:
| state |
| SC |
| SD |

As I recall, there were other state-oriented scenarios with value
limits, but I can't recall them well enough to generate examples.

I can certainly see how the above could be written as individual
scenarios, but it was both more convenient and less error prone to
implement it with tables.

- George

On 9/14/17 12:36 PM, Andrew Premdas wrote:
>
>
> On 14 September 2017 at 16:57, George Dinwiddie <li...@idiacomputing.com

> <aslak.h...@gmail.com <mailto:aslak.h...@gmail.com>
> <mailto:aslak.h...@gmail.com

> <mailto:gabriel.p...@gmail.com>
>     <mailto:gabriel.p...@gmail.com

> <mailto:gabriel.p...@gmail.com>>> wrote:
>
>         Hi Aslak,
>
>         Thanks for pointing this out - this was a mistake on
> our part.
>         The intent was to state that fact that bad requirements
> are one
>         of the many causes of project failures.
>
>         Already fixed on bit.ly/bdd_quality

> <http://bit.ly/bdd_quality> <http://bit.ly/bdd_quality>

>
>         On Wed, Sep 13, 2017 at 11:03 PM, Aslak Hellesøy
>         <as...@cucumber.io <mailto:as...@cucumber.io>

>                 <https://gpaoliveirablog.wordpress.com
> <https://gpaoliveirablog.wordpress.com>>/

[Chuck van der Linden]

Rather than an special 'reason' column this is a situation where  I would suggest you make use of the fact that scenario outlines can accept more than a single set of examples, and that the Examples: keyword can have other text after it.  e,g

Scenario Outline: Users are not allowed to use weak or 'bad' passwords when changing their password

  steps..

  Examples:  Password contains username
    example table

  Examples:  Password too short
    example table 

  Examples:  Password lacks complexity

    example table

  Examples: reusing prior or recent password

    example table

[Andrew Premdas]

I find it quite interesting to expand scenarios like this and then look into refactoring them with the aim to uncover business concepts rather than make my scenarios concise.

In this example it seems that there is a simple concept

in some states you can remove the secondary named insured

in other states you can't

What you examples here are doing is wasting large amounts of run time to repeat something that does not need repeating.

As far as the UI is concerned we only need 2 scenarios

1. A scenario that shows there is an option to remove the secondary named insured

2. A scenario that shows that option isn't present

As for testing whether a particular state is a removing state or not, you can do that in a unit test that runs 100 times faster.

Now you only have two states in your examples, but it would be very easy to add more and if you apply that across multiple scenarios in a large application you can see why people end up with cukes that have huge run times.

For a better example of taking apart a more complex scenario have a look at an ancient blog post of mine http://pages.andrew.premdas.org/2011/06/27/composable-features-and-tables.html

All best

On 14 September 2017 at 18:37, George Dinwiddie <li...@idiacomputing.com> wrote:

Nice example, Andrew. I have found that when using tables for password rules I needed a <reason> column to explain which rule was being tested. That's a sure sign to me that a different scenario is beneficial.

I was at a client recently and they were delighted that they could use Scenario Outlines rather than repeating the scenario. The issue had to do with different legal requirements in different states. I don't remember the details, so these made up examples will have to suffice.

Scenario Outline: State requirements for Spouse as SNI
  Given the Primary Named Insured resides in <state>
  And the Secondary Named Insured has a relationship of "Spouse"
  When the Agent views the policy
  Then there is no option to remove the Secondary Named Insured

  Examples:
    | state |
    | AL    |
    | AK    |

Scenario Outline: No state requirements for Spouse as SNI
  Given the Primary Named Insured resides in <state>
  And the Secondary Named Insured has a relationship of "Spouse"
  When the Agent views the policy
  Then the Agent can remove the Secondary Named Insured

  Examples:
    | state |
    | SC    |
    | SD    |

As I recall, there were other state-oriented scenarios with value limits, but I can't recall them well enough to generate examples.

I can certainly see how the above could be written as individual scenarios, but it was both more convenient and less error prone to implement it with tables.

 - George

        <aslak.h...@gmail.com <mailto:aslak.hellesoy@gmail.com>

        <mailto:aslak.hellesoy@gmail.com

        <mailto:gabriel.pa.oliveira@gmail.com>

             <mailto:gabriel.pa.oliveira@gmail.com

--
Posting rules: http://cukes.info/posting-rules.html
--- You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Premdas]

On 15 September 2017 at 17:37, Chuck van der Linden <sqa...@gmail.com> wrote:

Rather than an special 'reason' column this is a situation where  I would suggest you make use of the fact that scenario outlines can accept more than a single set of examples, and that the Examples: keyword can have other text after it.  e,g

Scenario Outline: Users are not allowed to use weak or 'bad' passwords when changing their password

  steps..

  Examples:  Password contains username
    example table

  Examples:  Password too short
    example table 

  Examples:  Password lacks complexity

    example table

  Examples: reusing prior or recent password

    example table

--

This has two flaws from my perspective

1. Its harder to read and debug.

2.  It still keeps the example data in the feature file. This is the really important point.

I much prefer to extract examples out of scenarios getting them from either step definition helpers or the application itself (if possible). The thing with example data in scenarios is that when the rules change the scenario has to change. Lets take a short password as an example.

Lets say we write

Scenario Outline:
   ...

   Example: short password

      email | password
      ...       |  1234567890

and we have a password rule that says passwords must be greater than 10 chars. Now when the business rule changes to passwords can be 8 chars (because the users are complaining) then we have to modify the sample data.

Consider the alternative

Scenario:

   When I login with a short password
   ...

Now in our step def we can have

When "I login with a short password" do

   login as: @i, password: short_password

end

and somewhere we can have

def short_password

  # here we go to the application, and get the minimum password length
  # and then return a shorter password

end

Now when the rule changes we don't have to modify any of our features and scenarios, the cost of changing this business rule has been dramatically reduced.

Again you can extend this idea to every piece of example data in a scenario.

Example data in implemented scenario == development debt (there is still work to do)

All best

Andrew

 

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Dinwiddie]

Andrew,

It's possible to hide too much, and lose the advantages of concrete
examples.

On 9/16/17 11:48 AM, Andrew Premdas wrote:
>
>
> On 15 September 2017 at 17:37, Chuck van der Linden <sqa...@gmail.com

> <mailto:sqa...@gmail.com>> wrote:
>
>
> Rather than an special 'reason' column this is a situation where  I
> would suggest you make use of the fact that scenario outlines can
> accept more than a single set of examples, and that the Examples:
> keyword can have other text after it.  e,g

<snip>

>
> This has two flaws from my perspective
>
> 1. Its harder to read and debug.
>
> 2.  It still keeps the example data in the feature file. This is the
> really important point.
>
> I much prefer to extract examples out of scenarios getting them from
> either step definition helpers or the application itself (if possible).
> The thing with example data in scenarios is that when the rules change
> the scenario has to change. Lets take a short password as an example.
>
> Lets say we write
>
> Scenario Outline:
>    ...
>
>    Example: short password
>       email | password
>       ...       |  1234567890
>
> and we have a password rule that says passwords must be greater than 10
> chars. Now when the business rule changes to passwords can be 8 chars
> (because the users are complaining) then we have to modify the sample data.

When the rule changes, it seems reasonable to modify the examples that
illustrate the rule.

>
> Consider the alternative
>
> Scenario:
>    When I login with a short password
>    ...
>
> Now in our step def we can have
>
> When "I login with a short password" do
>    login as: @i, password: short_password
> end
>
> and somewhere we can have
>
> def short_password
>   # here we go to the application, and get the minimum password length
>   # and then return a shorter password
> end

This is too far, in my opinion. With this, the test now depends on the
application for the threshold to check. If the application reports this
wrong, the test is wrong. We're back to a single point of failure.

Back in the early 2000s, there was a bookkeeper learning programming who
showed up on the XP list for awhile. It often seemed that he had only
one lens for viewing the world, and that was double-entry bookkeeping.
Even though he was often dismissed for sounding like a Johnny One-Note
on the list, I came to view this as a profound insight about the nature
of test-driven development. The tests might be wrong, but as long as
they are an INDEPENDENT and different representation of what's desired,
they are likely to catch errors. The error might be in the test or
application, and a human can figure that out.

Once we start using the application to tell us what the answer should
be, or even calculate the desired answer using the same algorithm, we've
lost that quality.

>
> Now when the rule changes we don't have to modify any of our features
> and scenarios, the cost of changing this business rule has been
> dramatically reduced.
>
> Again you can extend this idea to every piece of example data in a
> scenario.

I've seen this taken too far. In a recent discussion at a client, they
were going round and round on developing a particular scenario. They
were developing something like this:

Scenario: Driver One information must match Customer Application
Given the Customer Application has a different field values than
Prefill data
When the user lands on the driver screen
Then Driver One information displays the field values from the
Customer Application

People kept getting confused about what it meant. Essentially, the
scenario is restating the rule rather than provide an example.

I listened to this for awhile, and suggested they listen to the language
they were using to explain it to each other, and write the scenario
using that language. They ended up with something like this:

Scenario: When prefilling applicant data, data on the application
takes precedence
Given the Customer Application has an Occupation of "Lawyer"
And the Prefill returns an occupation of "Musician"
When the user lands on the driver screen
Then Driver One information displays an Occupation of "Lawyer"
And Driver One Prefill information displays an Occupation of "Musician"

Now this scenario could, IMO, be improved. The name "Prefill" is poorly
chosen. Using a verb as a name is generally confusing. It was, however,
the commonly used name, and therefore not easily changed.

Using concrete examples for a particular field and particular values in
that field, though, communicated much more clearly than the more
abstract descriptions. And that is precisely the advantage that accrues
from using examples to illustrate the rules.

And, bringing this back to the subject line of this thread, this
illustrates the difficulty in assessing the quality of a BDD scenario.
There is no one exact point that can be pronounced "good quality." Human
judgement is better than any mnemonic.

- George

[George Dinwiddie]

Andrew,

On 9/16/17 11:31 AM, Andrew Premdas wrote:
> I find it quite interesting to expand scenarios like this and then look
> into refactoring them with the aim to uncover business concepts rather
> than make my scenarios concise.
>
> In this example it seems that there is a simple concept
>
> in some states you can remove the secondary named insured
> in other states you can't
>
> What you examples here are doing is wasting large amounts of run time to
> repeat something that does not need repeating.
>
> As far as the UI is concerned we only need 2 scenarios
>
> 1. A scenario that shows there is an option to remove the secondary
> named insured
> 2. A scenario that shows that option isn't present

Were we only concerned with machines, I would agree with you. But we're
also concerned with people. And some of those people are focused on
business risk rather than technical risk. If the scenarios do not make
plain the things those people care about, then the scenarios are not
doing their job. In this example, the concern is less about making the
scenarios concise than it is about making the example clearly express
the actual needs, as perceived by the Three Amigos and the
constituencies they represent.

>
> As for testing whether a particular state is a removing state or not,
> you can do that in a unit test that runs 100 times faster. >
> Now you only have two states in your examples, but it would be very easy
> to add more and if you apply that across multiple scenarios in a large
> application you can see why people end up with cukes that have huge run
> times.

And those unit tests might as well be greek to the business people. They
would be back to testing all of these things by hand.

Scenarios don't have to be tied to the UI, and don't have to run slowly.
Given the nature of this particular project (a portal that integrates a
number of back-end systems) I suspect they will be, for the most part,
but I've encouraged them to organize their javascript so that some can
be tested without deploying to a server.

>
> For a better example of taking apart a more complex scenario have a look
> at an ancient blog post of mine
> http://pages.andrew.premdas.org/2011/06/27/composable-features-and-tables.html

In the original formulation of that, I notice that this feature is
checking two distinct and non-interacting sets of rules. One is the
rules about taxes, and one is the rules about shipping costs. I would
divide the feature file into separate features for each of these sets of
rules before doing anything else.

cheers,
George

>
> All best
>
> On 14 September 2017 at 18:37, George Dinwiddie <li...@idiacomputing.com

> <mailto:li...@idiacomputing.com

> <mailto:aslak.h...@gmail.com>

> <mailto:aslak.h...@gmail.com <mailto:aslak.h...@gmail.com>>
>         <mailto:aslak.h...@gmail.com
> <mailto:aslak.h...@gmail.com>
>
>         <mailto:aslak.h...@gmail.com

> <mailto:gabriel.p...@gmail.com>

>         <mailto:gabriel.p...@gmail.com
> <mailto:gabriel.p...@gmail.com>>
>              <mailto:gabriel.p...@gmail.com
> <mailto:gabriel.p...@gmail.com>
>         <mailto:gabriel.p...@gmail.com

> send an email to cukes+un...@googlegroups.com
> <mailto:cukes%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>
>
>
> --
> ------------------------
> Andrew Premdas
> blog.andrew.premdas.org <http://blog.andrew.premdas.org>

>
> --
> Posting rules: http://cukes.info/posting-rules.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

[Andrew Premdas]

On 17 September 2017 at 17:38, George Dinwiddie <li...@idiacomputing.com> wrote:

Andrew,

On 9/16/17 11:31 AM, Andrew Premdas wrote:

I find it quite interesting to expand scenarios like this and then look into refactoring them with the aim to uncover business concepts rather than make my scenarios concise.

In this example it seems that there is a simple concept

in some states you can remove the secondary named insured
in other states you can't

What you examples here are doing is wasting large amounts of run time to repeat something that does not need repeating.

As far as the UI is concerned we only need 2 scenarios

1. A scenario that shows there is an option to remove the secondary named insured
2. A scenario that shows that option isn't present

Were we only concerned with machines, I would agree with you. But we're also concerned with people. And some of those people are focused on business risk rather than technical risk. If the scenarios do not make plain the things those people care about, then the scenarios are not doing their job. In this example, the concern is less about making the scenarios concise than it is about making the example clearly express the actual needs, as perceived by the Three Amigos and the constituencies they represent.

I think you are trying to hard to defend the scenario here. Clearly the scenario is about secondary named insured, not about which state employs which particular rules. If you insist on munging the two things together then I can't help you. Often business wants to do this. My personal style is to challenge this and force the separation, but I understand in certain business contexts this might not be the politically correct thing to do. However I don't think that should colour this discussion.

What you seem to be saying is that when we discuss this scenario outside of a business context we should accept and prefer a scenario which convolutes two separate contexts over two separate tests that separate and clarify these contexts.

I'm quite happy for you to say that you will live with the scenario that deals with both contexts because educating your business users is to challlenging. My whole point is that the stuff you get from business IS NOT the canonical source it is a flawed expression that has to be examined, questioned and translated to become the code that tests your application.

As for testing whether a particular state is a removing state or not, you can do that in a unit test that runs 100 times faster. >
Now you only have two states in your examples, but it would be very easy to add more and if you apply that across multiple scenarios in a large application you can see why people end up with cukes that have huge run times.

And those unit tests might as well be greek to the business people. They would be back to testing all of these things by hand.

I have no concern that my unit tests or my integration tests are greek to business people. That is expected. The only thing my tests MAY have to do is communicate to business people that their intentions are being met. If business people are reading the output of Cucumber scenarios and using them to determine whether their application functions or not then they are idiots or completely misinformed. A cucumber scenario in itself going green is completely meaningless, as every scenario will go green with no code. Cucumber scenarios are just a way for business people to contribute to the drive for development. Their executions proves nothing.

No matter how I implement my scenarios they and their output will never be understood by business users in the same way that the code of my application won't be understood by business users. My implemented scenarios are code and their output is not for business users it is for the developers who will have to build open the provided functionality. This idea that you can use the output of cucumber scenarios to judge the health of an application baffles me. If your a business person and you want to judge the health of an application look at the application not a wall of green text from cucumber scenarios.

Scenarios don't have to be tied to the UI, and don't have to run slowly. Given the nature of this particular project (a portal that integrates a number of back-end systems) I suspect they will be, for the most part, but I've encouraged them to organize their javascript so that some can be tested without deploying to a server.

Scenarios will always run slowly as they are by definition abstract descriptions of the behaviour of an application described via its user interface. The fact that I can run several hundred scenarios over my application in 3 minutes in a single process on my laptop with a browser doing the javascript does not effect this. I could run 10 times the number of unit test in a few seconds IF I wrote them as well as I write my scenarios. Wasting scenario runtime to test/exercise things that have already been exercised is really detrimental to the overall effectiveness of using Cucumber to drive development.

All best

Andrew

        <mailto:aslak.hellesoy@gmail.com>
        <mailto:aslak.hellesoy@gmail.com <mailto:aslak.hellesoy@gmail.com>>
                 <mailto:aslak.hellesoy@gmail.com
        <mailto:aslak.hellesoy@gmail.com>

                 <mailto:aslak.hellesoy@gmail.com

        <mailto:gabriel.pa.oliveira@gmail.com>
                 <mailto:gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>>
                      <mailto:gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>
                 <mailto:gabriel.pa.oliveira@gmail.com

    send an email to cukes+unsubscribe@googlegroups.com

    <mailto:cukes%2Bunsubscribe@googlegroups.com>.

    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.




--
------------------------
Andrew Premdas
blog.andrew.premdas.org <http://blog.andrew.premdas.org>

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com <mailto:cukes+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
 ----------------------------------------------------------------------
  * George Dinwiddie *                      http://blog.gdinwiddie.com
  Software Development                    http://www.idiacomputing.com
  Consultant and Coach                    http://www.agilemaryland.org
 ----------------------------------------------------------------------

--
Posting rules: http://cukes.info/posting-rules.html
--- You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Steve Tooke]

On 18 Sep 2017, at 02:31, Andrew Premdas <apre...@gmail.com> wrote:

As for testing whether a particular state is a removing state or not, you can do that in a unit test that runs 100 times faster. >
Now you only have two states in your examples, but it would be very easy to add more and if you apply that across multiple scenarios in a large application you can see why people end up with cukes that have huge run times.

And those unit tests might as well be greek to the business people. They would be back to testing all of these things by hand.

I have no concern that my unit tests or my integration tests are greek to business people. That is expected. The only thing my tests MAY have to do is communicate to business people that their intentions are being met. If business people are reading the output of Cucumber scenarios and using them to determine whether their application functions or not then they are idiots or completely misinformed. A cucumber scenario in itself going green is completely meaningless, as every scenario will go green with no code. Cucumber scenarios are just a way for business people to contribute to the drive for development. Their executions proves nothing.

No matter how I implement my scenarios they and their output will never be understood by business users in the same way that the code of my application won't be understood by business users. My implemented scenarios are code and their output is not for business users it is for the developers who will have to build open the provided functionality. This idea that you can use the output of cucumber scenarios to judge the health of an application baffles me. If your a business person and you want to judge the health of an application look at the application not a wall of green text from cucumber scenarios.

If the output of Cucumber is meaningless to the people who require the software why bother going to the effort of automating a Gherkin scenario? Why add that layer that translate the Gherkin to code? You say their value is to contribute to driving the development. Couldn’t you achieve that simply by using concrete examples in conversations and then translating them to a RSpec or JUnit, etc. Why are we writing in English (or other natural language) if these things aren’t meant to be read by people. If they are only meant for developers code is a less ambiguous to way to specify things.

Cucumber & Gherkin has 3 values:

1) Executable specifications based on concrete examples – this is how we are able to consider the business rules and discuss whether they meet our (users’) needs, and because we specify software before we write it they fit naturally into a TDD process

2) Automated tests – as software is delivered, executable specs form part of our regression suite. We learn quickly if new changes have effected earlier work, and we can decide if this is expected or not

3) Documentation - these executable specifications, written in (almost) natural language, can document what our software does. This is valuable to people on the team and people outside of the team

There’s always a balance that needs to be struck between these three values, and its down to each team to strike the right balance for themselves. I do think though that the hardest to balance are automated tests and documentation. Automated tests push us to brevity and the minimum required as we get concerned with run length. Documentation screams for concrete examples, for richness to interest a reader.

Scenarios don't have to be tied to the UI, and don't have to run slowly. Given the nature of this particular project (a portal that integrates a number of back-end systems) I suspect they will be, for the most part, but I've encouraged them to organize their javascript so that some can be tested without deploying to a server.

Scenarios will always run slowly as they are by definition abstract descriptions of the behaviour of an application described via its user interface. The fact that I can run several hundred scenarios over my application in 3 minutes in a single process on my laptop with a browser doing the javascript does not effect this. I could run 10 times the number of unit test in a few seconds IF I wrote them as well as I write my scenarios. Wasting scenario runtime to test/exercise things that have already been exercised is really detrimental to the overall effectiveness of using Cucumber to drive development.

I disagree with your definition that scenarios must run against a user interface. Scenarios are concerned with being business readable. Some of them will run end-to-end, but they don’t need to.

I wrote about this before: http://tooky.co.uk/cucumber-and-full-stack-testing/ as did Seb Rose: http://claysnow.co.uk/the-testing-iceberg/.

cheers,

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

[Andrew Premdas]

On 18 September 2017 at 09:16, Steve Tooke <steve...@gmail.com> wrote:

On 18 Sep 2017, at 02:31, Andrew Premdas <apre...@gmail.com> wrote:

As for testing whether a particular state is a removing state or not, you can do that in a unit test that runs 100 times faster. >
Now you only have two states in your examples, but it would be very easy to add more and if you apply that across multiple scenarios in a large application you can see why people end up with cukes that have huge run times.

And those unit tests might as well be greek to the business people. They would be back to testing all of these things by hand.

I have no concern that my unit tests or my integration tests are greek to business people. That is expected. The only thing my tests MAY have to do is communicate to business people that their intentions are being met. If business people are reading the output of Cucumber scenarios and using them to determine whether their application functions or not then they are idiots or completely misinformed. A cucumber scenario in itself going green is completely meaningless, as every scenario will go green with no code. Cucumber scenarios are just a way for business people to contribute to the drive for development. Their executions proves nothing.

No matter how I implement my scenarios they and their output will never be understood by business users in the same way that the code of my application won't be understood by business users. My implemented scenarios are code and their output is not for business users it is for the developers who will have to build open the provided functionality. This idea that you can use the output of cucumber scenarios to judge the health of an application baffles me. If your a business person and you want to judge the health of an application look at the application not a wall of green text from cucumber scenarios.

If the output of Cucumber is meaningless to the people who require the software why bother going to the effort of automating a Gherkin scenario? Why add that layer that translate the Gherkin to code? You say their value is to contribute to driving the development. Couldn’t you achieve that simply by using concrete examples in conversations and then translating them to a RSpec or JUnit, etc. Why are we writing in English (or other natural language) if these things aren’t meant to be read by people. If they are only meant for developers code is a less ambiguous to way to specify things.

Because it drives the development of the the functionality that the business people want. The automation provides regression so that the next piece of functionality can be developed. We write scenarios to facilitate communication between devs and business so the devs can have an understanding of WHAT is wanted and WHY its important before they start building stiff. But once a scenario has been written and is in use then I see it having much less functionality for the business people and much more functionality for the developers. This transition from a scenario being a driver of development, to scenario being code is key. And a necessary part of making that transition is extracting meaning from examples.
 

Cucumber & Gherkin has 3 values

1) Executable specifications based on concrete examples – this is how we are able to consider the business rules and discuss whether they meet our (users’) needs, and because we specify software before we write it they fit naturally into a TDD process

2) Automated tests – as software is delivered, executable specs form part of our regression suite. We learn quickly if new changes have effected earlier work, and we can decide if this is expected or not

3) Documentation - these executable specifications, written in (almost) natural language, can document what our software does. This is valuable to people on the team and people outside of the team

There’s always a balance that needs to be struck between these three values, and its down to each team to strike the right balance for themselves. I do think though that the hardest to balance are automated tests and documentation. Automated tests push us to brevity and the minimum required as we get concerned with run length. Documentation screams for concrete examples, for richness to interest a reader.

I don't really disagree with this, but I feel you are not appreciating the lifespan of the scenario. In its creation 1) is paramount, but after its implementation 1) is almost irrelevant. 2) doesn't even exist before a scenario is implemented  after its implementation 2) is paramount. 3) is a sort of constant presence.

As I'm talking about what it takes to implement and live with scenarios 1) has much less value at this point in the scenarios lifecycle. Surely it makes more sense for a business person to experience the functionality of something by interacting with its implementation (remember at this point in time the functionality is implemented) rather than be reading a bit of Gherkin.

Finally the original point of the thred is about exploring the idea specifications based on concrete examples are not sufficient to implement business rules. The examples need their meaning extracted so  we can explicitly state the intention of each rule and give it a name. If you don't do this you are left with ambiguity and overly complex scenarios which hinder future development.

Scenarios don't have to be tied to the UI, and don't have to run slowly. Given the nature of this particular project (a portal that integrates a number of back-end systems) I suspect they will be, for the most part, but I've encouraged them to organize their javascript so that some can be tested without deploying to a server.

Scenarios will always run slowly as they are by definition abstract descriptions of the behaviour of an application described via its user interface. The fact that I can run several hundred scenarios over my application in 3 minutes in a single process on my laptop with a browser doing the javascript does not effect this. I could run 10 times the number of unit test in a few seconds IF I wrote them as well as I write my scenarios. Wasting scenario runtime to test/exercise things that have already been exercised is really detrimental to the overall effectiveness of using Cucumber to drive development.

I disagree with your definition that scenarios must run against a user interface. Scenarios are concerned with being business readable. Some of them will run end-to-end, but they don’t need to.

I never made the assertion that scenarios must run against a user interface. I made the assertion that scenarios will always run slower than unit tests (assuming they are equally well written) because scenarios execute larger chunks of functionality. (I hope we are agreed that using Gherkin to write unit tests is a pretty big waste of time ;))

Anyhow good talking to you. Hope the above has cleared some things up.

All best

Andrew

All best

Andrew

        <mailto:lists@idiacomputing.com

        <mailto:aslak.hellesoy@gmail.com <mailto:aslak.hell...@gmail.com>>

                      <gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>
                 <mailto:gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>>
                      <mailto:gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>
                 <mailto:gabriel.pa.oliveira@gmail.com
        <mailto:gabriel.pa.oliveira@gmail.com>>>> wrote:

                          Hi Aslak,

                          Thanks for pointing this out - this was a
        mistake on
                 our part.
                          The intent was to state that fact that bad
        requirements
                 are one
                          of the many causes of project failures.

                          Already fixed on bit.ly/bdd_quality
        <http://bit.ly/bdd_quality>
                 <http://bit.ly/bdd_quality> <http://bit.ly/bdd_quality>

                          On Wed, Sep 13, 2017 at 11:03 PM, Aslak Hellesøy

                          <as...@cucumber.io <mailto:asl...@cucumber.io>

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com<mailto:cukes+unsub...@googlegroups.com>.

[Chuck van der Linden]

On Sunday, September 17, 2017 at 9:03:38 AM UTC-7, George Dinwiddie wrote:

Andrew,

It's possible to hide too much, and lose the advantages of concrete
examples.

On 9/16/17 11:48 AM, Andrew Premdas wrote:
>
>
> On 15 September 2017 at 17:37, Chuck van der Linden <sqa...@gmail.com
> <mailto:sqa...@gmail.com>> wrote:
>
>
>     Rather than an special 'reason' column this is a situation where  I
>     would suggest you make use of the fact that scenario outlines can
>     accept more than a single set of examples, and that the Examples:
>     keyword can have other text after it.  e,g
        <snip>
>
> This has two flaws from my perspective
>
> 1. Its harder to read and debug.
>
> 2.  It still keeps the example data in the feature file. This is the
> really important point.
>
> I much prefer to extract examples out of scenarios getting them from
> either step definition helpers or the application itself (if possible).
> The thing with example data in scenarios is that when the rules change
> the scenario has to change. 

I'm with George on this one.  IMHO a large purpose of the examples is to describe the business logic with specific examples (because just words are often too ambiguous).  I'm all for hiding implementation detail in the steps, things like what button to click, which field to fill in.. stuff that might all change without the business logic being any different.   But I'm strongly against burying any of the details of the expected behavior due to the business rules down in steps.  The eventual result of seeking to simplify and make more generic the step wording is to muddy the waters around specifics that are important for all to understand.   The other things that happens with that approach, as your wording becomes more and more generalized is you head towards what I call the 'one ring' scenario.

   Given our application and a user

   When the user uses our application

   Then everything works the way we expect it to. 

  

[Andrew Premdas]

Of course you can take any idea, and use a bad example to dismiss it. My point is that examples need to be translated into specific, precise wording that names concepts and provides an understanding that is relevant to the context and can be shared. How have you got from that to your one ring scenario?

Extracting meaning from examples is challenging work, which is one reason people don't do it! You have to find the underlying concept behind each example and give it a meaningful name, this requires in in depth understanding of the business process, establishing a consensus about the vocabulary you are using and ensuring uniqueness in your naming. Its precisely the opposite of wording becoming more generalized. Scenarios do become more concise but at the same time they become more precise.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.