[Cucumber, Capybara] Ordering the sequence of tagged scenarios

[James GameSparks]

Hello all,

I have multiple .features in my test suite.

scores.feature
teams.feature

Within those .features are multiple scenarios.

scores.feature

Scenario: create Score
....
Scenario: edit Score
....

teams.feature

Scenario: create Team
....
Scenario: edit Team
....

My features are organised and grouped by areas of our system.    However, I want to add some new scenarios within a feature (for example, that create team scores) which would rely on other features being run first (i.e. create the teams first in teams.feature):

scores.feature

Scenario: create Score
....
Scenario: edit Score
...

Scenario: create Team Score
....

Because I want to add scenarios that relate more to the scores.feature, that's where they should belong.  But I can't run them if some of the scenarios in the teams.feature haven't run first.  How can I do that?  Let's also say that scores.feature MUST run before teams.feature because of other dependencies.

I tried running the features (in sequence) with tags:

scores.feature

@default
Scenario: create Score
....
@default
Scenario: edit Score
...

@teams
Scenario: create Team Score
....

teams.feature

@default
Scenario: create Team
....
@default
Scenario: edit Team
....

then running them with:

cucumber BASE_URL=https://mygame.com RESOURCES_DIR=tests/features/resources tests/features/scores.feature tests/features/teams.feature --tags @default @teams

or

cucumber BASE_URL=https://mygame.com RESOURCES_DIR=tests/features/resources tests/features/scores.feature tests/features/teams.feature --tags @default,@teams

But this will just run all the scenarios within scores.feature first (including any with @teams tag) and then run teams.feature afterwards.  Obviously, when it gets to the "create Team Score" scenario, it fails because it doesn't have info it needs from the teams.feature being run first.

How can I make it run through the scores.feature, run the original scenarios ignoring the "create Team Scores" scenario for now, then run through the teams.feature, coming back to scores.feature and run the new "create Team Scores" scenario, without having to run two separate commands such as:

--tags @default --tags ~@teams

then:

--tags @teams

[Steve Tooke]

Hi James,

Answers are inline below.

I would suggest that you should work really hard to make your scenarios _stateless_.

You want to get to having each scenario being able to run completely on its own, independent of any other scenarios.

Each scenario should be responsible for creating everything it needs to run - it’s context.

Use the given steps to create the context that you are interested in.

So with your examples you might expect something like:

Feature: Scores
Scenario: Create Score
Given there are no scores
When I create a score "xxx"
Then there should be a score “xxx”

Scenario: Create Score
Given a score “xxx"
When I change score “xxx” to “yyy"
Then there should be a score “yyy”

Feature: Teams
# something similar to the above

Feature: Team Scores
Given a team “abc”
When I create a team score “xxx” for “abc"
Then there should be a score “xxx” for team “abc”

By making your scenarios independent and responsible for their own context you will find that the resulting tests are much less brittle because they do not have interdependencies. You will also benefit from being about to run each one independently.

Another benefit is that you can then start to vary the context, and explore the problem in more detail.

Feature: Team scores again
Given a team score “xxx” for “abc”
When I create a team score “yyy” for “abc"
Then …. what should happen now?

If you’re scenarios all create interdependent state you can’t create new scenarios without breaking all of the other scenarios that rely on a particular data model.

One of the features we are adding in Cucumber 2.0 is the ability to randomise the order that your scenarios are executed. We want to do this to help people ensure that there is no order-dependency in their scenarios. It might be worth giving it a try to help flush out these scenarios.

Cheers,
Steve
--
E: st...@boxjump.co.uk
T: +44 7919 337 463
http://tooky.co.uk | http://kickstartacademy.io | https://twitter.com/tooky

[aslak hellesoy]

I guess this is still up for debate, but I personally think Cucumber Ruby 2.0 should offer the ability to *not* randomise the order.

If we want users to adopt the idea that independent scenarios is a good thing we need to make randomised ordering the default behaviour. RSpec already does this.

If people want a predictable order they can specify it with --plugin alphabetic-order or something. People could also specify custom ordering schemes, for example a scheme that would run the scenarios most likely to fail first.

Aslak

 

We want to do this to help people ensure that there is no order-dependency in their scenarios. It might be worth giving it a try to help flush out these scenarios.

Cheers,
Steve
--
E: st...@boxjump.co.uk
T: +44 7919 337 463
http://tooky.co.uk | http://kickstartacademy.io | https://twitter.com/tooky

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Tooke]

Agreed - but I think we should wait for Cucumber-Ruby 2.1 before making it the default. We are trying really hard to make 2.0 a drop in replacement and I guess(?) there are enough order-dependent cucumber suites out there that it could be a surprise.

Steve

[Rob Park]

2 cents: 

Wouldn't 2.0 be a more appropriate place from a semver perspective to change defaults? 

And if there's a switch to make it possible that feels similar to rspec changing it's 3.0 formatter with an allowance for rspec-legacy_formatters.

@robpark

[Steve Tooke]

On 12 Dec 2014, at 21:02, Rob Park <robert...@gmail.com> wrote:

On Fri, Dec 12, 2014 at 10:57 AM, Steve Tooke <st...@boxjump.co.uk> wrote:

On 12 Dec 2014, at 15:30, aslak hellesoy <aslak.h...@gmail.com> wrote:

On Fri, Dec 12, 2014 at 2:19 PM, Steve Tooke <st...@boxjump.co.uk> wrote:

One of the features we are adding in Cucumber 2.0 is the ability to randomise the order that your scenarios are executed.

I guess this is still up for debate, but I personally think Cucumber Ruby 2.0 should offer the ability to *not* randomise the order.

If we want users to adopt the idea that independent scenarios is a good thing we need to make randomised ordering the default behaviour. RSpec already does this.

Agreed - but I think we should wait for Cucumber-Ruby 2.1 before making it the default. We are trying really hard to make 2.0 a drop in replacement and I guess(?) there are enough order-dependent cucumber suites out there that it could be a surprise.

2 cents: 

Wouldn't 2.0 be a more appropriate place from a semver perspective to change defaults? 

And if there's a switch to make it possible that feels similar to rspec changing it's 3.0 formatter with an allowance for rspec-legacy_formatters.

Normally I would agree, but there are a couple of problems with doing that here.

We want to give people a deprecation warning if they are running specs with no options that default ordering will be changing. With that warning we will need to let them know how they can use the new ordering and check if their features will run with the new. If it doesn’t we will need to let them know how to configure things to keep the current default ordering.

Unfortunately we have no way of providing random ordering on the 1.3.x versions of cucumber.

So I think we should introduce the warning in 2.0, and wait until 2.1 to change the ordering. We could wait until 3.0 to make the change? But is this a breaking change, or an update to a default?

Steve

[Rob Park]

On Mon, Dec 15, 2014 at 6:11 AM, Steve Tooke <st...@boxjump.co.uk> wrote:

On 12 Dec 2014, at 21:02, Rob Park <robert...@gmail.com> wrote:

On Fri, Dec 12, 2014 at 10:57 AM, Steve Tooke <st...@boxjump.co.uk> wrote:

On 12 Dec 2014, at 15:30, aslak hellesoy <aslak.h...@gmail.com> wrote:

On Fri, Dec 12, 2014 at 2:19 PM, Steve Tooke <st...@boxjump.co.uk> wrote:

One of the features we are adding in Cucumber 2.0 is the ability to randomise the order that your scenarios are executed.

I guess this is still up for debate, but I personally think Cucumber Ruby 2.0 should offer the ability to *not* randomise the order.

If we want users to adopt the idea that independent scenarios is a good thing we need to make randomised ordering the default behaviour. RSpec already does this.

Agreed - but I think we should wait for Cucumber-Ruby 2.1 before making it the default. We are trying really hard to make 2.0 a drop in replacement and I guess(?) there are enough order-dependent cucumber suites out there that it could be a surprise.

2 cents: 

Wouldn't 2.0 be a more appropriate place from a semver perspective to change defaults? 

And if there's a switch to make it possible that feels similar to rspec changing it's 3.0 formatter with an allowance for rspec-legacy_formatters.

Normally I would agree, but there are a couple of problems with doing that here.

We want to give people a deprecation warning if they are running specs with no options that default ordering will be changing. With that warning we will need to let them know how they can use the new ordering and check if their features will run with the new. If it doesn’t we will need to let them know how to configure things to keep the current default ordering.

That's a really good point. 

Unfortunately we have no way of providing random ordering on the 1.3.x versions of cucumber.

So I think we should introduce the warning in 2.0, and wait until 2.1 to change the ordering. We could wait until 3.0 to make the change? But is this a breaking change, or an update to a default?

It seems it comes down to your interpretation of "new, backwards compatible functionality". 

And yeah, I see your point that changing the default is still backwards compatible as long as providing the extra option keeps it working as before.

Steve

Thanks. Makes sense to me.

@rob 

[James GameSparks]

According to the documentation here:
https://github.com/cucumber/cucumber/wiki/Tags

@billing
Feature: Verify billing

  @important
  Scenario: Missing product description

  Scenario: Several products

cucumber --tags @billing --tags @important    # Runs the first scenario (Scenarios with @important AND @billing)
cucumber --tags @billing,@important           # Runs both scenarios (Scenarios with @important OR @billing)

But this isn't representative of how it really works. If my feature looks like this:

@main
Feature: Totals

Scenario: Create Total
...

@teams
Scenario: Create Team Total
...

and I run:

--tags @default --tags @teams

It runs BOTH scenarios, not just the LAST one.

So what is the correct behavior?  Are the docs wrong?

[byrnejb]

On Friday, 12 December 2014 10:57:10 UTC-5, Steve Tooke wrote:

Agreed - but I think we should wait for Cucumber-Ruby 2.1 before making it the default. We are trying really hard to make 2.0 a drop in replacement and I guess(?) there are enough order-dependent cucumber suites out there that it could be a surprise.

Steve

Would it not make more sense, from a semantic version point of view, to release the drop-in replacement, with deprecation, as the last release of 1.X .Y series?  And then move right on 2.0.0.

[aslak hellesoy]

On Mon, Dec 15, 2014 at 3:47 PM, James GameSparks <james....@gamesparks.com> wrote:

According to the documentation here:
https://github.com/cucumber/cucumber/wiki/Tags

@billing
Feature: Verify billing

  @important
  Scenario: Missing product description

  Scenario: Several products

cucumber --tags @billing --tags @important    # Runs the first scenario (Scenarios with @important AND @billing)
cucumber --tags @billing,@important           # Runs both scenarios (Scenarios with @important OR @billing)

But this isn't representative of how it really works. If my feature looks like this:

@main
Feature: Totals

Scenario: Create Total
...

@teams
Scenario: Create Team Total
...

and I run:

--tags @main --tags @teams

It runs BOTH scenarios, not just the LAST one.

Works for me:

cucumber --tags @main --tags @teams  https://gist.githubusercontent.com/aslakhellesoy/aa9a87381d129eb29902/raw/248c24d9da10e363d36091523a6a2b89621255e7/tags.feature

Aslak

 

So what is the correct behavior?  Are the docs wrong?

[James GameSparks]

I want to run a set of features excluding tagged scenarios (~@teams) within those features.

cucumber BASE_URL=https://mygame.com  scores.feature teams.feature results.feature --tags @default ~@teams

Once they are completed, I want to run some of those features again but to exclusively run the tagged scenarios (@teams) to generate the team scores:

cucumber BASE_URL=https://mygame.com  scores.feature teams.feature results.feature --tags @teams

However, I can't do this within a single command.  It must be within a single command.  I tried:

cucumber BASE_URL=https://mygame.com  scores.feature teams.feature results.feature --tags @default ~@teams --tags @teams


and

cucumber BASE_URL=https://mygame.com  scores.feature teams.feature results.feature --tags @default ~@teams scores.feature teams.feature results.feature --tags @teams


That doesn't work.

I tried using profiles in my cucumber.yml, the first command line for one profile (preTeams) and the second command line in another (postTeams) and calling both profiles, one after the other.

cucumber BASE_URL=https://mygame.com -p resources -p preTeams -p postTeams -p json_report

But still, I run into the same problem.  The tests never run:

Using the resources, preTeams, postTeams and json_report profiles...
0 scenarios
0 steps
0m0.000s

How can I run those features, excluding certain tags, then run the same tests only including those tags, all within the same command?

On Friday, December 12, 2014 12:50:28 PM UTC, James GameSparks wrote:

[James B. Byrne]

On Wed, January 7, 2015 08:31, James GameSparks wrote:
>
> How can I run those features, excluding certain tags, then run the
> same tests only including those tags, all within the same command?

At some point the cucumber API changed so that all tag flags are
ANDed. That means --tags @teams ~@teams will return nothing for the
intersection of 'is X' and 'is not X' is null; as you have discovered
empirically. It not clear from your message why you deem it necessary
to do so. What are you trying to accomplish? Why will two successive
runs of cucumber with different tag flags checked not produce the
results you seek?

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:Byr...@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

[James GameSparks]

Because the way my Features are structured (based on areas of the system, not by use cases), some scenarios within my features require some information (e.g. team data) to have been set up, which is done later in the process.  The scenarios that are related to this information are skipped and then run again at a later stage.  I know it is not the standard when it comes to creating Cucumber scenarios, but this is what is in place.

Secondly, I can't run more than one command because I need to output the whole test results into one .json file at the end.  The -out parameter does not allow me to append the test results to the end of an existing file.  Running separate commands will either have the results in separate output files or the test results in the .json file overwritten by the following tests executed by the next command if using the same output filename.

On Friday, December 12, 2014 12:50:28 PM UTC, James GameSparks wrote:

[James B. Byrne]

On Wed, January 7, 2015 09:42, James GameSparks wrote:
> Because the way my Features are structured (based on areas of the
> system,
> not by use cases), some scenarios within my features require some
> information (e.g. team data) to have been set up, which is done later
> in
> the process. The scenarios that are related to this information are
> skipped and then run again at a later stage. I know it is not the
> standard
> when it comes to creating Cucumber scenarios, but this is what is in
> place.

This practice is called scenario coupling and it is gravely deprecated
by the Cucumber community. I believe that you will find it more
useful to construct some sort of setup and tear-down step-definitions
that you can call from in the Given section of the relevant scenarios
or the Before section of the feature file.

>
> Secondly, I can't run more than one command because I need to
> output the whole test results into one .json file at the end.
> The -out parameter does not allow me to append the test results
> to the end of an existing file.
>

But, you do not need the --out parameter at all. Use the '>' redirect
on the first run to create the file and the '>>' on the second to
append.

cucumber --tags @X --format=json feature.feature > output.json
cucumber --tags ~@X --format=json feature.feature >> output.json

[James B. Byrne]

On Wed, January 7, 2015 10:08, James B. Byrne wrote:

>
> This practice is called scenario coupling and it is gravely deprecated
> by the Cucumber community.
>

If you absolutely MUST do this and you need to run cucumber multiple
times against aggregated result sets then take a look in
./features/support/env.rb and read up about:

DatabaseCleaner.strategy = :truncation, {:except => %w[tbl-x tbl-y]}

But, if you do continue down this road you will reach a point where
your project testing with cucumber will drop off a cliff.

[James GameSparks]

I totally understand your points of view here.
Ultimately, I decided to split the features.  It seems the best option for now until we ever decide to base them around individual use cases.

So I've gone for scores_preTeams.feature and scores_postTeams.feature for example.