Do Not Skip Next Steps On Step Failure in Cucumber

[external...@decathlon.com]

Hello,

First of all, thanks for this wonderful framework.

Step execution results can have two main states: PASSED or FAILED (if we ignore PENDING, SKIPPED, UNDEFINED, etc.).

I'd like steps to have another middle state: WARN.

This state would threat the step as FAILED but will not skip the execution of the following steps.

Here is why:

I have such a scenario:

When I go to the category page "CATEGORY ONE" of my website

Then The product "VISIBLE 1" is "Visible"

And The product "VISIBLE 2" is "Visible"

And The product "NOT VISIBLE 1" is "Not visible"

And The product "NOT VISIBLE 2" is "Not visible"

This is one scenario for speed reasons:

All products are (or are not)  on the page: we load the category page once.

If we divide the scenario in 4 (with eg. an "Examples:" section), we would load the category page 4 times (we cannot have a Background section because the .feature file has more scenarios that don't share that background).

And this scenario is quite simple, but in real life we get more complex scenarios that navigate through several pages and have additional OPTIONAL assertions on each page: such assertion failures should be reported as errors, but the test must continue in order to spot all assertion failures, and not stop at the first failure and hide other potential failures...

Is it possible for a future version of Cucumber to add such FAILED step that continues to execute the next steps?

So that an optional assert failure will not hide other failures by not executing their steps.

This could be done with an annotation like that:

    @WarningInsteadOfFailure

    @Then("^The product " + PLACEHOLDER_STRING + " is " + PLACEHOLDER_STRING + "$")

    public void the_product_is(String productName, Visibility visibility) {

        assertThat(page(CategoryPage.class).getProduct(productName) != null).isEqualTo(visibility.isVisible());

    }

Or this could be more fine-grained:

Instead of a @WarningInsteadOfFailure annotation, we could request only specific exceptions to be treated as warnings:

@WarningInsteadOfFailure({AssertionError.class, NoSuchElementException.class})

What do you think about this proposal?

Have a nice day,

Sébastien Laoût.

[George Dinwiddie]

Sébastien,

On 1/28/16 5:19 AM, external...@decathlon.com wrote:
> Hello,
>
> First of all, thanks for this wonderful framework.
>
> Step execution results can have two main states: PASSED or FAILED (if we
> ignore PENDING, SKIPPED, UNDEFINED, etc.).
> I'd like steps to have another middle state: WARN.
> This state would threat the step as FAILED but will not skip the
> execution of the following steps.
>
> Here is why:
> I have such a scenario:
>
> When I go to the category page "CATEGORY ONE" of my website
> Then The product "VISIBLE 1" is "Visible"
> And The product "VISIBLE 2" is "Visible"
> And The product "NOT VISIBLE 1" is "Not visible"
> And The product "NOT VISIBLE 2" is "Not visible"
>
> This is one scenario for speed reasons:
> All products are (or are not) on the page: we load the category page once.
> If we divide the scenario in 4 (with eg. an "Examples:" section), we
> would load the category page 4 times (we cannot have a Background
> section because the .feature file has more scenarios that don't share
> that background).

Note that Background would execute for each scenario, also. Does it take
a long time for the page to load? Have you considered testing at an API
below the GUI?

Have you considered a data table?

When I go to the category page "CATEGORY ONE" of my website

Then the page reflects the following:
| VISIBLE 1 | Visible |
| VISIBLE 2 | Visible |
| NOT VISIBLE 1 | Not visible |
| NOT VISIBLE 2 | Not visible |

>
> And this scenario is quite simple, but in real life we get more complex
> scenarios that navigate through several pages and have additional
> OPTIONAL assertions on each page: such assertion failures should be
> reported as errors, but the test must continue in order to spot all
> assertion failures, and not stop at the first failure and hide other
> potential failures...

Automated verification works better with separate, little scenarios.
It's very different from manual verification where you're trying to save
human time.

>
> Is it possible for a future version of Cucumber to add such FAILED step
> that continues to execute the next steps?
> So that an optional assert failure will not hide other failures by not
> executing their steps.
>
> This could be done with an annotation like that:
>
> @WarningInsteadOfFailure
> @Then("^The product " + PLACEHOLDER_STRING + " is " +
> PLACEHOLDER_STRING + "$")
> public void the_product_is(String productName, Visibility visibility) {
> assertThat(page(CategoryPage.class).getProduct(productName) !=
> null).isEqualTo(visibility.isVisible());
> }
>
> Or this could be more fine-grained:
> Instead of a @WarningInsteadOfFailure annotation, we could request only
> specific exceptions to be treated as warnings:
>
> @WarningInsteadOfFailure({AssertionError.class,
> NoSuchElementException.class})
>
> What do you think about this proposal?

I wonder if you're using Cucumber as a testing tool. What is the process
that results in the scenarios?

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Andrew Premdas]

On 28 January 2016 at 10:19, <external...@decathlon.com> wrote:

Hello,

First of all, thanks for this wonderful framework.

Step execution results can have two main states: PASSED or FAILED (if we ignore PENDING, SKIPPED, UNDEFINED, etc.).

I'd like steps to have another middle state: WARN.

This state would threat the step as FAILED but will not skip the execution of the following steps.

Here is why:

I have such a scenario:

When I go to the category page "CATEGORY ONE" of my website

Then The product "VISIBLE 1" is "Visible"

And The product "VISIBLE 2" is "Visible"

And The product "NOT VISIBLE 1" is "Not visible"

And The product "NOT VISIBLE 2" is "Not visible"

For speed, simplicity, and maintenance I would write this scenario with a single Then e.g.

Then the correct products should be displayed

And then implement that in a way that eloquently expresses the idea that some products should be there and some shouldn't e.g

Then "the hidden products should hidden and visible products should be visible" do

  should_be_visible @visible_products

  should_be_hidden @hidden_products

end

I'd further improve the above by extracting into a helper method.

Cucumber does not need extra programming abilities for features. I'd argue it has far to many of these already. Perhaps this contributes too so many people trying to program in Gherkin.

Use language and abstraction to write scenarios that are elegant, concise and not littered with detail.

All best

Andrew

 

This is one scenario for speed reasons:

All products are (or are not)  on the page: we load the category page once.

If we divide the scenario in 4 (with eg. an "Examples:" section), we would load the category page 4 times (we cannot have a Background section because the .feature file has more scenarios that don't share that background).

And this scenario is quite simple, but in real life we get more complex scenarios that navigate through several pages and have additional OPTIONAL assertions on each page: such assertion failures should be reported as errors, but the test must continue in order to spot all assertion failures, and not stop at the first failure and hide other potential failures...

Is it possible for a future version of Cucumber to add such FAILED step that continues to execute the next steps?

So that an optional assert failure will not hide other failures by not executing their steps.

This could be done with an annotation like that:

    @WarningInsteadOfFailure

    @Then("^The product " + PLACEHOLDER_STRING + " is " + PLACEHOLDER_STRING + "$")

    public void the_product_is(String productName, Visibility visibility) {

        assertThat(page(CategoryPage.class).getProduct(productName) != null).isEqualTo(visibility.isVisible());

    }

Or this could be more fine-grained:

Instead of a @WarningInsteadOfFailure annotation, we could request only specific exceptions to be treated as warnings:

@WarningInsteadOfFailure({AssertionError.class, NoSuchElementException.class})

What do you think about this proposal?

Have a nice day,

Sébastien Laoût.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Andrew Premdas]

sry typos in above , corrections follow

Then the correct products should be displayed    AND

Then "the hidden products should hidden and visible products should be visible"

SHOULD BE

Then hidden products are hidden and visible products are visible

On 28 January 2016 at 18:15, Andrew Premdas <apre...@gmail.com> wrote:

On 28 January 2016 at 10:19, <external...@decathlon.com> wrote:

Hello,

First of all, thanks for this wonderful framework.

Step execution results can have two main states: PASSED or FAILED (if we ignore PENDING, SKIPPED, UNDEFINED, etc.).

I'd like steps to have another middle state: WARN.

This state would threat the step as FAILED but will not skip the execution of the following steps.

Here is why:

I have such a scenario:

When I go to the category page "CATEGORY ONE" of my website

Then The product "VISIBLE 1" is "Visible"

And The product "VISIBLE 2" is "Visible"

And The product "NOT VISIBLE 1" is "Not visible"

And The product "NOT VISIBLE 2" is "Not visible"

For speed, simplicity, and maintenance I would write this scenario with a single Then e.g.

Then the correct products should be displayed

Then hidden products are hidden and visible products are visible
 

And then implement that in a way that eloquently expresses the idea that some products should be there and some shouldn't e.g

Then "the hidden products should hidden and visible products should be visible" do

Then "hidden products are hidden and visible products are visible" do

[Matt Wynne]

I understand why you need this.

I can imagine that, rather than adding a "WARN" status, we simply allow the user to choose (either per scenario or per run) whether we should skip remaining steps or carry on when a step fails. I can see how we could implement this, at least in the Ruby version.

We'd still fail the scenario if any of the steps failed, it's just that we'd give you the chance to see whether any of the other steps failed too. I'm not against this.

[external...@decathlon.com]

Hello,

I agree this scenario is perhapse not the best one to show my need.

In real life, I have several such scenarios, and they are a little bit more complicated.

But I wanted to keep the sample short here, to not submerge you with more or less relevant details.

And yes, I have some tests that use scenario-outlines but also need to check several assertions at once.

In our company, we have another team who is using Cucumber.

Theire solution to the problem is to currentScenario.write("WARNING"), mark the step as PASSED, and interpret this in their custom report builder to show a warning.

But in JUnit (or any runner that don't use their report builder), everything is green and good: I don't want such a wobbly solution: JUnit is of a good help when developing and debugging tests in Eclipse.

Yes, we are also testing underlying APIs.

Our company is commited to have several types of tests:

- Human test explorations

- Automated service integration tests: test APIs with FitNess

- Automated regression tests: test the user interface with Selenium, like a real user would see it

We have a big database of human tests already there in a tool.

We pinpointed a subset of critical "Sanity Check" tests that we need to automate.

The goal is to achieve continuous integration and run the tests several times per day, ideally after each commit on Git.

So, execution need to be fast in order for the system to be adopted by developers team.

And the tool needs to map well to existing sanity check tests to be adopted by the QA team...

We adapted some tests to better map to Cucumber spirit.

But the "several checks needs to be performed at once" is a recurrent problem we currently face and can't adapt to Cucumber spirit without a big multiplication of the number of scenarios, and then a big duplication of them, and then a big execution time increase...

I'm aware a lot of other people already asked you to tread failures as temporar successes or such bizarre things.

But, IMHO, I think the proposal of a @WarningInsteadOfFailure({AssertionError.class}) annotation (in cucumber-JVM, at least, I don't know how to do it in Ruby or...) would keep the spirit of Cucumber intact (again, it's only my opinion, I don't have the big picture in minde like you): failures are failures.

It's about the same working as Hibernate's @Transactional(noRollbackFor = {MailException.class})...

Or an alternative to the annotation would be the step definition to throw a WarningException... It would work weel with Ruby, I suppose.

Or a better name could be OptionalAssertFailureException: as the step would be FAILED (and not WARNING) but this failure being optional means it can continue the next steps.

I think the annotation or WarningException proposal would somewhat reply to what the following users wanted (not exactly what they wanted, but in a more correct way, or at least a less hacky way):

1) http://stackoverflow.com/questions/6841467/how-can-i-make-cucumber-run-all-the-steps-not-skip-them-even-if-one-of-them-fa

=> Their solution is wobbly (catch exception, log it but the step won't throw any exception):

Quote: "This makes the overall scenario fail, but it masks the step failure from Cucumber, so all the step results are green, including the ones that aren't right." 

2) http://stackoverflow.com/questions/15298521/continue-running-cucumber-steps-after-a-failure

=> Theire solution is also wobbly (log the exception in steps, don't throw any exception in steps, but throw it in @After):

Quote: "The only issue would be that, in the report, all the steps would look green but the test case looks failed." That's exactly why you don't want to do this.

3) I've also read another thread on that forum, the guy asked to mark some errors as PASSED because they are known and will be fixed in next release (but I don't find back this thread right now).

If we use @WarningInsteadOfFailure AND we add another state (WARNING), this guy could use that mecanism to achieve somewhat what he wanted, in a more strict way.

From what I see in the Cucumber-JVM code source, the Runtime.runStep(String, Step, Reporter, I18n) method sets skipNextStep to true here:

try {

    match.runStep(i18n);

} catch (Throwable t) {

    error = t;

    status = isPending(t) ? "pending" : Result.FAILED;

    addError(t);

    skipNextStep = true; // <= HERE!

}

I think it could work if it's set to true only if the exception is not part of the @WarningInsteadOfFailure annotation (or is not WarningException.class).

I have not tested that assumption, tough.

Kind regards,

Sébastien Laoût.

[external...@decathlon.com]

@Matt Wynne:

Oh, I should have refreshed the page before replying.

Yes, that's exactly what I mean.

You said it in a (lot) more succinct way ;-)

[external...@decathlon.com]

@apremdas:

Ah, that's a good idea.

Do you mean that:

Given These products should be hidden:

      | NOT VISIBLE 1 |

      | NOT VISIBLE 2 |

  And These products should be visible:

      | VISIBLE 1 |

      | VISIBLE 2 |

 When I go to the category page "CATEGORY ONE" of my website

 Then Then hidden products are hidden and visible products are visible

That solve one of my use cases.

Thanks.

Though, this needs some work in step definitions in order to create a human readable message aggregating all failures: "The products {1} are visible but should be hidden and the products {2} are hidden visible but should be visible".

With one step per product, there is the elegant solution that each failed step appears red at the first time, and we do not need to construct a unique exception message aggregating all failed steps (here, all failed product visibilities).

That solve one of my use cases, but the problem remains for other use cases where, for instance, a lot of pages need to be accessed on the same scenario without being able to break it into several scenarios.

For instance, on an e-commerce website, adding a product to cart, going to the checkout process, and paying: we cannot break this scenario into 3 scenarios, and each one needs "optional" checks that don't impede other steps ability to succeed, but imply the scenario as a whole would fail.

@Matt Wynne

Abandoning the WARNING state, the annotation could rather be named @ContinueNextStepsFor(Throwable[] t)

Or the exception to be thrown by step definitions could be named ContinueNextStepsException(Throwable t)

[Andrew Premdas]

On 29 January 2016 at 11:02, <external...@decathlon.com> wrote:

@apremdas:

Ah, that's a good idea.

Do you mean that:

Given These products should be hidden:

      | NOT VISIBLE 1 |

      | NOT VISIBLE 2 |

  And These products should be visible:

      | VISIBLE 1 |

      | VISIBLE 2 |

 When I go to the category page "CATEGORY ONE" of my website

 Then Then hidden products are hidden and visible products are visible

Yes, though I would make it even simpler. This is the essence of writing good scenarios, refining the language until you have the just enough of the essence of the problem without all the froth of detail. So perhaps:

Given a category with hidden and visible products

When I view the category

Then I should see the visible products

And I should not see the hidden products

would be an improvement.

That solve one of my use cases.

Thanks.

Though, this needs some work in step definitions in order to create a human readable message aggregating all failures: "The products {1} are visible but should be hidden and the products {2} are hidden visible but should be visible".

With one step per product, there is the elegant solution that each failed step appears red at the first time, and we do not need to construct a unique exception message aggregating all failed steps (here, all failed product visibilities).

Yes thats the point. The place to do this sort of work is in the step definitions (or better yet in support code that the step definitions use). If you use cucumber in this way you can do just about anything, because the step definitions and below are using a programming language. Another benefit of this approach is that your scenarios become much simpler to write and maintain.

That solve one of my use cases, but the problem remains for other use cases where, for instance, a lot of pages need to be accessed on the same scenario without being able to break it into several scenarios.

For instance, on an e-commerce website, adding a product to cart, going to the checkout process, and paying: we cannot break this scenario into 3 scenarios, and each one needs "optional" checks that don't impede other steps ability to succeed, but imply the scenario as a whole would fail.

Actually you can break any process into seperate scenarios very easily. With your ecommerce example we would have

Scenario: Adding a product to the cart

  Given I am viewing a product

  When I add the product to the cart

  Then the cart should contain the product

Scenario: Going to the checkout

  Given I have added a product to the cart

  When I go to the checkout

  Then I should see the product
   ...

Scenario: Paying

  Given i have a checkout with products

  When I pay
  ...

Each scenario build on what is done previously. This isn't seperate in the sense that paying doesn't include product selection and adding to the cart, but it is seperate in the sense that you can add the product to the cart without having to have the payment code. Cucumber is designed to drive development (rather than follow development), and when you use it this way its alot easier to break scenarios down into smaller components. In your case this is a harder thing to do, but its not impossible, you just have to examine whats there and find the boundaries between things (in this case viewing, choosing, and paying - or perhaps product, cart, payment)

All best

Andrew
 

@Matt Wynne

Abandoning the WARNING state, the annotation could rather be named @ContinueNextStepsFor(Throwable[] t)

Or the exception to be thrown by step definitions could be named ContinueNextStepsException(Throwable t)

--

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[external...@decathlon.com]

With this:

Given a category with hidden and visible products
When I view the category
Then I should see the visible products
And I should not see the hidden products

I see how Cucumber should be used.

But:

- All details are hidden in the Java code: they are so abstract that business people don't understand all what's going on: what are the visible and hidden products that are checks? They can't tell if they don't have access to the Java code. And as they are not developer, we do not give them access to the Java code. We use Cucumber as a discussion layer between business people and technical people: it's the missing link :-) So we don't want to hide too much data deep into Java, so discussion can happen without misunderstanding.

- In fact, a few dozen existing test cases are based on Excel files: datasets have carefully be prepared by QA team (not developer people) to test all cases. When automating these Excel-based tests, I use scenario outline (or repeated steps if it helps), and I can easily transform their Excel file into Examples section (or suite of steps). When a "line" (or step) fails, I can easily tell QA people what happened, and where, by annotating their Excel file, or giving them the Cucumber results they can easily map to their original Excel file (the Excel file thus can be deleted and Cucumber scenarios become the reference).

- Step definition sentences are short and without a lot of context: it's easier to create duplicate sentences: "I view the category" can mean a lot of different things or a lot of different categories in all scenarios.

- Your two "Then" and "And" last sentences imply that both are executed even if the first fails, so I sense you also secretly dream of a a @ContinueNextStepsFor(Throwable) annotation ;-) Come on, it's only a few lines of code to add to Cucumber :-) OK... and a few days to test and adapt to other languages...

As for breaking processes into separate scenarios, that's a great idea.

I'll think about it for our own project, to see if that's possible.

I perhaps missed something in the Cucumber documentation, but all these tips could be a good documentation topic on how Cucumber is opinionated and how to use it in the best way, with a variety of samples like the ones in that thread, and the ones in other threads of this forum.

[George Dinwiddie]

Sébastien

On 1/29/16 4:18 AM, external...@decathlon.com wrote:
> Hello,
>
> I agree this scenario is perhapse not the best one to show my need.
> In real life, I have several such scenarios, and they are a little bit
> more complicated.
> But I wanted to keep the sample short here, to not submerge you with
> more or less relevant details.
> And yes, I have some tests that use scenario-outlines but also need to
> check several assertions at once.

Perhaps you didn't read my post carefully. I suggested using a data
table and letting the step definition check all of the assertions in one
go. It can report which of them pass and which of them fail, and then
fail the test if any of them fail.

>
> In our company, we have another team who is using Cucumber.
> Theire solution to the problem is to currentScenario.write("WARNING"),
> mark the step as PASSED, and interpret this in their custom report
> builder to show a warning.
> But in JUnit (or any runner that don't use their report builder),
> everything is green and good: I don't want such a wobbly solution: JUnit
> is of a good help when developing and debugging tests in Eclipse.

I would never want to return success for a failure and depend on human
eyes to notice the difference.

>
> Yes, we are also testing underlying APIs.
> Our company is commited to have several types of tests:
> - Human test explorations
> - Automated service integration tests: test APIs with FitNess
> - Automated regression tests: test the user interface with Selenium,
> like a real user would see it
> We have a big database of human tests already there in a tool.
> We pinpointed a subset of critical "Sanity Check" tests that we need to
> automate.
> The goal is to achieve continuous integration and run the tests several
> times per day, ideally after each commit on Git.
>
> So, execution need to be fast in order for the system to be adopted by
> developers team.
> And the tool needs to map well to existing sanity check tests to be
> adopted by the QA team...
> We adapted some tests to better map to Cucumber spirit.
> But the "several checks needs to be performed at once" is a recurrent
> problem we currently face and can't adapt to Cucumber spirit without a
> big multiplication of the number of scenarios, and then a big
> duplication of them, and then a big execution time increase...

To me, it's easy enough to specify all the things you want to see and
not see on a page in a data table. I'm not completely convinced of the
value of such, but sometimes it's helpful.

Proceeding on to subsequent steps sounds like a bad idea to me.

- George

> --
> Posting rules: http://cukes.info/posting-rules.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cukes+un...@googlegroups.com

> <mailto:cukes+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

[external...@decathlon.com]

Hello,

After a lot of discussion in our team, we reworked some scenarios, but not all of them were reworkable.

We decided we really needed the @ContinueNextStepsFor annotation.

So I've made a fork of Cucumber, and we're now using it.

The fork is available here:

https://github.com/slaout/cucumber-jvm/tree/continue-next-steps-for-exceptions-1.2.4

Its whole purpose is to offer the @ContinueNextStepsFor(Class<? extends Throwable>[]) annotation.

The README.md explains how to use it (published on Open Source Project Repository Hosting).

In this README, I also added a lot of links to explain why we've forked.

And the first links point to your explanations on how we're supposed to use Cucumber, to warn people who want to use the fork, so they use it if really needed.

@Matt Wynne: you were not totally opposed to this: what do you think about it?

I've implemented it only in Java, for our own purpose.

I don't know how to do it in other languages like Ruby...

In addition to the annotation, step definitions could ALSO throw a Cucumber's ContinueNextStepsException that would implicitely have the same effect as the annotation.

[external...@decathlon.com]

Based on our current use of the fork in our project, I added a section "Recommendations for Using the Annotation" in the README.md.

It aggregates some best practices we found by using this "super-power".

It's only a few days old, so this will evolve.

This will allow you to get our feedback for using the feature in the wild, so you can decide it's definitely not going into Cucumber, or it could be integrated in a very light mode that ensure all the best practices are applied.

[Justin C]

@MattWynne

If my team were to embark on a PR to accomplish this for cucumber-ruby, would there be love for it on the cucumber org side? I quite like the implementation route that was chosen for cucumber-js by Sébastien Laoût, this is similar to how Thoughtworks solved the problem for their 'Gauge' framework. 

On Thursday, January 28, 2016 at 7:12:59 AM UTC-5, external...@decathlon.com wrote:

[external...@decathlon.com]

Hello,

Thanks Justin.

I didn't know about Gauge.

After some research, this project looks promising.

I found what you told us about:

http://getgauge.io/documentation/user/current/advanced_readings/continue_on_failure.html

It's a simple @ContinueOnFailure, without exception class parameters.

I'm OK with it too.

I realize my last message was "It's only a few days old, so this will evolve.".

Now, it's a few months old, and in fact, it didn't evolve.

We still use this annotation extensively, and only use it with these two exception classes: @ContinueNextStepsFor({AssertionError.class, NoSuchElementException.class })

It's simpler for developers to be able to copy/paste the same thing without thinking.

And when another exception is thrown, it's practically always because of a critical bug that needs fixing.

Best regards,

Sébastien Laoût.

[Matt Wynne]

Hi Justin,

Sorry for the lag!

Yes, I’d be happy to support someone exploring this in a PR, I can understand why you might need it sometimes.

In the Ruby version, changes would need to be concentrated on the state machine in https://github.com/cucumber/cucumber-ruby-core/blob/master/lib/cucumber/core/test/runner.rb

Ping me on bitter if you need any more support with it.

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/xTqSyR1qvSc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

cheers,
Matt

—
+44(0)7974430184
ma...@cucumber.io
https://cucumber.io
https://twitter.com/mattwynne

The Cucumber logo is the intellectual property of Cucumber Ltd, a limited company registered in Scotland, number 456793.

UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead, Cairndow, Argyll, PA24 8AN UK.

CONFIDENTIALITY NOTICE: The information in this e-mail is confidential and privileged; it is intended for use solely by the individual or entity named as the recipient hereof. Disclosure, copying, distribution, or use of the contents of this e-mail by persons other than the intended recipient is strictly prohibited and may violate applicable laws. If you have received this e-mail in error, please delete the original message and notify us by email immediately. Thank you. Cucumber Ltd.

[Aslak Hellesøy]

I'm open to this feature request, but let's lift the discussion out of cucumber-ruby into the parent project. Please open an issue on github.com/cucumber/cucumber where we can discuss this.

I'm in charge of making sure Cucumber implementations in different languages converge towards similar behaviour. We should avoid adding new functionality to only one Cucumber implementation.

Aslak

[external...@decathlon.com]

Thanks very much.

I've opened this issue: https://github.com/cucumber/cucumber/issues/79

[dhru]

Hello sebastien, I have tried this project and  @ContinueNextStepsFor(AssertionError.class) this it is not working!!!! I have followed the instructions by adding dependencies listed and I have added tag@ContinueNextStepsFor(AssertionError.class) and my tests are being skipped...as there is no error I am not able to troubleshoot...Please help... 

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

Hi dhru,

I put this:

        <dependency>

            <groupId>com.github.slaout.fork.info.cukesthreads</groupId>

            <artifactId>cucumber-core</artifactId>

            <version>1.2.4-SNAPSHOT</version>

        </dependency>

        <dependency>

            <groupId>com.github.slaout.fork.info.cukesthreads</groupId>

            <artifactId>cucumber-java</artifactId>

            <version>1.2.4-SNAPSHOT</version>

        </dependency>

        <dependency>

            <groupId>com.github.slaout.fork.info.cukesthreads</groupId>

            <artifactId>cucumber-spring</artifactId>

            <version>1.2.4-SNAPSHOT</version>

        </dependency>

        <dependency>

            <groupId>com.github.slaout.fork.info.cukesthreads</groupId>

            <artifactId>cucumber-junit</artifactId>

            <version>1.2.4-SNAPSHOT</version>

        </dependency>

Does it work with these dependencies?

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

What do you mean by your tests are being skipped?

The scenarios do not execute?

An exception on the given steps make the scenario stop execute?

[Swathi Reddy]

Hi,

Is the spring depedency required for the implementaion to work.

When i added spring depedency it needed other spring librabries to compile.

And without the spring depedency , my tests on Assetion error with this @ContinueNextSteps({Assertion.class}) still stop execution

and the rest of the tests are skipped.

Its not working as intended.Can you please help?

[Mahesh babu]

Hi,

This solution is not working as expected.

@Sample
Scenario Outline: Test Cucumber Next step on failure functionality
  Given I have two numbers "<Number1>" and "<Number2>"
  Then Sum will be "<Sum>"
  And Difference will be "<Difference>"
  And Multiplication will be "<Multiply>"
  Examples:
  |Number1|Number2|Sum|Difference|Multiply|
  |7      |3      |10 |2         |21      |

@ContinueNextStepsFor(AssertionError.class)
@Then("^Sum will be \"(.*?)\"$")
public void sum_will_be(String arg1) throws Throwable {
    // Write code here that turns the phrase above into concrete actions
    sum = Integer.parseInt(arg1);
    if(sum!=a+b)
        throw new AssertionError("Addition is incorrect");
    System.out.println("Sum is correct");
}

@ContinueNextStepsFor({AssertionError.class})
@Then("^Difference will be \"(.*?)\"$")
public void difference_will_be(String arg1) throws Throwable {
    // Write code here that turns the phrase above into concrete actions
    diff = Integer.parseInt(arg1);
    if(diff!=a-b)
        throw new AssertionError("Difference is incorrect");
    System.out.println("Difference is correct");
}

@ContinueNextStepsFor(Throwable.class)
@Then("^Multiplication will be \"(.*?)\"$")
public void multiplication_will_be(String arg1) throws Throwable {
    // Write code here that turns the phrase above into concrete actions
    mul = Integer.parseInt(arg1);
    if(mul!=a*b)
        throw new AssertionError("Multiplication is incorrect");
    System.out.println("Multiplication is correct");
}

With the given data, I expect Multiplication should not skip (Difference step is fail),but still the step is skipped

Need help.

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

@Swathi Reddy Yes, I've created a new project with only the Cucumber fork, and I had to include these two other dependencies:

        <dependency>

            <groupId>org.springframework</groupId>

            <artifactId>spring-context</artifactId>

            <version>4.0.2.RELEASE</version>

        </dependency>

        <dependency>

            <groupId>org.springframework</groupId>

            <artifactId>spring-test</artifactId>

            <version>4.0.2.RELEASE</version>

        </dependency>

I'm sorry.

I will update the doc, and try to find a way to make it optional.

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

@Mahesh babu : I don't understand: you added @ContinueNextStepsFor(Throwable.class) on top of @Then("^Multiplication will be \"(.*?)\"$")

So you expect the step to make the scenario continue even if the step is failed.

If the step fails, it is not "marked as success", if that's what you meant by "skipped".

It would be counter-productive to hide errors.

The purpose of the fork is to be able to spot ALL errors, even those hidden after (an)other error(s); it is not meant to reduce the number of errors by ignoring some.

[Yugandhar Nalam]

Hi, 

I have used all the dependencies as mentioned in https://github.com/slaout/cucumber-jvm/tree/parallel-scenarios-execution-1.2.4 

But unable to navigate to next step on current step failure. Please review the step definitions and feature file description 

 @ContiueOnFailure

Scenario: Verify next step on current step failure

Given Perform Action one

Then Verify Object One

And Verify Object Two

And Verify Object Three

        @Given("^Perform Action one$")

public void perform_Action_one() throws Throwable {

System.out.println("First Action Statment");  

}

@Then("^Verify Object One$")

public void verify_Object_One() throws Throwable {

System.out.println("First Verification");

}

@ContinueNextStepsFor({AssertionError.class,NullPointerException.class})

@Then("^Verify Object Two$")

public void verify_Object_Two() throws Throwable {

System.out.println("Error in Second Verification");

throw new AssertionError("Step is failed but next one will execute");

}

@Then("^Verify Object Three$")

public void verify_Object_Three() throws Throwable {

System.out.println("Third Verification");

}

Please let me know if any additional dependencies need to be add

Thanks, 

Yugandhar