Continuation of Cucumber Step even after assertion failure

[Eajaz Ali]

Hi Team,

I am using cucumber-jvm framework for my automation tests. I want to implement continuation of my step even after there is an assertion failure. For example in my step definition:

@Given("^I perform validation$")

public I_perform_validation(){

//assertion1  -> fail

//assertion2  -> pass

//assertion3   -> pass

}

If assertion1 fails, I still want to continue to assertion2 and assertion3.

Can someone please help. This will be really helpful, as I don't want to go over @Given 3 times for three different assertions.

Regards

Eajaz

[Paolo Ambrosio]

On Wed, May 13, 2015 at 1:50 AM, Eajaz Ali <eajaz...@gmail.com> wrote:
> Hi Team,
>
> I am using cucumber-jvm framework for my automation tests. I want to
> implement continuation of my step even after there is an assertion failure.
> For example in my step definition:
>
> @Given("^I perform validation$")
> public I_perform_validation(){
>
> //assertion1 -> fail
> //assertion2 -> pass
> //assertion3 -> pass
> }
>
> If assertion1 fails, I still want to continue to assertion2 and assertion3.

What is supposed to be the outcome in that case? What are you trying to achieve?

I cannot think of a simple way to achieve what you want. Cucumber does
not define assertions itself but relies on external frameworks like
JUnit or TestNG. Cucumber only checks if the step has thrown an
exception or not.

[Eajaz Ali]

My outcome s this step needs to completely executed, irrespective of assertion failures. I am trying to implement asserts from junit library.

Regards

Eajaz

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/OTUjEupNjYk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aslak Hellesøy]

On Wednesday, 13 May 2015 at 07:24, Eajaz Ali wrote:

My outcome s this step needs to completely executed, irrespective of assertion failures.

Why do you need to run the steps after the failing one?

I am trying to implement asserts from junit library.

Can you explain what you mean by this? JUnit provides static assertion methods that you can call directly.

Aslak 

You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

[Mucahit Kurt]

Hi,

You can catch the assertion exceptions like below but in this case you can not see the failure assertions/steps via cucumber.

​

> @Given("^I perform validation$")
> public I_perform_validation(){
>

try{
> //assertion1  -> fail

} cathc(AssertionException e) {

//log something

}
> //assertion2  -> pass
> //assertion3   -> pass
> }

Mücahit Kurt

You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

[Paul Rayner]

'Given' is for setting up context, the state of the system under test, not for making assertions. That is what 'Then' is for.

The 'Then' (plus any subsequent 'And' statements) is where the assertion is made. Hence, "Then".

It's generally considered very poor practice to continue when an assertion fails, since the whole point of making such an assertion in BDD is to verify that some piece of important functionality is currently missing and needs to be implemented. Why would you want to continue?

[aslak hellesoy]

On Wed, May 13, 2015 at 7:52 AM, Mucahit Kurt <mucah...@gmail.com> wrote:

Hi,

You can catch the assertion exceptions like below but in this case you can not see the failure assertions/steps via cucumber.

​

> @Given("^I perform validation$")
> public I_perform_validation(){
>

try{
> //assertion1  -> fail

} cathc(AssertionException e) {

//log something

}
> //assertion2  -> pass
> //assertion3   -> pass
> }

This is bad advice.

Tests are warning lights that are supposed to start flashing when something is broken.

What you're suggesting here is to unplug the warning lights.

Never catch exceptions in step definitions. Let them bubble through and fail the scenario when an exception is thrown.

Aslak

[Paolo Ambrosio]

On Wed, May 13, 2015 at 7:16 AM, Paul Rayner <pa...@virtual-genius.com> wrote:
> On Tue, May 12, 2015 at 6:50 PM, Eajaz Ali <eajaz...@gmail.com> wrote:
>>
>> Hi Team,
>>
>> I am using cucumber-jvm framework for my automation tests. I want to
>> implement continuation of my step even after there is an assertion failure.
>> For example in my step definition:
>>
>> @Given("^I perform validation$")
>> public I_perform_validation(){
>>
>> //assertion1 -> fail
>> //assertion2 -> pass
>> //assertion3 -> pass
>> }
>>
>> If assertion1 fails, I still want to continue to assertion2 and
>> assertion3.
>>
>> Can someone please help. This will be really helpful, as I don't want to
>> go over @Given 3 times for three different assertions.
>
>
> 'Given' is for setting up context, the state of the system under test, not
> for making assertions. That is what 'Then' is for.

I disagree with this statement. I find it very useful to have
assertions in Given steps. The purpose is to check that the data setup
has been done correctly. Sometimes I even use assertions in When steps
for sanity checks (e.g. to verify that the server didn't return an
internal server error). In both cases of course the assertions do not
verify business rules, as that must happen in Then steps.

> The 'Then' (plus any subsequent 'And' statements) is where the assertion is
> made. Hence, "Then".
>
> It's generally considered very poor practice to continue when an assertion
> fails, since the whole point of making such an assertion in BDD is to verify
> that some piece of important functionality is currently missing and needs to
> be implemented. Why would you want to continue?
>

[aslak hellesoy]

On Wed, May 13, 2015 at 9:04 AM, Paolo Ambrosio <pa...@ambrosio.name> wrote:

On Wed, May 13, 2015 at 7:16 AM, Paul Rayner <pa...@virtual-genius.com> wrote:
> On Tue, May 12, 2015 at 6:50 PM, Eajaz Ali <eajaz...@gmail.com> wrote:
>>
>> Hi Team,
>>
>> I am using cucumber-jvm framework for my automation tests. I want to
>> implement continuation of my step even after there is an assertion failure.
>> For example in my step definition:
>>
>> @Given("^I perform validation$")
>> public I_perform_validation(){
>>
>> //assertion1  -> fail
>> //assertion2  -> pass
>> //assertion3   -> pass
>> }
>>
>> If assertion1 fails, I still want to continue to assertion2 and
>> assertion3.
>>
>> Can someone please help. This will be really helpful, as I don't want to
>> go over @Given 3 times for three different assertions.
>
>
> 'Given' is for setting up context, the state of the system under test, not
> for making assertions. That is what 'Then' is for.

I disagree with this statement. I find it very useful to have
assertions in Given steps. The purpose is to check that the data setup
has been done correctly.

When I explain Given-When-Then I often compare to a theatrical act.

Before the act starts, the stage workers work behind the curtains to set up the *stage*. They place furniture in the right locations, adjust the lighting to give the impression of dusk. The actors walk onto the stage. Mrs Alving is in the dining room. Oswald is in the living room. The curtains unfold, and the stage is set.

This is the Given. There is nothing to verify, just setting up the scene. We're only describing what's happened in an imaginary recent past.

Now the act unfolds. The actors talk and move. At one point something important *happens*.

This is the When.

Later, the act has finished. The curtains are drawn again. The spectators are sitting in their seats, thinking about everything that unraveled before their eyes. What happened? What was essential? What was the *outcome*?

This is the Then.

----

Now, back to Cucumber. Sometimes we're dealing with a system where test automation is an after-thought. We can't easily set up the stage the way we want, because we're not in control over the venue, the decorations or even the actors. We have to deal with a system that is in a state we can't control. So we have to improvise. We can go on with the act if certain elements are present. Are we indoors? Check. Is there a table? Check. Do we have a woman and a man? Check.

I think this is what you're describing Paolo. I agree that in such cases, Given steps can make assertions. It's important to emphasise that this is a different kind of game though. You only do that when things are unpredictable. If we can, we want to make things predictable. We want to control (or stub out) external dependencies. When we can do that, we don't need assertions in our Given steps.

The same goes for When steps. You *can* use assertions here, but you shouldn't have to.

The only place where you absolutely *should* use assertions is Then steps.

A final word of advice: Never trust a test you haven't seen fail. You want to make sure your steps will fail with an error when the system doesn't behave as expected. If you catch exceptions or continue after a step has failed, your tests are useless, because they *cannot* fail.

It's like taking the batteries out of your smoke detector.

Aslak

[mila.rodrig...@gmail.com]

With regards to the original question about whether you can run all assertions even if one in the series fails, JUnit supports the concept of an "Error Collector" which does just that. The API documentation can be found here: http://junit.org/apidocs/org/junit/rules/ErrorCollector.html. Since this is vanilla JUnit, it should work properly with cucumber, though I haven't personally attempted it myself.

Mila

[Eajaz Ali]

Hi All,

Thank you very much for your great advices on using Assertions. I want to perform field level validation on a particular field with different kinds of test data like blank data, invalid data, special characters.

If I want to pass them from features file, then for each test data, the selenium driver is launched in @Before method. That would eat lot of time in execution only, but with manual testing if would hardly take seconds. To accommodate  this, I want to include all possible assertions w.r.t UI level field level validation in my one step , for eg in @Then. At the same time. I record all my failures and successful assertions in one single run.

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/OTUjEupNjYk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

[Andrew Premdas]

On 13 May 2015 at 16:58, Eajaz Ali <eajaz...@gmail.com> wrote:

Hi All,

Thank you very much for your great advices on using Assertions. I want to perform field level validation on a particular field with different kinds of test data like blank data, invalid data, special characters.

If I want to pass them from features file, then for each test data, the selenium driver is launched in @Before method. That would eat lot of time in execution only, but with manual testing if would hardly take seconds. To accommodate  this, I want to include all possible assertions w.r.t UI level field level validation in my one step , for eg in @Then. At the same time. I record all my failures and successful assertions in one single run.

Well Cucumber is not really a tool for doing this kind of stuff :). Assuming you know that, you can still do this reasonably elegantly by pushing all your assertion testing down from the step definition into a helper method.

The helper method would look something like (in rubyish pseudocode)

module MultipleAssertionStepHelper

  def assert_lots( ...

     results = []
     assertions.each do |assertion|

        results << assertion.result

     end

     results.contains failures ? report failures : passed

  end

end

and your Then would be something like

Then "blah should be good" do

   expect(assert_lots assertions: blah_assertions).to be_good

end

By pushing the code down you take it out of cucumber and into a domain you control, now you can do whatever you want.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Björn Rasmusson]

JUnit ErrorCollectors cannot be used in Cucumber-JVM step definitions in the same way they are used in JUnit test methods (JUnit preforms some "magic" with respect to the @Rule annotated ErrorCollector fields when running its test methods, that Cucumber-JVM does not do when running scenarios). But with some tweaks it is possible to use the JUnit ErrorCollectors in step definitions to be able to perform multiple assert (and get all failures, not just the first failure).

First the verify() method of the error collector needs to be called explicitly in the step definition. And secondly, since the verify() method is protected in the ErrorCollector class, it need to sub-classed to make it public.

With a ErrorCollector subclass like:

class MyCollector extends ErrorCollector {
  @Override
  public void verify() throws Throwable {
    super.verify();
  };
}

It is possible to define step definitions like:

@Then("<good description that makes the step understandable>")
public void step_def_method_name() throws Throwable {
  MyCollector collector = new MyCollector();
  collector.checkThat(actual1, equalTo(expected1));
  collector.checkThat(actual2, equalTo(expected2));
  collector.verify();
}

Björn

Mila