Is there any flow control or conditional processing in Gherkin or Behat?

[John Brookes]

Am I missing something?

I want to model a scenario:

Given I try to log in as "Bob"

If I see a message "User not recognised"

Then I create a user "Bob"

And I try to log in as "Bob"

If I see a message "User not authenticated"

Then I authenticate as "Bob"

And I try to log in as "Bob"

If I can see "Welcome Bob"

Then .....

I can write the individual steps quite easily, but neither Gherkin nor Behat appear to have any flow control or conditional operators.  I want as much written in 'English' as possible, so it can be maintained by my customers - when they change the wording of the messages for instance.

Is this possible?

If this is a question that's answered elsewhere, please point me there - I have looked but found nothing.

Regards

John

[Jakub Zalas]

Hi John,

I'm sorry but you're doing it wrong. 

Instead of "trying" to do something you should either "do it" or "make an assertion". Your scenario is completely not readable, I don't really see what you're trying to achieve. Consider following scenarios:

Scenario: Successful login

  Given I am a registered user

    And I visit the homepage

  When I fill in my login details

    And I press "Log in"

  Then I should be logged in

    And I should see "You've been successfully logged in"

Scenario: Failed login

  Given I am a registered user

    And I visit the homepage

  When I fill in my login details incorrectly

    And I press "Log in"

  Then I should not be logged in

    And I should see "Incorrect e-mail or password"

There's no need for what you called "flow of control". Gherkin is not a programming language. Its purpose is to have a readable acceptance criteria (requirements) that can be executed to automatically verify your application's behaviour. You're not suppose to code your login in Gherkin.

Go through behat docs, they're not long: http://docs.behat.org/

Gherkin docs might be a good start to see what it has to offer: http://docs.behat.org/guides/1.gherkin.html

I also recommend reading the RSpec book. It's a great introduction to BDD (don't be scared by the tools used there, it's not about the tools).

[John Brookes]

You appear to be approaching this as a pure standalone test environment, the stage or UAT scenario where you know a given account exists - 'Given I am a registered user'.

I am trying to develop a sanity checker to quickly run through on our live system, after deploying the latest updates.  I do not know the exact status of my account, and I cannot just dive into the database to delete it - I must use my test process to see if the account exists, and to react accordingly.

Having scenarios which fail is not acceptable as we want to build this in to Jenkins to automatically roll back if the sanity test fails,  so the 'given I am a registered user' approach will not work.

If there was some way to break out of a scenario, we could do...

Scenario 1

  Given I set global status to 'loggedout'

  And I break out if I am not logged in

  Then I set global status to 'loggedin'

  And I can do things knowing I am logged in

  And I will not get a failure

Scenario 2

  Given I break out if global status is not 'loggedout' 

  Then I can do things knowing I am logged out

  And I will not get a failure

(yes, I know scenarios are supposed to be independent, but I already have status passing working)

Perhaps Behat / Gherkin is the wrong language to be writing these tests, but it is important that the customer can manage them for himself (ie they need to be in 'pseudo-english') and they need to be structured in such a way that the customer can follow them.

Regards

John

[Marijn Huizendveld]

How are your customers? I'm fairly confident that no customer will understand the examples as you've shown them.

Can't you run those assertions in your step definitions? I would argue they don't belong in your scenario's.

Your features and scenarios are about business value, not about technical implementation details.

Cheers

[Jakub Zalas]

On Wednesday, October 10, 2012 11:47:02 AM UTC+1, John Brookes wrote:

You appear to be approaching this as a pure standalone test environment, the stage or UAT scenario where you know a given account exists - 'Given I am a registered user'.

Yes, Behat is an acceptance testing framework. You're expecting an application to behave in a certain way. If it behaves differently than you expected in the scenarios, it doesn't work properly.

 

I am trying to develop a sanity checker to quickly run through on our live system, after deploying the latest updates.  I do not know the exact status of my account, and I cannot just dive into the database to delete it - I must use my test process to see if the account exists, and to react accordingly.

 

Having scenarios which fail is not acceptable as we want to build this in to Jenkins to automatically roll back if the sanity test fails,  so the 'given I am a registered user' approach will not work.

Forgetting the fact you're misusing Behat, I think you could achieve your goal by extending Gherkin (it's a separate project, https://github.com/behat/gherkin). You might even forget Behat and use Gherking+Mink in some weird custom way...

 

Anyway, do you really believe your customers will write those scenarios in a language you developed?

[John Brookes]

I am sorry, I understood Behat to be a behavioural driven development tool, which allows an analyst and a customer to sit down and agree exactly how a system will work, and then take that through to provide tools to test the developing system and to help maintain it in future.  If I cannot get a set of tests (a subset of the overall suite) which can run without failing on the live system, then the tool is failing at the point where it should be most useful.

I am sure I do not need to tell you that more time is spent updating and maintaining code than is spent in writing it, so I need a test which will work with real data on a real, live server to ensure the whole update / test / deploy process has worked.  At present we test this by bringing in anybody we can find to run through a few sample scripts, so we test the specific fix but miss the side effects.  I am trying to identify a tool which will allow us to automate these tests, which we can then use to drive BDD and TDD up through the whole development process.

If Behat can only provide UAT tools, perhaps you can suggest an alternative which will provide the full range of ability.  The important thing is that it allows my customers to write 'tests' (or specifications) in an english-like language, with as natural a flow as possible, so branching within a scenario to respond to the state of the system is important.

Regards

John

[John Brookes]

I would not expect a customer to understand the examples like this.  My ideal is to keep as much as possible in pseudo-english, so something like:

Given I am on the home page

If I can see the login button

  Then I fill in 'username' with 'bob'

  And I fill in 'password' with '12345'

  And I press 'login'

  And I see 'login successful'

...

The only difference here between 'Given' or 'When' and 'If' is that Given and When break out with an error, while if breaks out without an error.

I can easily do this in a step, using PHP, but I want to do it in 'english' so the customer can see what is happening, and adjust it if they change the field names or response message.

Regards

John

[Konstantin Kudryashov]

You're approaching BDD from wrong direction. It's not human-readable nor client-understandable
testing. Your *.feature files are business values descriptions, not tests. Keeping this in mind, Gherkin
(in any programming language) will never support conditionals or loops or any other programmatic
construction as Gherkin is not a programming language, it's business-specific markup language.

Behat tests your features, using your system. Not opposite (tests your system, using features). *.feature(s)
hold human descriptions (though in special format) of your application business values. Behat with help
of your FeatureContext tests them. FeatureContext holds tests, that's why it's written in php, uses php and
provides tools for php - this is the place you should use loops, conditionals and variables. Gherkin is just
a markdown for your user stories. Features are testable, but not tests themselves.

John Brookes

11 October 2012 10:29

I would not expect a customer to understand the examples like this.  My ideal is to keep as much as possible in pseudo-english, so something like:

Given I am on the home page

If I can see the login button

  Then I fill in 'username' with 'bob'

  And I fill in 'password' with '12345'

  And I press 'login'

  And I see 'login successful'

...

The only difference here between 'Given' or 'When' and 'If' is that Given and When break out with an error, while if breaks out without an error.

I can easily do this in a step, using PHP, but I want to do it in 'english' so the customer can see what is happening, and adjust it if they change the field names or response message.

Regards

John

On Wednesday, October 10, 2012 11:55:25 AM UTC+1, Marijn wrote:

Marijn Huizendveld

10 October 2012 12:55

How are your customers? I'm fairly confident that no customer will understand the examples as you've shown them.

Can't you run those assertions in your step definitions? I would argue they don't belong in your scenario's.

Your features and scenarios are about business value, not about technical implementation details.

Cheers

John Brookes

10 October 2012 12:47

You appear to be approaching this as a pure standalone test environment, the stage or UAT scenario where you know a given account exists - 'Given I am a registered user'.

I am trying to develop a sanity checker to quickly run through on our live system, after deploying the latest updates.  I do not know the exact status of my account, and I cannot just dive into the database to delete it - I must use my test process to see if the account exists, and to react accordingly.

Having scenarios which fail is not acceptable as we want to build this in to Jenkins to automatically roll back if the sanity test fails,  so the 'given I am a registered user' approach will not work.

If there was some way to break out of a scenario, we could do...

Scenario 1

  Given I set global status to 'loggedout'

  And I break out if I am not logged in

  Then I set global status to 'loggedin'

  And I can do things knowing I am logged in

  And I will not get a failure

Scenario 2

  Given I break out if global status is not 'loggedout' 

  Then I can do things knowing I am logged out

  And I will not get a failure

(yes, I know scenarios are supposed to be independent, but I already have status passing working)

Perhaps Behat / Gherkin is the wrong language to be writing these tests, but it is important that the customer can manage them for himself (ie they need to be in 'pseudo-english') and they need to be structured in such a way that the customer can follow them.

Regards

John

On Wednesday, October 10, 2012 9:44:13 AM UTC+1, Jakub Zalas wrote:

Jakub Zalas

10 October 2012 10:44

Hi John,

I'm sorry but you're doing it wrong. 

Instead of "trying" to do something you should either "do it" or "make an assertion". Your scenario is completely not readable, I don't really see what you're trying to achieve. Consider following scenarios:

Scenario: Successful login

  Given I am a registered user

    And I visit the homepage

  When I fill in my login details

    And I press "Log in"

  Then I should be logged in

    And I should see "You've been successfully logged in"

Scenario: Failed login

  Given I am a registered user

    And I visit the homepage

  When I fill in my login details incorrectly

    And I press "Log in"

  Then I should not be logged in

    And I should see "Incorrect e-mail or password"

There's no need for what you called "flow of control". Gherkin is not a programming language. Its purpose is to have a readable acceptance criteria (requirements) that can be executed to automatically verify your application's behaviour. You're not suppose to code your login in Gherkin.

Go through behat docs, they're not long: http://docs.behat.org/

Gherkin docs might be a good start to see what it has to offer: http://docs.behat.org/guides/1.gherkin.html

I also recommend reading the RSpec book. It's a great introduction to BDD (don't be scared by the tools used there, it's not about the tools).

On Tuesday, October 9, 2012 3:30:43 PM UTC+1, John Brookes wrote:

John Brookes

9 October 2012 16:30

Am I missing something?

I want to model a scenario:

I can write the individual steps quite easily, but neither Gherkin nor Behat appear to have any flow control or conditional operators.  I want as much written in 'English' as possible, so it can be maintained by my customers - when they change the wording of the messages for instance.

Is this possible?

If this is a question that's answered elsewhere, please point me there - I have looked but found nothing.

Regards

John

--
Konstantin Kudryashov
--- PHP developer, BDD evangelist, Scrum master

a: http://about.me/everzet
b: http://everzet.com
g: https://github.com/everzet

[John Brookes]

OK, so I do not know what I am doing, and I am turning to the experts for advice.  This is my first venture into BDD, TDD and structured testing, so perhaps I am mis-using and mis-understanding, and I hope you can help me find the best path.

I have been given the task of creating a test suite which can be used to fully test a web based system for user acceptance testing and to validate any updates post deployment.

The tests should be written in a language the customer can learn easily, as it is expected that the customer will want to modify the configuration post deployment.  This also means the test suite must be able to detect the current configuration and run the appropriate tests automatically, without failing.

Ideally, once the customer has learnt the language, it can be used to define future upgrades, so the customer and the developer both fully understand what is required.  This definition can then be converted into an acceptance test with minimal work.

I was advised that Behat was the best tool to use for this (and apart from the lack of conditions and flow control, it ticks all the boxes), but I am rapidly coming to the conclusion that there must be something better out there.  Can anybody here advise me on what other tools I should be looking at?

Regards

John

[Jelle Munk]

Hi John, 

I think you're at the right spot for your problem. I'm also new to BDD and behat so I'm no expert but I do the following to solve you're problem. 

One of the most important things with BDD/Behat is that all information is in the test. This way you never have to check if a specific resource or user exists in the application because you state in your test that it does. 

You can use background for this.

Background: Given The following users are registered

| username | password |

| foo           | bar          |

Then you create your scenario using the variables. 

Scenario: Given I am on Login page

When I fill foo in username

And I fill bar in password

And I click login

Then I should see 'welcome'

 
Backgrounds are run before a scenario so there you make sure you insert the user in the db. Then you use hooks to delete it after all tests are run. 

Does that make sense?

Cheers, Jelle

Op donderdag 11 oktober 2012 09:38:39 UTC+2 schreef John Brookes het volgende:

[John Brookes]

Hi Jelle, thanks for getting back to me.

The problem I have is that I am trying to test a service based structure where the database access layer is somewhere behind a Soap server, and the backend boys will not expose any functionality to let me directly access the database.

This imposes some restrictions:

The only way I can tell if an account exists is to try to create a duplicate.  If I succeed, the account exists.  If I fail, the account already exists and I just have to hope it has the correct password.

The only way I can tell if the password is correct is to try to log in; if I succeed, then the password is correct, otherwise - I am stuck.

So, I cannot just assert that the account exists, I have to try to create it, but a failure to create it does not mean there is a fault in the system, just in the test.

Incidentally, I have found a way round the initial problem, by returning PendingException from a step which has 'soft failed'.  This in effect breaks out of the running scenario without flagging a failure - it shows in the logs as 'todo', which we can set Jenkins to ignore.  When I get this fully tested and working, I will submit it through this group so others can see my solution.

Regards

John

[Jelle Munk]

Hi John, 

I think it's impossible to test this scenario, or extract any meaning out of it. Whenever something is wrong it always can be explained very logically so you still have to figure out if something really went wrong. And I think that was the purpose of writing the test :)

Why don't you create a test account and put the credentials in the test, you have to know the password of at least one account in order to test if you can login, you can't get around that one..

Cheers, 

Jelle 

Op vrijdag 12 oktober 2012 18:33:05 UTC+2 schreef John Brookes het volgende:

[John Brookes]

Hi Jelle

I am using the login as an example; the overall test plan is to create an account, log in, update the account details, log out, do some tests as a logged out user, log in, do some tests as a logged in user, delete account.  If something goes wrong with the login or delete scenarios, the account will not be removed, and the next iteration of the test will fail.


Feature: Post deployment testing of user accounts

  In order to verify that an end user can create an account, log in, update the account and delete the account

  As  a Jenkins server that has just deployed an update to the live server

  I need to roll back the update if the test returns an error exception.

  Scenario:  Create an account, which may need deleting and recreating if it already exists

    Given I am on the home page

    When I try to delete the account

    And I visit "Account/NewUser"

    And I enter my credentials

    And I press "Create Account"

    Then I should see "Your account has been created"

  Scenario:  Try to delete the account

    Given I am logged out

    When I try to log in as "Behat"

    Then I must not get an error if the account does not exist

    But I should break out and not run the following steps

    And I can see "My Account"

    And I follow "Close this account"

    And I press "Confirm Deletion"

    And I should see "Your account has been closed"

Unfortunately Gherkin does not allow nested scenarios, so 'Try to delete account' has to be written in PHP, like this:

    /**

     * Try to log in as arg1 -

     * If you can, delete the account

     *

     * @Given /^I try to delete account "([^"]*)"$/

     */

    public function iTryToDeleteAccount($arg1)

    {

        try {

            //  log out then try to log in

            //  Returns status of login or logout

            $this->iTryToLoginAs($arg1);

            //  are we logged in?

            $this->getMain()->iSeeTheStatusIs('login');

            $this->getMain()->assertPageContainsText("My Account");

            //  if so, go to my account page

            $this->getMain()->iClickOn("My Account");

            //  and close the account

            $this->getMain()->iClickOn("Close Account");

            $this->getMain()->iClickOn("Confirm Close Account");

            $this->getMain()->assertPageContainsText("Your account has been closed");

            return;

        }

        catch (Exception $exc) {

            # do nothing  

        }

        

        $this->getMain()->iSeeTheStatusIs('logout');

        // cannot log in so assume account does not exist

        $this->getMain()->setStatus('no account');

    }

    

The big problem with this is that I am writing PHP code, which needs testing - so every time I find an error, I spend an hour verifying that the problem is not in my PHP before I pass it on to the developers.

The more I look at this, the more time I spend trying to find an alternative, but Behat appears to be the only tool which takes story BDD and converts it into PHP tests.  If anybody knows an alternative - please pass it on.

Regards

John

On Friday, October 12, 2012 8:28:24 PM UTC+1, Jelle Munk wrote:

Hi John, 

[Marijn Huizendveld]

Hi John,

The problem is that you are trying to use Gherkin to write your tests.

Gherkin is a language that helps you describe your features in a way you can discuss them with your customers.

Behat helps you ensure that the features work as described by allowing you to write tests for the features.

As such, you should write features that take the perspective of the beneficiary of that feature.

I take it Jenkins is not a beneficiary.

I would suggest you read the RSpec book and watch the video of Everzet during Symfony Live.

It will give you some perspective how you should write your features and how you can best test them.

Cheers,

Marijn

[acoulton]

Hi John,

I actually think Behat can do what you want, but that you need to rethink how you're structuring the scenarios to start from the perspective of the story, ignoring constraints of your underlying system, and then implement what you need in the PHP step logic to make them work. I think you could express the requirement you posted above like this:

Feature: User account functionality is tested after deployment

  In order to verify that a deployment is successful

  As a system administrator

  I need to verify that a user can create an account, login, update the account and delete the account

 

Scenario: A user can create an account

  Given the user "notreg...@test.com" is not registered

  And I am on the homepage

  And I visit "Account/NewUser"

  And I enter my details

  When I press "Create Account"

  Then I should see "Your account has been created" 

 

Scenario: A user can login

  Given the user "regis...@test.com" is registered with the password "foobar"

  And I am on the homepage 

  And I visit "Login"

  When I press "Login"

  Then I should see "You have logged in"

Scenario: A registered user can delete their account

  Given the user "regis...@test.com" is registered with password "foobar"

  And I login as "regis...@test.com" with the password "foobar"

  And I am on the account page

  And I press "Close this Account"

  When I press "Confirm Deletion"

  Then I should be logged out 

  And the user "regis...@test.com" should not be registered

  And I should see "Your account has been closed

I think this structure is much more readable for the client and for you, as each scenario is self contained, isolated, and doesn't require any branching (which can be challenging for non-developers to follow in written form even when the language is designed, structured and formatted to support it). 

You then just need to implement /^Given the user "([^"]+)" is (not |)registered( with the password "[^"]+|)$/ and /^Then the user "([^"]+)" should (not |)be registered$/ which could either internally deal direct with an API to create and delete the test accounts or wrap the relevant steps to test for and create if necessary an account. You could also write an after-scenario hook to delete the test account(s) after the script successfully runs, or clear them out as part of your deploy script before running the test.

I'd say the fact that you are asking Jenkins to automate this test and rollback if required doesn't make them it the stakeholder - it's just acting on behalf of the system administrator.  Also defining it in this way means that the sysadmin can easily work through the same steps manually at any point and verify whether or not the system is performing as expected.

Cheers,

Andrew

[Konstantin Kudryashov]

Fully agree with Andrew.

acoulton

16 October 2012 11:25

John Brookes

15 October 2012 12:55

Jelle Munk

12 October 2012 21:28

Hi John, 

I think it's impossible to test this scenario, or extract any meaning out of it. Whenever something is wrong it always can be explained very logically so you still have to figure out if something really went wrong. And I think that was the purpose of writing the test :)

Why don't you create a test account and put the credentials in the test, you have to know the password of at least one account in order to test if you can login, you can't get around that one..

Cheers, 

Jelle 

Op vrijdag 12 oktober 2012 18:33:05 UTC+2 schreef John Brookes het volgende:

John Brookes

12 October 2012 18:33

Hi Jelle, thanks for getting back to me.

The problem I have is that I am trying to test a service based structure where the database access layer is somewhere behind a Soap server, and the backend boys will not expose any functionality to let me directly access the database.

This imposes some restrictions:

The only way I can tell if an account exists is to try to create a duplicate.  If I succeed, the account exists.  If I fail, the account already exists and I just have to hope it has the correct password.

The only way I can tell if the password is correct is to try to log in; if I succeed, then the password is correct, otherwise - I am stuck.

So, I cannot just assert that the account exists, I have to try to create it, but a failure to create it does not mean there is a fault in the system, just in the test.

Incidentally, I have found a way round the initial problem, by returning PendingException from a step which has 'soft failed'.  This in effect breaks out of the running scenario without flagging a failure - it shows in the logs as 'todo', which we can set Jenkins to ignore.  When I get this fully tested and working, I will submit it through this group so others can see my solution.

Regards

John

On Friday, October 12, 2012 4:15:25 PM UTC+1, Jelle Munk wrote:

Jelle Munk

12 October 2012 17:15

Hi John, 

I think you're at the right spot for your problem. I'm also new to BDD and behat so I'm no expert but I do the following to solve you're problem. 

One of the most important things with BDD/Behat is that all information is in the test. This way you never have to check if a specific resource or user exists in the application because you state in your test that it does. 

You can use background for this.

Background: Given The following users are registered

| username | password |

| foo           | bar          |

Then you create your scenario using the variables. 

Scenario: Given I am on Login page

When I fill foo in username

And I fill bar in password

And I click login

Then I should see 'welcome'

 
Backgrounds are run before a scenario so there you make sure you insert the user in the db. Then you use hooks to delete it after all tests are run. 

Does that make sense?

Cheers, Jelle

Op donderdag 11 oktober 2012 09:38:39 UTC+2 schreef John Brookes het volgende: