Linking or Chaining test cases

[Peter Wilson]

Hi,

New to Cucumber, please forgive my ignorance.

We have been writing cucumber test cases for about 3 months now at a system test level (for manual execution wile we got the automation framework up and running) and enjoying the clarity it provides, but now that we are starting to automate these tests we are finding some inefficiencies.  If we want to test an end to end flow and check lots of stuff along the way then cucumber doesn't seem really suited to the task.

We are thinking we will keep our manual cuces but when we automate, we'll create one super cuce that does many steps and many checks along the way

eg:

Given blah

When I login as a such and such user

Then the welcome page appears

And the user's name is shown

When I open the my policies link

Then all of my current policies are shown

But none of my expired policies are shown

When I select a policy

Then the details of my policy are shown

And the policy number matches the number on the policy list page

Each of these When and Then blocks matches an individual test case that is more story centric, but the above multi-when then case is much more efficient for automated regression execution because I don't need to restart the browser and get it into position for each individual cases, and I don't need to create new test data for each individual case, one set of data works for the whole long regression case.

I hope my ponderings make sense, feedback would be appreciated.

Cheers

Peter

[Andrew Premdas]

This is a really really bad idea, which has been discussed in various forms on this mailing list so many times. When your 'super cuke' fails on line 465 and you have to spend 3 days working out what was wrong then you'll truly understand.

All best

Andrew

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Peter Wilson]

Thanks for the reply.  Can you tell me what I can search for to find these discussions?  I got one using "super cuke" and not much with "end to end".  

I see what you mean, any break in the chain would mean dependent tests could not be run which would make us blind to all post-break functionality.

[Peter Wilson]

Another possible solution (that you may have already covered), is there a way in cucumber to express an expected "after" state like given, except perhaps something like "now", once the test has completed?  That way cucumber could match the givens with the afters and just chain tests by itself, but if there was no matching given/after, just run the givens on the test.  This would be pretty powerful and efficient, might impact the product owner's view a bit, but you can't have everything in life :)

For example:

Scenario: Valid Login

Given the logon page is displayed

When I log in with as a valid customer

Then the landing page is displayed

Now I am on the landing page

Scenario: View Special

Given I am on the landing page

When I view a product that is on special

Then the full product description and price is displayed

Now I am viewing a product description

The cucumber framework can now chain these two independent scenarios together (as long as the first scenario passes). It could develop graphs of scenarios based on what is working and what is not.  It could run quickly and re-use test data and still preserve the atomic flexibility that you implied in your reply.

Has this possibility already been discussed and discarded?

Cheers

Peter

[Andrew Premdas]

On 27 July 2014 23:43, Peter Wilson <peter....@gmail.com> wrote:

Another possible solution (that you may have already covered), is there a way in cucumber to express an expected "after" state like given, except perhaps something like "now", once the test has completed?  That way cucumber could match the givens with the afters and just chain tests by itself, but if there was no matching given/after, just run the givens on the test.  This would be pretty powerful and efficient, might impact the product owner's view a bit, but you can't have everything in life :)

For example:

Scenario: Valid Login

Given the logon page is displayed

When I log in with as a valid customer

Then the landing page is displayed

Now I am on the landing page

Scenario: View Special

Given I am on the landing page

When I view a product that is on special

Then the full product description and price is displayed

Now I am viewing a product description

The cucumber framework can now chain these two independent scenarios together (as long as the first scenario passes). It could develop graphs of scenarios based on what is working and what is not.  It could run quickly and re-use test data and still preserve the atomic flexibility that you implied in your reply.

Has this possibility already been discussed and discarded?

Cheers

Peter

Your Now steps are exactly the same as your Then steps. Why do something twice.

What benefit do you get from chaining the tests together. Surely you realise that the combinatorial explosion you would get even after just creating a few tests would mean your test suite would go on for a very long time.  Features are not about exhaustive testing or 'proving' that something works, that is beyond their capability.

All best

Andrew

On Friday, July 25, 2014 3:29:40 PM UTC+10, Peter Wilson wrote:

Hi,

New to Cucumber, please forgive my ignorance.

We have been writing cucumber test cases for about 3 months now at a system test level (for manual execution wile we got the automation framework up and running) and enjoying the clarity it provides, but now that we are starting to automate these tests we are finding some inefficiencies.  If we want to test an end to end flow and check lots of stuff along the way then cucumber doesn't seem really suited to the task.

We are thinking we will keep our manual cuces but when we automate, we'll create one super cuce that does many steps and many checks along the way

eg:

Given blah

When I login as a such and such user

Then the welcome page appears

And the user's name is shown

When I open the my policies link

Then all of my current policies are shown

But none of my expired policies are shown

When I select a policy

Then the details of my policy are shown

And the policy number matches the number on the policy list page

Each of these When and Then blocks matches an individual test case that is more story centric, but the above multi-when then case is much more efficient for automated regression execution because I don't need to restart the browser and get it into position for each individual cases, and I don't need to create new test data for each individual case, one set of data works for the whole long regression case.

I hope my ponderings make sense, feedback would be appreciated.

Cheers

Peter

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Wilson]

The benefit would be quicker execution time.  This would be due to two reasons

1. Test data doesn't need to be created for each test case.  If chained then the next test can just pick up where the last left off

2. No need to redo step that have already been done to get to a "given" state.

For example, imagine if a test case needs to go through screen A, B C and D to get to screen E to do its test, the "Given" then has to create the necessary test data, and navigate to screen E and then the test can start.  Now if we already have tests that look at screens A, B C and D, then we run those first and then run our next test.  It just picks up where the test for screen D left off.  Much quicker... yes?

That's the only benefit I can see.  

So as it currently works, Cucumber would have to Do a test for screen A, blow away the browser and test data, Create new test data for screen B, navigate screen A to B, test then blow away test data and browser, then create new test data, navigate screens A, B and to C, test screen C and so on.  Seems a bit wasteful when gui automation is pretty slow on execution. 

Still finding my way, so feedback appreciated.

[Andrew Premdas]

On 28 July 2014 01:44, Peter Wilson <peter....@gmail.com> wrote:

The benefit would be quicker execution time.  This would be due to two reasons

1. Test data doesn't need to be created for each test case.  If chained then the next test can just pick up where the last left off

2. No need to redo step that have already been done to get to a "given" state.

For example, imagine if a test case needs to go through screen A, B C and D to get to screen E to do its test, the "Given" then has to create the necessary test data, and navigate to screen E and then the test can start.  Now if we already have tests that look at screens A, B C and D, then we run those first and then run our next test.  It just picks up where the test for screen D left off.  Much quicker... yes?

No, a bit quicker, but the costs are catastrophic

That's the only benefit I can see.  

So as it currently works, Cucumber would have to Do a test for screen A, blow away the browser and test data, Create new test data for screen B, navigate screen A to B, test then blow away test data and browser, then create new test data, navigate screens A, B and to C, test screen C and so on.  Seems a bit wasteful when gui automation is pretty slow on execution. 

This is because you are thinking like a programmer, and a programmer who is committing one of the cardinal programming sins of optimizing to early.

When using Cucumber you need to first think like the business. So stop thinking about screens and think about functionality; or to put it another way forget about the HOW and think about the WHAT and WHY.

With you examples of A B C and D. We have that each thing is important to the business, so we would like something that tells us if they are broken and gives us some confidence that they are working. So we have a feature for each

Feature A, Feature B, ...

Now we have lots of possible combinations, some that you won't have thought about e.g. ABA ABC ABCD, DCBA, AAABCE ad infinitum. You can't test all of these its just not practical. Even if you had super quick tests and great hardware what would you do when ABCDBCDAfailBC. See http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/ for more on this

<aside>

We have to realise that features (acceptance tests), whatever you want to call them DO NOT prove that your application works. In some ways they should not even be considered to be tests, perhaps we need a new word for them. What they do is check that things are probably OK'ish.  Despite this they have great benefits one being they are good for regression, but even here when features pass the best they can do is say something like

"Things are probably OK'ish, you probably haven't broken anything obvious"

</aside>

Your main use case is when you get a business feature that is a superset of a previous business. So we have D which builds on A and B, and perhaps adds some component. Here you still want to run A and B features in isolation, becuase when you break something they will narrow down your target. When you implement the 'check' that D works you can leverage the stuff you've done with A and B so you can write the check much faster (a good thing). However trying to save run time by not executing the A and B multiple times is a very bad thing.

If you understand my aside. You will realise that each check/test we write has a complex cost/benefit relationship which we should implicitely understand in the context of our particular business. Generally we check/test the happy path, and check/test a sad path or two for each bit of business functionality. Doing more (with features/acceptance tests) is generally counter productive.

If we need more we have to go down to Unit testing which is totally different from writing features. When Unit testing it is possible, maybe even reasonable to say things like 'test everything that can possibly break'.  With features such a statement is complete nonsense.

Finally the point where features become really useful is when they fail. But if you write features in the style you are proposing they will

1. Be much less useful when they fail

2. Sometimes they will fail when the application is actually fine

Disrupting the utility of a failing feature (or unit test) is just about the biggest mistake you can make in testing.

All best

Andrew