Cucumber Code Re-use / Data Passing

[dgoodwin]

We've been struggling quite a bit with our usage of Cucumber,
specifically how to organize/re-use code, and essentially pass data
between "Given" and "When" and "Then" statements.

Our application is a REST webapp, so it's somewhat technical in nature
and not something a user would interact with directly. It involves
resources such as Organizations, Users, Subscriptions, and Products.

Testing thus involves some complex setup, at times we need multiple
instances of one resource to test behaviour, such as a User in one
Organization cannot view resources in another Organization. Our
understanding was this should happen in a "Given" statement, we should
ideally be re-using one step definition to create the each resource,
but then how do we differentiate between them in a "When" or a "Then",
when all step definitions and variable names are essentially globals.

The solution we're drifting towards is maintaining global hashes, one
for each resource type, where we store every entity we're creating by
some key defined in the feature. That key can then be used in "When"
and "Then" statements to reference a specific resource.

This however concerned us, as it feels like we are "meta-programming"
in the feature files, in a rather clunky and unpleasant manner. The
keys are essentially variable names, the hashes are a substitution for
class/method scoping, etc.

An example:

Feature: Manipulate Subscriptions
As a user
I can view and modify my subscription data

Background:
Given an owner admin "testowner"
And I am logged in as "test_owner"
And product "some_product" exists
And product "another_product" exists
And product "one_more_product" exists
And product "monitoring" exists

Scenario: List existing subscriptions
Given test owner has 2 entitlements for "some_product"
And test owner has 3 entitlements for "another_product"
And test owner has 2 entitlements for "one_more_product"
When I am logged in as "testowner"
Then I have 3 subscriptions

Scenario: Delete a subscription
Given test owner has 5 entitlements for "monitoring"
When I am logged in as "testowner"
And I delete the subscription for product "monitoring"
Then I have 0 subscriptions

This is a relatively tame example (we have others with much more
complex setup) but it highlights some of the problems we're having.
We're creating multiple products with one step definition, after which
we reference them by a key originating from the feature. We often have
multiple Organizations and Users around, and have to switch back and
forth with respect to who is authenticated.

Are we doing something fundamentally wrong? Is this approach of global
hashes + variable names common? In general, how do you pass specific
data between a Given and a When/Then when working with what is
essentially a flat global namespace of step definitions and variables?

Any help/advice would be greatly appreciated, thanks!

Devan

[Matt Wynne]

Most people keep state between steps using instance variables, but I've seen it done using a hash before. Keeping the keys visible in the steps as you seem to be doing is a good idea so there's no magic going on that you can't see from the features. I think you're basically taking the right approach.

I have a couple of thoughts. One is that you seem to have quite a lot of data floating around in your scenarios from the Background, which isn't being used in all of the scenarios. I wonder if that's contributing to the problem. Could it help to either split the feature (making list and delete their own smaller, more focussed features) or to simply move some of the steps out of the background back into the scenarios they're actually used in?

Another question I have which I'm afraid isn't clear to me from your post is exactly what pain you're feeling from this. Are you just losing confidence in your approach, or are you actually finding things hard to read and maintain? Or some other problem?

>
> Any help/advice would be greatly appreciated, thanks!
>
> Devan
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "Cukes" group.
> To post to this group, send email to cu...@googlegroups.com.
> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cukes?hl=en.
>

cheers,
Matt

http://blog.mattwynne.net
+44(0)7974 430184

[dgoodwin]

On Jul 20, 8:44 pm, Matt Wynne <m...@mattwynne.net> wrote:

> Most people keep state between steps using instance variables, but I've seen it done using a hash before. Keeping the keys visible in the steps as you seem to be doing is a good idea so there's no magic going on that you can't see from the features. I think you're basically taking the right approach.

Thanks for the feedback.

We got a little nervous with this hash approach as it felt like
variable names (essentially) were leaking into the feature files, but
it's good to know we're not out in left field.

We weren't real fond of the instance variables as it seems like this
couples your step definitions together. Because those are essentially
in a global namespace (any regex could match in any file), the process
of tracking down the dependencies for a step you wish to re-use is a
little scary, as is the possibility of unintentionally clobbering an
instance variable you didn't intend to.

>
> I have a couple of thoughts. One is that you seem to have quite a lot of data floating around in your scenarios from the Background, which isn't being used in all of the scenarios. I wonder if that's contributing to the problem. Could it help to either split the feature (making list and delete their own smaller, more focussed features) or to simply move some of the steps out of the background back into the scenarios they're actually used in?

True, we could definitely split the setup up into more features quite
a bit I think.

>
> Another question I have which I'm afraid isn't clear to me from your post is exactly what pain you're feeling from this. Are you just losing confidence in your approach, or are you actually finding things hard to read and maintain? Or some other problem?

We struggle with a few issues, but the above problem with simply
passing data from a Given to When/Then while still re-using steps and
not creating a regex mess, is a big piece of it. To some extent we're
concerned we're inventing bad solutions to workaround the absence of
classes, fixtures, methods, variables and scoping, etc.

Any thoughts on the Background test setup in general? In our case
we're leaking quite a bit of this into feature files when really it
has little to do with the behaviour we want to test. But because the
before/after blocks in step definitions are globally run with every
scenario in every feature (not really desired), this seems like the
only place it can go?

Some team members also find the process of text -> regex -> code
unpleasant to write, re-use, and maintain, but that's not a universal
opinion and probably just a fact of life if you're using cukes.

Thanks again!

Cheers,

Devan

[Matt Wynne]

On 21 Jul 2010, at 01:54, dgoodwin wrote:

>
>
> On Jul 20, 8:44 pm, Matt Wynne <m...@mattwynne.net> wrote:
>
>> Most people keep state between steps using instance variables, but I've seen it done using a hash before. Keeping the keys visible in the steps as you seem to be doing is a good idea so there's no magic going on that you can't see from the features. I think you're basically taking the right approach.
>
> Thanks for the feedback.
>
> We got a little nervous with this hash approach as it felt like
> variable names (essentially) were leaking into the feature files, but
> it's good to know we're not out in left field.
>
> We weren't real fond of the instance variables as it seems like this
> couples your step definitions together. Because those are essentially
> in a global namespace (any regex could match in any file), the process
> of tracking down the dependencies for a step you wish to re-use is a
> little scary, as is the possibility of unintentionally clobbering an
> instance variable you didn't intend to.
>
>>
>> I have a couple of thoughts. One is that you seem to have quite a lot of data floating around in your scenarios from the Background, which isn't being used in all of the scenarios. I wonder if that's contributing to the problem. Could it help to either split the feature (making list and delete their own smaller, more focussed features) or to simply move some of the steps out of the background back into the scenarios they're actually used in?
>
> True, we could definitely split the setup up into more features quite
> a bit I think.
>
>>
>> Another question I have which I'm afraid isn't clear to me from your post is exactly what pain you're feeling from this. Are you just losing confidence in your approach, or are you actually finding things hard to read and maintain? Or some other problem?
>
> We struggle with a few issues, but the above problem with simply
> passing data from a Given to When/Then while still re-using steps and
> not creating a regex mess, is a big piece of it. To some extent we're
> concerned we're inventing bad solutions to workaround the absence of
> classes, fixtures, methods, variables and scoping, etc.

You do realise you can use your own classes and method in the step definitions don't you? Have a look at libraries like webrat and aruba for some very generic implementations of this idea. I guess what you've done with your hash is a start for this, but you could build up your own DSL library specific to your product, and have the step definitions depend on that. This wiki article will also give you some tips:

http://wiki.github.com/aslakhellesoy/cucumber/a-whole-new-world

Or, are you saying you'd prefer to be able to model each step definition as a separate class instance? That's an interesting idea.

Obviously, as you've said, all the step definitions currently run in the same namespace, so feasibly any of them could be running in any given scenario. We've talked about being able to use tags to namespace step definitions, so that you could ringfence off groups of step definitions to only be available for scenarios with certain tags. Would that help you?

> Any thoughts on the Background test setup in general? In our case
> we're leaking quite a bit of this into feature files when really it
> has little to do with the behaviour we want to test. But because the
> before/after blocks in step definitions are globally run with every
> scenario in every feature (not really desired), this seems like the
> only place it can go?

Tags are probably the answer here. Have you seen how the @javascript tag in Capybara works? You can specify Before / After hooks per tag, so run can attach setup code to certain scenarios by tagging them.

> Some team members also find the process of text -> regex -> code
> unpleasant to write, re-use, and maintain, but that's not a universal
> opinion and probably just a fact of life if you're using cukes.

The idea ought to be that you spend less and less time in the step defs / regex because you've built up a nice re-usable library of step definitions. Certainly on the biggest project I've built with Cucumber, we saw the rate of step definition creation flatten off a great deal once we got up and running. I wonder whether part of this is just a reflection of the maintenance headaches you're experiencing with the step definitions.

>
> Thanks again!
>
> Cheers,

[dgoodwin]

On Jul 21, 7:05 am, Matt Wynne <m...@mattwynne.net> wrote:
> On 21 Jul 2010, at 01:54, dgoodwin wrote:
>
> You do realise you can use your own classes and method in the step definitions don't you? Have a look at libraries like webrat and aruba for some very generic implementations of this idea. I guess what you've done with your hash is a start for this, but you could build up your own DSL library specific to your product, and have the step definitions depend on that. This wiki article will also give you some tips:
>
> http://wiki.github.com/aslakhellesoy/cucumber/a-whole-new-world
>
> Or, are you saying you'd prefer to be able to model each step definition as a separate class instance? That's an interesting idea.

No, more so about how in the past we've become accustomed to using a
unit test framework for writing our functional tests. So our test code
would be organized into a test suite and cases, with a defined setup/
teardown, inheritance, and scope we're familiar with from regular
coding.

>
> Obviously, as you've said, all the step definitions currently run in the same namespace, so feasibly any of them could be running in any given scenario. We've talked about being able to use tags to namespace step definitions, so that you could ringfence off groups of step definitions to only be available for scenarios with certain tags. Would that help you?

Probably not a terrible amount of help with the step definitions but I
could definitely see that being useful below where you mention tagging
the before/after blocks.

>
> > Any thoughts on the Background test setup in general? In our case
> > we're leaking quite a bit of this into feature files when really it
> > has little to do with the behaviour we want to test. But because the
> > before/after blocks in step definitions are globally run with every
> > scenario in every feature (not really desired), this seems like the
> > only place it can go?
>
> Tags are probably the answer here. Have you seen how the @javascript tag in Capybara works? You can specify Before / After hooks per tag, so run can attach setup code to certain scenarios by tagging them.
>
> > Some team members also find the process of  text -> regex -> code
> > unpleasant to write, re-use, and maintain, but that's not a universal
> > opinion and probably just a fact of life if you're using cukes.
>
> The idea ought to be that you spend less and less time in the step defs / regex because you've built up a nice re-usable library of step definitions. Certainly on the biggest project I've built with Cucumber, we saw the rate of step definition creation flatten off a great deal once we got up and running. I wonder whether part of this is just a reflection of the maintenance headaches you're experiencing with the step definitions.

Quite possible we just haven't reached that point yet.

Thanks for the suggestions!

Cheers,

Devan

[Matt Wynne]

On 21 Jul 2010, at 12:21, dgoodwin wrote:

> On Jul 21, 7:05 am, Matt Wynne <m...@mattwynne.net> wrote:
>> On 21 Jul 2010, at 01:54, dgoodwin wrote:
>>
>> You do realise you can use your own classes and method in the step definitions don't you? Have a look at libraries like webrat and aruba for some very generic implementations of this idea. I guess what you've done with your hash is a start for this, but you could build up your own DSL library specific to your product, and have the step definitions depend on that. This wiki article will also give you some tips:
>>
>> http://wiki.github.com/aslakhellesoy/cucumber/a-whole-new-world
>>
>> Or, are you saying you'd prefer to be able to model each step definition as a separate class instance? That's an interesting idea.
>
> No, more so about how in the past we've become accustomed to using a
> unit test framework for writing our functional tests. So our test code
> would be organized into a test suite and cases, with a defined setup/
> teardown, inheritance, and scope we're familiar with from regular
> coding.

Okay. Remember each scenario runs inside an instance of an object (known as the World) which is destroyed at the end of the scenario. You can mix in specific behaviour to the world based on tags using the before hooks:

Before('@authentication') do
World(AuthenticationHelper)
end

It's admittedly not as neat as having classes in xunit, but it might help give you some help to keep on top of the dependencies between steps.

The other thing you could consider doing is breaking up your actual features folder into separate folders / test runs. You can put common step definition code into a shared folder and require it from the step_definitions folders of the separated features folders. This might again give you a tool to understand and clarify where the seams / dependencies are.

>
>>
>> Obviously, as you've said, all the step definitions currently run in the same namespace, so feasibly any of them could be running in any given scenario. We've talked about being able to use tags to namespace step definitions, so that you could ringfence off groups of step definitions to only be available for scenarios with certain tags. Would that help you?
>
> Probably not a terrible amount of help with the step definitions but I
> could definitely see that being useful below where you mention tagging
> the before/after blocks.
>
>>
>>> Any thoughts on the Background test setup in general? In our case
>>> we're leaking quite a bit of this into feature files when really it
>>> has little to do with the behaviour we want to test. But because the
>>> before/after blocks in step definitions are globally run with every
>>> scenario in every feature (not really desired), this seems like the
>>> only place it can go?
>>
>> Tags are probably the answer here. Have you seen how the @javascript tag in Capybara works? You can specify Before / After hooks per tag, so run can attach setup code to certain scenarios by tagging them.
>>
>>> Some team members also find the process of text -> regex -> code
>>> unpleasant to write, re-use, and maintain, but that's not a universal
>>> opinion and probably just a fact of life if you're using cukes.
>>
>> The idea ought to be that you spend less and less time in the step defs / regex because you've built up a nice re-usable library of step definitions. Certainly on the biggest project I've built with Cucumber, we saw the rate of step definition creation flatten off a great deal once we got up and running. I wonder whether part of this is just a reflection of the maintenance headaches you're experiencing with the step definitions.
>
> Quite possible we just haven't reached that point yet.
>
> Thanks for the suggestions!
>
> Cheers,
>
> Devan
>

[dgoodwin]

On Jul 21, 9:09 am, Matt Wynne <m...@mattwynne.net> wrote:

> Okay. Remember each scenario runs inside an instance of an object (known as the World) which is destroyed at the end of the scenario. You can mix in specific behaviour to the world based on tags using the before hooks:
>
> Before('@authentication') do
>   World(AuthenticationHelper)
> end
>
> It's admittedly not as neat as having classes in xunit, but it might help give you some help to keep on top of the dependencies between steps.
>
> The other thing you could consider doing is breaking up your actual features folder into separate folders / test runs. You can put common step definition code into a shared folder and require it from the step_definitions folders of the separated features folders. This might again give you a tool to understand and clarify where the seams / dependencies are.

I think we totally missed the existence of tags, I thought those were
a theoretical feature to be. Thanks!

[Andrew Premdas]

Generally there are three ways to communicate between steps in features

1. Use an identifier in the feature e.g. Given there is a color "Red"

2. Create a variable in the step definition e.g. Given there is a color (create @color in you step definition)

3. Only have one of the thing you are working with e.g. Given there is a color (Use Color.last or something similar in your step definition)

There is no outright winner between these 3 IMO, although the second option probably has the most significant downsides in general.

I would recommend using only one approach in each feature, but also believe that each approach has its place. In particular alot of features using (1), like your example feature, become very wordy.

Another thing I notice with your example feature is that it is not focused on what the feature is about. The feature says its about a user manipulating subscriptions. If this is the case why are the scenarios talking about entitlements, products etc. If you want to write a technical feature explicitly describing the relationship between entitlements then do so, but not here. Following this your feature becomes:

Feature: Manipulate Subscriptions
   As a user
   I can view and modify my subscription data

  Scenario: List existing subscriptions

      Given I have 2 subscriptions

      When I view my subscriptions

      Then I should see a list of 2 subscriptions

   Scenario: Delete a subscription

      Given I have 2 subscriptions

      When I view my subscriptions

      And I delete a subscription

      Then I should have one subscription

All the other stuff is irrelevant to this feature and can be put in the step definitions (which because they are Ruby are far more expressive and flexible). Note how easy it is to communicate between the Given and the When, you only have one user in this scenario, so you can just use User.last (or something equivalent)

HTH

Andrew