How to set and clear test data before and after all scenarios run

[Yianna]

Hello,

This question is for Cucumber-JVM
I'm using @before and @after annotations for setting up some test
data but I realised that they are been invoked independently for each
scenario
even if they are implemented in the same file.
Do you know how I can structure my code so that before and after are
only been invoked once for many scenarios?

Cheers

[aslak hellesoy]

On Tue, Feb 28, 2012 at 3:09 PM, Yianna <yiann...@gmail.com> wrote:
> Hello,
>
> This question is for Cucumber-JVM
>  I'm using @before and @after annotations for setting up some test
> data but I realised that they are been invoked independently for each
> scenario
> even if they are implemented in the same file.

Yes. Before and After hooks run before and after each and every scenario.

> Do you know how I can structure my code so that before and after are
> only been invoked once for many scenarios?
>

What is it that you want to do once, and why do you want to do it only once?

Cheers,
Aslak

> Cheers
>
> --
> You received this message because you are subscribed to the Google Groups "Cukes" group.
> To post to this group, send email to cu...@googlegroups.com.
> To unsubscribe from this group, send email to cukes+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cukes?hl=en.
>

[Jon Kern]

have you explored DatabaseCleaner and Cucumber?

you can also google for "cucumber seed data"

jon

blog: http://technicaldebt.com
twitter: http://twitter.com/JonKernPA

Yianna said the following on 2/28/12 10:09 AM:

[David Kowis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/28/2012 01:18 PM, Jon Kern wrote:
> have you explored DatabaseCleaner and Cucumber?

I haven't tried using that stuff with the cucumber-jvm jruby backend,
I don't know how it'd behave. It might work, but the necessary hooks
might be missing. It'd also depend on how DatabaseCleaner talks to the
database and such.

Good idea for ruby, but it might not work in cucumber-jvm.

David

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGcBAEBAgAGBQJPTSqxAAoJEMnf+vRw63ObUncL/2+1uJ48bDgDI2AbHF7nY3bX
T3P+Hn86y0ocDjhc8YLTirNPol2UF9trWqQzGTD4jMbNXiGiDqBxjxdija4//N/Z
cGs/Y91XGG51atAhKLUxcyk1BLJq3bynyS+MeACfS/i0VC2ryUjnT5l4/FaTnnD7
0mp89DWbtUUm7nV3+ECAtzXSuc+KbYubguhPHyTLruCPGcWHe91dUUc+8mxr4cK+
KjbjYm/Sz3R2Ju/zdwCGFbxpvfJgA4YjtrOJYTdLtjJ/nzxeSd3jP95SOC9Pn4CD
u8J0y30FQxASR3+Ph7APmd5rh5nY6MG7ninsaySCQHBWnCLAHmfFgpnZ7pV97nUE
9iaILf9gq/MN2lwpLqrpnmmitfvxWxLYWiRyytUI+Xs/B2QUmmg0uM8mBNBv9BcF
exgrjr9LVK7xh1CuR3NcM4NHSo1Pc9sUpfbWre2DXmPsLLZLj7l5pOPh2JTCB+gI
qdGeDEru55ElZM0knJI9ap0EugUM6GX9vNx8EsFvDA==
=ox10
-----END PGP SIGNATURE-----

[Matthew]

In the java world you could try DBUnit - it hasn't had much done to it
recently, but seems to work.

I think it actually helps having the before and after run for each
scenario, in that it then lets you setup and tear down the data
independently for each one. What we're trying is using a counter to
tell us which scenario we're in in the feature, and then loading the
appropriate test data in the Before. We then remove it in the After
(we're actually using the XLS loader to load the data).

This works, although I do wonder whether it would be better to define
'standard' data chunks by name in the Given, and load it there - but
we went with the before for now.

Having said all that there is also the thought that if possible it
might be best to create any data through the system itself (rather
than put stuff direct into the db), so that you know the data you have
has the correct format as far as the app is concerned.

Hope that helps,

Matt

> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/

[Keith Bennett]

Hi, Aslak. We met at CukeUpNYC a couple of months ago.

We're using Cucumber to test network servers.  There's some time consuming automated server setup that needs to happen before the test suite is run, and we'd like to do some time consuming cleanup afterwards as well.  Doing so before and after each scenario would increase the test of each cost so high as to be prohibitive, resulting in a drastic reduction of either the number of tests or the frequency with which we run them, or both.

We understand the reasoning behind not providing 'before/after all' functionality, but our reality is that the automated tests will not be practical without it.  Without a Cucumber feature to accommodate this cleanly, we have to create ugly hacks, like using global variables, or class variables to carry state across test runs (which is probably a much worse test practice than one time setup or teardown).

Why not add the capability, and let us take responsibility for using it wisely?

- Keith

[George Dinwiddie]

Keith,

> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style

fixing this before replying

On 11/22/13 5:33 PM, Keith Bennett wrote:
>>
>> On Tuesday, February 28, 2012 10:27:17 AM UTC-5, Aslak Hellesøy wrote:
>>
>> On Tue, Feb 28, 2012 at 3:09 PM, Yianna <yiann...@gmail.com
>> <javascript:>> wrote:
>> > Hello,
>> >
>> > This question is for Cucumber-JVM
>> > I'm using @before and @after annotations for setting up some test
>> > data but I realised that they are been invoked independently for each
>> > scenario
>> > even if they are implemented in the same file.
>>
>> Yes. Before and After hooks run before and after each and every
>> scenario.
>>
>> > Do you know how I can structure my code so that before and after are
>> > only been invoked once for many scenarios?
>> >
>>
>> What is it that you want to do once, and why do you want to do it
>> only once?
>>
>> Cheers,
>> Aslak
>

> Hi, Aslak. We met at CukeUpNYC a couple of months ago.
>
> We're using Cucumber to test network servers. There's some time
> consuming automated server setup that needs to happen before the
> test suite is run, and we'd like to do some time consuming cleanup
> afterwards as well. Doing so before and after each scenario would
> increase the test of each cost so high as to be prohibitive,
> resulting in a drastic reduction of either the number of tests or the
> frequency with which we run them, or both.
>
> We understand the reasoning behind not providing 'before/after all'
> functionality, but our reality is that the automated tests will not
> be practical without it. Without a Cucumber feature to accommodate
> this cleanly, we have to create ugly hacks, like using global
> variables, or class variables to carry state across test runs (which
> is probably a much worse test practice than one time setup or
> teardown).

Are you having a hard time figuring out how to do this in your code?

Hint 1: It's been shown before on this mailing list.

Hint 2: You can have a setup step that initializes the first time it is
needed, setting a flag or populating a variable (which doesn't have to
be global in Java; it can be a class variable) to avoid re-initializing.
This step can also install a shutdown hook (See
http://docs.oracle.com/javase/7/docs/technotes/guides/lang/hook-design.html)
to clean up on termination.

If these hints aren't enough, perhaps you can put an example of your
code and tests in gist or github and others can give more detailed
suggestions.

Are you suggesting that an "ugly hack" should be added to Cucumber to
accommodate a special case that is often not needed? This need is really
about your context, not the general need of testing. It's really better
handled for the specific need.

> Why not add the capability, and let us take responsibility for using
> it wisely?

Do you mean this question as stated? I think Aslak has said many times
why he won't add this to Cucumber. And your comment about taking
responsibility would be more credible if you'd read the list rules
before top-posting. ;-)

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Paolo Ambrosio]

There is an open issue about it, with alternatives:
https://github.com/cucumber/cucumber-jvm/issues/515

Another alternative is to use a DI framework like Spring and have
singleton beans with init and destroy methods. An example of this
approach in src/test/resources/jetty-context.xml (for web server
startup and shutdown) here:
https://bitbucket.org/paoloambrosio/web-java/

> Hint 2: You can have a setup step that initializes the first time it is
> needed, setting a flag or populating a variable (which doesn't have to be
> global in Java; it can be a class variable) to avoid re-initializing. This
> step can also install a shutdown hook (See
> http://docs.oracle.com/javase/7/docs/technotes/guides/lang/hook-design.html)
> to clean up on termination.
>
> If these hints aren't enough, perhaps you can put an example of your code
> and tests in gist or github and others can give more detailed suggestions.
>
> Are you suggesting that an "ugly hack" should be added to Cucumber to
> accommodate a special case that is often not needed? This need is really
> about your context, not the general need of testing. It's really better
> handled for the specific need.
>
>> Why not add the capability, and let us take responsibility for using
>> it wisely?

I've accepted a pull request for BEFORE/AFTER_ALL hooks in
Cucumber-CPP because a lot of users keep asking it for Cucumber-JVM. I
don't consider the current alternative a hack, but I understand it can
be confusing for some users, expecially coming from other frameworks
that provide it out of the box.

The best chance to have this included in Cucumber-JVM is to send a
pull request, as Aslak wrote on ticket #515. After all, if it's a
feature you want implemented, why should someone else spend time
implementing it? ;-)

> Do you mean this question as stated? I think Aslak has said many times why
> he won't add this to Cucumber. And your comment about taking responsibility
> would be more credible if you'd read the list rules before top-posting. ;-)

:-D

> - George
>
> --
> ----------------------------------------------------------------------
> * George Dinwiddie * http://blog.gdinwiddie.com
> Software Development http://www.idiacomputing.com
> Consultant and Coach http://www.agilemaryland.org
> ----------------------------------------------------------------------
>
>

> --
> -- Rules --
>
> 1) Please prefix the subject with [Ruby], [JVM] or [JS].

> 2) Please use interleaved answers
> http://en.wikipedia.org/wiki/Posting_style#Interleaved_style

> 3) If you have a question, don't reply to an existing message. Start a new
> topic instead.

>
> You received this message because you are subscribed to the Google Groups
> Cukes group. To post to this group, send email to cu...@googlegroups.com. To
> unsubscribe from this group, send email to
> cukes+un...@googlegroups.com. For more options, visit this group at

> https://groups.google.com/d/forum/cukes?hl=en
> --- You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cukes+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Keith Bennett]

George, Paolo -

Firstly, George, thank you for the warm welcome! :)

(Please see my response at bottom.)

George, Paolo -

I apologize for not familiarizing myself with the rules of the forum. In addition, my question was about both standard Ruby Cucumber and Cucumber-JVM (much more so, the former), and I should have made that clear.

Regarding interleaving, I do usually use interleaving where it makes sense, but didn't do so in this case, thinking that my comment referred to the broad subject of the thread, and not any parts in particular.

I further apologize if the subject of before_all was already asked and answered, and I wasn't aware of it, or was and forgot.

I do appreciate the toil and effort of open source contributors, and Aslak has given us something of great value, for which we should all be grateful.

* * *

First, an explanation of terms. Whatever I say about before_all, I would also like to apply to after_all. Also, by before_all, I don't mean "before all tests of the Cucumber run", but rather "before all tests in a given feature file"...like the RSpec before(:all) and describe blocks, for which I provide an illustration at https://gist.github.com/keithrbennett/7712890. We need this because we have different setup for different types of tests, but still want to perform all tests in a single Cucumber run.

George, I disagree with your assertion that this is "a special case that is not often needed".  There does seem to be significant interest in this feature, and you might want to count not only current Cucumber users, but others who may have decided not to use Cucumber, in part at least because this was a need they had. It may very well be that Cucumber is targeted at a specific group, and I acknowledge that it's not my place to have any expectations otherwise, but I just wanted to express my view so that it is available for consideration.

* * *

Regarding implementing before_all functionality myself, I now have a working solution in my Ruby/Cucumber testing.  I use a class method (so that it is globally available, but does not pollute the global namespace).  This method manages a set of initialization group keys internally (currently, I use __FILE__ as the key, but anticipate needing to be more precise later). These keys are needed because the setup and teardown may need to be done multiple times in a given Cucumber run (e.g. once for each of 2 feature files).  So I can call MyInitializationClass.init_once(__FILE__), and the initialization will only be performed if it has not yet been sent that key. I think this is the best I'm going to get for my needs, but would appreciate any suggested improvements.

My problem with this and any other solution I can think of is that, in my opinion, all of them are less intention revealing, more complex, more verbose, and/or more error prone than a built-in Cucumber before_all would be.

* * *

Paolo, regarding your statement that if I want it, I should write it myself, you're right, of course.  However, this unfortunately is not likely to happen, since I suspect the time I would need to invest just to understand the source base well enough to present an intelligent solution would probably be prohibitive for me.  In contrast, someone already familiar with the code base would probably know exactly where this modification should go, what gotchas to look out for, etc.  In addition, even if I were to provide a patch, it might still not be accepted, either because it was not a good technical fit, or there may be a philosophical opposition to it. I do realize that this is open source software and I am not a paying customer, so can't complain if I don't do it myself.
 
- Keith

[Paolo Ambrosio]

All right, so this is different from global hooks. I guess you want
feature hooks.

> We
> need this because we have different setup for different types of tests, but
> still want to perform all tests in a single Cucumber run.

You could annotate features or single scenarios and write a tagged
before (scenario) hook that would run before every scenario within an
annotated feature and every annotated scenario. In my opinion,
features represent the point of view of the business, hooks are
technical details, related to step definitions.

There is a rejected pull request to Cucumber-JVM on adding feature
hooks: https://github.com/cucumber/cucumber-jvm/pull/295

> George, I disagree with your assertion that this is "a special case that is
> not often needed". There does seem to be significant interest in this
> feature, and you might want to count not only current Cucumber users, but
> others who may have decided not to use Cucumber, in part at least because
> this was a need they had. It may very well be that Cucumber is targeted at a
> specific group, and I acknowledge that it's not my place to have any
> expectations otherwise, but I just wanted to express my view so that it is
> available for consideration.
>
> * * *
>
> Regarding implementing before_all functionality myself, I now have a working
> solution in my Ruby/Cucumber testing. I use a class method (so that it is
> globally available, but does not pollute the global namespace). This method
> manages a set of initialization group keys internally (currently, I use
> __FILE__ as the key, but anticipate needing to be more precise later). These
> keys are needed because the setup and teardown may need to be done multiple
> times in a given Cucumber run (e.g. once for each of 2 feature files). So I
> can call MyInitializationClass.init_once(__FILE__), and the initialization
> will only be performed if it has not yet been sent that key. I think this is
> the best I'm going to get for my needs, but would appreciate any suggested
> improvements.

Can't understand why the initialization should depend on features, and
it's difficult to suggest an alternative without any context. Step
definitions depending on features is usually a test smell, as
technical details should be separated from the logical grouping of
scenarios in features.

> My problem with this and any other solution I can think of is that, in my
> opinion, all of them are less intention revealing, more complex, more
> verbose, and/or more error prone than a built-in Cucumber before_all would
> be.
>
> * * *
>
> Paolo, regarding your statement that if I want it, I should write it myself,
> you're right, of course. However, this unfortunately is not likely to
> happen, since I suspect the time I would need to invest just to understand
> the source base well enough to present an intelligent solution would
> probably be prohibitive for me. In contrast, someone already familiar with
> the code base would probably know exactly where this modification should go,
> what gotchas to look out for, etc. In addition, even if I were to provide a
> patch, it might still not be accepted, either because it was not a good
> technical fit, or there may be a philosophical opposition to it. I do
> realize that this is open source software and I am not a paying customer, so
> can't complain if I don't do it myself.

It's always good to check in the mailing list before working on a new
feature. No one was able to provide an example of feature hooks that
cannot be already handled with tags and hooks, or other techniques.


Paolo

[Keith Bennett]

Paolo -

Thanks for your thoughtful and thorough response.  More below...

Yes, precisely.  Sorry I wasn't clear about that.

> We
> need this because we have different setup for different types of tests, but
> still want to perform all tests in a single Cucumber run.

You could annotate features or single scenarios and write a tagged
before (scenario) hook that would run before every scenario within an
annotated feature and every annotated scenario. In my opinion,
features represent the point of view of the business, hooks are
technical details, related to step definitions.

There is a rejected pull request to Cucumber-JVM on adding feature
hooks: https://github.com/cucumber/cucumber-jvm/pull/295

Yes, I took a good look at it, and the gist. The approach to teardown is to do it on entry to a scenario not having the initialization tag.  This may *usually* work, but what about when the initialization is both a cleanup and a setup, and must be redone on feature file boundaries?  Put differently, it doesn't really let you control when the cleanup code runs, which also determines whether or not setup code will run.

Also, even when it works, it's a bit circuitous in that it's doing teardown/after processing in a setup/before hook.  In that sense, it's not as clear to the reader. Compared with the 100% intention revealing 'before feature' and 'after feature', it's a kludge, clever as it may be.

You said in that thread "It uploads the files before it finds a scenario that is tagged with @mysetup (the first in the feature if you tag the whole feature) and it removes them when it finds a scenario not tagged with @mysetup (in your case before it executes the next feature file or before it shuts down if it's the last feature file to be executed)."

Regarding the last part of that statement, "...or before it shuts down if it's the last feature file to be executed", I don't think that's true.  I just did a test in Ruby Cucumber with an After('~@foo') expression, where all my scenarios were tagged with @foo, and the call was never called.  You would probably need to add a shutdown hook, wouldn't you? Or am I doing something wrong?

If I'm correct, then teardown of a feature is accomplished via 1) setup of a subsequent feature, and 2) a shutdown hook.  Isn't that a cognitive burden of unnecessary complexity for Cucumber test authors, compared with an after_feature hook?
 

> George, I disagree with your assertion that this is "a special case that is
> not often needed".  There does seem to be significant interest in this
> feature, and you might want to count not only current Cucumber users, but
> others who may have decided not to use Cucumber, in part at least because
> this was a need they had. It may very well be that Cucumber is targeted at a
> specific group, and I acknowledge that it's not my place to have any
> expectations otherwise, but I just wanted to express my view so that it is
> available for consideration.
>
> * * *
>
> Regarding implementing before_all functionality myself, I now have a working
> solution in my Ruby/Cucumber testing.  I use a class method (so that it is
> globally available, but does not pollute the global namespace).  This method
> manages a set of initialization group keys internally (currently, I use
> __FILE__ as the key, but anticipate needing to be more precise later). These
> keys are needed because the setup and teardown may need to be done multiple
> times in a given Cucumber run (e.g. once for each of 2 feature files).  So I
> can call MyInitializationClass.init_once(__FILE__), and the initialization
> will only be performed if it has not yet been sent that key. I think this is
> the best I'm going to get for my needs, but would appreciate any suggested
> improvements.

Can't understand why the initialization should depend on features, and
it's difficult to suggest an alternative without any context.

I understand what you're saying; the feature boundary is somewhat artificial; I just think it's the least bad of all the alternatives.  And since a feature file is intended to be a set of logically related tests, it's a likely candidate for locating that boundary.

Parameterized tags could help; so could the ability to programmatically examine all the tags used in the current suite.  Then the code could look to see if this test were the last scenario to run with a given key; but I don't know Cucumber, and don't know if this level of introspection is possible or feasible.

As for context, I'm testing network servers in an environment whose setup may take minutes.  It's absolutely essential for us to minimize the number of times this setup is done. There's a good chance that many of these setup and teardown tasks will line up on feature boundaries. Does that help with the context?
 

Step
definitions depending on features is usually a test smell, as
technical details should be separated from the logical grouping of
scenarios in features.

If that were true, wouldn't it be true for any use of feature level tags by the underlying code?
 

> My problem with this and any other solution I can think of is that, in my
> opinion, all of them are less intention revealing, more complex, more
> verbose, and/or more error prone than a built-in Cucumber before_all would
> be.
>
> * * *
>
> Paolo, regarding your statement that if I want it, I should write it myself,
> you're right, of course.  However, this unfortunately is not likely to
> happen, since I suspect the time I would need to invest just to understand
> the source base well enough to present an intelligent solution would
> probably be prohibitive for me.  In contrast, someone already familiar with
> the code base would probably know exactly where this modification should go,
> what gotchas to look out for, etc.  In addition, even if I were to provide a
> patch, it might still not be accepted, either because it was not a good
> technical fit, or there may be a philosophical opposition to it. I do
> realize that this is open source software and I am not a paying customer, so
> can't complain if I don't do it myself.

It's always good to check in the mailing list before working on a new
feature. No one was able to provide an example of feature hooks that
cannot be already handled with tags and hooks, or other techniques.

Of course it can be implemented in other ways...to take it to an absurd but illustrative extreme, it could be implemented in C or assembler, but that doesn't mean it's a good idea.  I continue to believe that the current mechanisms are verbose and complex workarounds compared with a simple before_feature or after_feature hook.

- Keith

 

[Keith Bennett]

A clarification and a couple of comments below...

Another challenge of this approach is that if an error occurs during this setup/teardown, then it is not reported in the scenario that really "owns" it.

 

 

> George, I disagree with your assertion that this is "a special case that is
> not often needed".  There does seem to be significant interest in this
> feature, and you might want to count not only current Cucumber users, but
> others who may have decided not to use Cucumber, in part at least because
> this was a need they had. It may very well be that Cucumber is targeted at a
> specific group, and I acknowledge that it's not my place to have any
> expectations otherwise, but I just wanted to express my view so that it is
> available for consideration.
>
> * * *
>
> Regarding implementing before_all functionality myself, I now have a working
> solution in my Ruby/Cucumber testing.  I use a class method (so that it is
> globally available, but does not pollute the global namespace).  This method
> manages a set of initialization group keys internally (currently, I use
> __FILE__ as the key, but anticipate needing to be more precise later). These
> keys are needed because the setup and teardown may need to be done multiple
> times in a given Cucumber run (e.g. once for each of 2 feature files).  So I
> can call MyInitializationClass.init_once(__FILE__), and the initialization
> will only be performed if it has not yet been sent that key. I think this is
> the best I'm going to get for my needs, but would appreciate any suggested
> improvements.

Can't understand why the initialization should depend on features, and
it's difficult to suggest an alternative without any context.

I understand what you're saying; the feature boundary is somewhat artificial; I just think it's the least bad of all the alternatives.  And since a feature file is intended to be a set of logically related tests, it's a likely candidate for locating that boundary.

Parameterized tags could help; so could the ability to programmatically examine all the tags used in the current suite.  Then the code could look to see if this test were the last scenario to run with a given key; but I don't know Cucumber, and don't know if this level of introspection is possible or feasible.

I meant to say "I don't know Cucumber's internal implementation".

Cucumber's great strength is its communicative clarity for test writers and readers.  I feel that the before/after feature suggestion is consistent with that value, for developers who implement the tests. We would still have the same tag based options, but would have a much cleaner option for feature-specific code.

 

- Keith

[Paolo Ambrosio]

I wish someone else could join this thread, as it seems we have
reached a point where Keith and I agree to disagree. It would be good
to have others' point of view, especially from members of the Cucumber
team and people that have been using it for years.

What I can't understand is why you say it should be done on feature
file boundaries. All you should know is that a scenario is executed
after the setup is completed and cleaned up at the end or before a
different setup is needed by another scenario.

> Also, even when it works, it's a bit circuitous in that it's doing
> teardown/after processing in a setup/before hook. In that sense, it's not
> as clear to the reader. Compared with the 100% intention revealing 'before
> feature' and 'after feature', it's a kludge, clever as it may be.

From what you wrote, I think hooks might not be the correct approach.
If it is of importance to the reader, it should not be hidden in a
hook. Cleanup instead should never be explicit, as it is a technical
detail.

In my opinion, using the context you provided, describing the network
server setup in a Feature Background would make it clearer to the
reader what is the expected environment. Something like this:

Feature: ...

Background:
Given I have the following services:
| DNS |
| SMTP |
| ... |

Scenario: ...
...

...or anything that makes sense.

> You said in that thread "It uploads the files before it finds a scenario
> that is tagged with @mysetup (the first in the feature if you tag the whole
> feature) and it removes them when it finds a scenario not tagged with
> @mysetup (in your case before it executes the next feature file or before it
> shuts down if it's the last feature file to be executed)."
>
> Regarding the last part of that statement, "...or before it shuts down if
> it's the last feature file to be executed", I don't think that's true. I
> just did a test in Ruby Cucumber with an After('~@foo') expression, where
> all my scenarios were tagged with @foo, and the call was never called. You
> would probably need to add a shutdown hook, wouldn't you? Or am I doing
> something wrong?

Yes, a shutdown hook should be registered the first time the hook is
processed, and it should clean up if needed.

If the setup needs to be different for each feature, a Feature
Background might be a solution.

> Parameterized tags could help; so could the ability to programmatically
> examine all the tags used in the current suite. Then the code could look to
> see if this test were the last scenario to run with a given key; but I don't
> know Cucumber, and don't know if this level of introspection is possible or
> feasible.
>
> As for context, I'm testing network servers in an environment whose setup
> may take minutes. It's absolutely essential for us to minimize the number
> of times this setup is done. There's a good chance that many of these setup
> and teardown tasks will line up on feature boundaries. Does that help with
> the context?
>
>> Step
>> definitions depending on features is usually a test smell, as
>> technical details should be separated from the logical grouping of
>> scenarios in features.
>>
>
> If that were true, wouldn't it be true for any use of feature level tags by
> the underlying code?

AFAIK feature-level tags in the current implementation can be
considered syntactic sugar to apply them to all scenarios in that
feature.

Very funny. Another extreme would be including a feature that is
useful to a bunch of people only. That would not be a good idea
either.

> I continue to believe that the current
> mechanisms are verbose and complex workarounds compared with a simple
> before_feature or after_feature hook.

I continue to believe that before_feature or after_feature hooks are a
bad idea. I might be wrong and everyone else needs feature hooks but I
can't see it. Again, this is just my opinion and hopefully someone
else will join this thread.


Paolo

[George Dinwiddie]

Paolo,

On 12/8/13 12:03 PM, Paolo Ambrosio wrote:
> I wish someone else could join this thread, as it seems we have
> reached a point where Keith and I agree to disagree. It would be good
> to have others' point of view, especially from members of the Cucumber
> team and people that have been using it for years.

I agree with your statement that feature files should organize according
to the needs of the business view of the functionality, not the
technical needs of how to test it. I have yet to see anything from Keith
that hints at the business needs. I suspect, from that, he may have the
wrong tool for what he wants to do. Or that he's forgotten the business
viewpoint, in which case I recommend more conversations with them.

- George

[Keith Bennett]

George, Paolo -

On Dec 8, 2013, at 12:58 PM, George Dinwiddie <li...@iDIAcomputing.com> wrote:

> Paolo,
>
> On 12/8/13 12:03 PM, Paolo Ambrosio wrote:
>> I wish someone else could join this thread, as it seems we have
>> reached a point where Keith and I agree to disagree. It would be good
>> to have others' point of view, especially from members of the Cucumber
>> team and people that have been using it for years.
>
> I agree with your statement that feature files should organize according to the needs of the business view of the functionality, not the technical needs of how to test it. I have yet to see anything from Keith that hints at the business needs.

I totally get what you’re saying about feature files. Yes, they are intended to be units of organization of the high level, business logic of the tests. Implementation tasks will not necessarily align on feature boundaries, and organizing scenarios in feature files based on implementation details is a bad idea, and violates the separation of high level business logic and low level test implementation. I also understand that if before/after feature were added, it might encourage people to organize scenarios based on implementation, and that would be unfortunate, and violate the Cucumber philosophy of valuing high level test readability as the highest value.

My opinion, however, was that the cost of *not* doing so is greater; at least in my case, the workarounds are much more complex. I didn’t claim the strategy was a good thing, just that it was the least bad alternative.

> I suspect, from that, he may have the wrong tool for what he wants to do.

I wonder that as well. I’m using Ruby, and if a test were implemented in plain Ruby (not Cucumber), then the implementation would be a lot simpler; for one thing, setup and teardown could be handled automatically in a block, as in:

with_my_test_environment do
test_1
…
test_n
end

…or, if using RSpec, then having the option of using before(:all) or after(:all).

There would be no need to use instance variables to communicate state across steps, and the test implementation would be easier to write and read, since the author would have total freedom to organize code in whatever way would be most logical and clear, possibly even a single 4 or 5 line method.

However, then I’d obviously lose the communication value of the feature files, which is Cucumber’s strength, and extremely important to us.

So it’s a cost/benefit analysis. As the implementer of the tests, I’m most sensitive to the implementation side of the wall. I realize that, and that any solution I choose needs to safeguard the readability of the tests. I don’t believe that a before/after feature capability would compromise the readability, though, I think it would improve it.

That is exactly how I am doing it now, by using a background step. My issue is with the *implementation* of that background step.

That’s kind of how I do it; I have the same step definition but use a variable to set the initialization mode. Again, my problems are not on the feature file side, but on the step definition side.

Fair enough. However, I wonder if there are many who have felt the same pain points as I, but are not as outspoken, and have perhaps left the Cucumber community. I recognize that the decision is not mine to make, and I’m grateful for Cucumber either way.

- Keith

[Andrew Premdas]

On 8 December 2013 17:03, Paolo Ambrosio <pa...@ambrosio.name> wrote:

I wish someone else could join this thread, as it seems we have
reached a point where Keith and I agree to disagree. It would be good
to have others' point of view, especially from members of the Cucumber
team and people that have been using it for years.

Ok, I'll have a go. I'm someoene who has been using Cucumber pretty much every day since the days of RSpec plain text stories, and have read/scanned pretty much every post that has ever been written on this group, so I've seen this issue come up time and again in various different formats. 

What Keith is asking for is for Cucumber to implement one of the truly fundamental testing anti-patterns. Ever since Kent Beck brought test automation into the mainstream with JUnit, its been established that the independence of each individual test is primary. Those of us who have experience of breaking this pattern realise just how serious the consequences are. So not only would we never do this work, we would do our best to ensure that this work never became a part of Cucumber (i.e. something that is part of the language of features - Gherkin).

So what are these consequences:

Introducing any kind of global setup (i.e. setup that is not reset between tests) means that:

- Every test that is run must guarantee not to change any aspect of the global setup in any way.

- For cucumber this means that every step definition must guarantee not to change the global setup in any way.

- This means we can't write steps like, "When I login" if this changes the global setup, which it does, because now we are logged in. Of course we can fix this by ensuring we logout. But that now means that every step definition that changes the global setup must now have a counter step to reverse the change e.g. "When I logout"

- Now we have a situation where every step must be accompanied by a counter step. So every feature now has to be something like

Given ...

When ...

Then ...

UnGiven

which of course makes every feature completely unreadable,

These problems already arise in many of the scenarios that less experienced Cukers write, where you  have

Given ....

# another twenty lines of feature text trying to setup some kind of state

When 

# another five or ten lines of stuff doing something

Then

# another long bunch of checks

When

# Oh yeh now we can do some more stuff

... etc

Long features suck because every step definition that is run can change the state of the application, and any one of these changes can end breaking the feature in some way that is difficult (expensive) to pinpoint and understand. They suck in so many other ways as well.

Keith's proposal makes every feature file a feature, it increases the length of each feature.

I could (and have in the past) go on about this for several more pages, but generally I've found this is fruitless. There is a consistent pattern of people like Keith being quite stubborn that their particular case needs xxx, and getting involved in the minutiae does nothing to resolve this.

So perhaps we need to talk about this in a different way and change our perpective. Perhaps the history of cucumber can tell us something about why these issues keep on arising.

Over Cucumbers history two radically different groups have emerged. I'm going to give them silly names: the featurists and the abstractors.

Featurists love features. They want to add to the expressiveness of Gherkin. They use tables, example groups tags and every bit of Cucumbers functionality, and they often think of all sorts of ways that features could be improved and extended.

Abstractors love features but in addition they love the power of abstraction. They write simple features that rarely use tables, examples groups, even parameters.

Personally I was a fanatical featurist in the early days of Cucumber and now of course I'm an arch abstractor!

Featurism shaped the early days of Cucumber. RSpec plain text stories were incredibly difficult to use in comparison to modern day Cukes. Featurism gave us tables, example groups, 'but', tags, `before`, transforms. It also gave us nested steps, web steps, pickle, factories. Without featurism there would be no Cucumber and no Abstractors.

However many fanatical featurists soon discoverd that all was not good with featurism. This was very disturbing, it seemed that Cucumber was fundamentally flawed. As our features grew in number and became more complex our test suite became much harder to work and very unreliable. Very often it would take 10 times longer to write a scenario than implement the change, and when things went wrong this could increase by another one or two orders of magnitude. As featurists we initially looked at how we could improve our tool to deal with this. 

Generally the featurist response was to put more into features, however a pattern quickly emerged which is basically this:

      -- the more you do in features, the worse things get --

This was very disturbing. To cut a very long story short what we discovered is that features are for the concise natural language expression of `what` you are doing, with a sprinkling of `why` you are doing it. Step definitions are the place to translate features into method calls. The methods called make up the API of our Cucumber World. Finally the implementation of this API is where we make the features work. This is where the `how` is done. Doing the how is programming.

The art of writing cucumber features is to use natural language skills like abstraction and naming to express concisely, clearly and above all simply any problem. If we have not expressed our problem in simple clear concise language, then we haven't done enough writing. 

The causal force behind wanting to have a global before, or tables within tables, or importation of excel spreadsheets or huge example groups or nested features etc., is fundamentally a lack of writing.

Cucumber is very good at is telling you when you haven't done enough writing. But you have to be willing to listen.

All best

Andrew

--
-- Rules --

1) Please prefix the subject with [Ruby], [JVM] or [JS].

2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style

3) If you have a question, don't reply to an existing message. Start a new topic instead.

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
---

You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Roberto Lo Giacco]

l giorno mercoledì 29 febbraio 2012 19:00:12 UTC+1, Matthew ha scritto:

In the java world you could try DBUnit - it hasn't had much done to it
recently, but seems to work.

It DOES work, I'm one of the developers :-D