Cucumber Configuration

[aslak hellesoy]

Hi all,

I'm at CukeUp! (duh, obviously) and getting lots of great ideas talking to people.

Kirill from JetBrains told me they would like to run each Scenario with different configurations.

JetBrains uses Cucumber-JVM to test RubyMine - their excellent Ruby IDE. RubyMine needs to work with various versions of Ruby, and they would like to run each scenario once for each Ruby version they want to support.

This is similar to the situation where a team developing a web app uses a browser testing library and wants to run each Scenario once for each browser, say Firefox, Chrome and IE.

Currently, the only way to do this is to launch Cucumber once for each environment, setting the environment settings in environment variables or system properties.

This is fine for a CI environment, but for a developer it becomes rather tedious to launch Cucumber N times.

I'm therefore considering adding a new feature to Cucumber that would make it able to run each Scenario N times within the same Cucumber execution. All the results would end up in the same report.

What I'm thinking is that the configuration of the various environments would be done in a configuration file. Configuration would be written in JSON or TOML [1]. See the example [2].

If we're introducing a configuration file, we might as well take the opportunity to add some more to it, like command line options.

You'd specify it with --config path/to/config.{json|toml}. For Cucumber-Ruby this would replace cucumber.yml. We also have a related ticket for this [3].

WDYT?

[1] https://github.com/mojombo/toml

[2] https://gist.github.com/aslakhellesoy/5310313

[3] https://github.com/cucumber/cucumber-jvm/issues/330

[Tim Mertens]

Aslak,

A config file could certainly be useful, but specifying multiple environment configurations seems like a potential black hole as there are many different possible configurations a developer/tester might want to use when running tests.  It would need to be totally configuration agnostic and the heavy lifting of using the environment configuration settings at run-time should be done in the step definitions (@Before, @After, or in the stepdefs themselves) and implemented by the developer writing the test code.

Regarding the configuration file format, have you considered YAML? I admittedly only glanced at TOML from the link in your post but it looks like it is fairly young and not nearly as mature as the YAML or JSON formats.  I would be concerned with the long term stability and support of the TOML API.  YAML has more flexibility and extensibility than JSON in my experience.  I used JSON initially for configuration files for a project I am currently working on but ultimately moved to YAML for its flexibility and readability, especially in Java using snakeyaml.  I have only used it in Java, however, so I'm not sure how capable it is in other languages that Cucumber supports.

Cheers,
Tim

[Matt Wynne]

Fixing Rule (2)

On 4 Apr 2013, at 15:46, Tim Mertens <t...@filmchicago.org> wrote:

On Thursday, April 4, 2013 8:53:06 AM UTC-5, Aslak Hellesøy wrote:

Hi all,

I'm at CukeUp! (duh, obviously) and getting lots of great ideas talking to people.

Kirill from JetBrains told me they would like to run each Scenario with different configurations.

JetBrains uses Cucumber-JVM to test RubyMine - their excellent Ruby IDE. RubyMine needs to work with various versions of Ruby, and they would like to run each scenario once for each Ruby version they want to support.

This is similar to the situation where a team developing a web app uses a browser testing library and wants to run each Scenario once for each browser, say Firefox, Chrome and IE.

Currently, the only way to do this is to launch Cucumber once for each environment, setting the environment settings in environment variables or system properties.

This is fine for a CI environment, but for a developer it becomes rather tedious to launch Cucumber N times.

I'm therefore considering adding a new feature to Cucumber that would make it able to run each Scenario N times within the same Cucumber execution. All the results would end up in the same report.

What I'm thinking is that the configuration of the various environments would be done in a configuration file. Configuration would be written in JSON or TOML [1]. See the example [2].

If we're introducing a configuration file, we might as well take the opportunity to add some more to it, like command line options.

You'd specify it with --config path/to/config.{json|toml}. For Cucumber-Ruby this would replace cucumber.yml. We also have a related ticket for this [3].

WDYT?

Aslak,

A config file could certainly be useful, but specifying multiple environment configurations seems like a potential black hole as there are many different possible configurations a developer/tester might want to use when running tests.  It would need to be totally configuration agnostic and the heavy lifting of using the environment configuration settings at run-time should be done in the step definitions (@Before, @After, or in the stepdefs themselves) and implemented by the developer writing the test code.

Regarding the configuration file format, have you considered YAML? I admittedly only glanced at TOML from the link in your post but it looks like it is fairly young and not nearly as mature as the YAML or JSON formats.  I would be concerned with the long term stability and support of the TOML API.  YAML has more flexibility and extensibility than JSON in my experience.  I used JSON initially for configuration files for a project I am currently working on but ultimately moved to YAML for its flexibility and readability, especially in Java using snakeyaml.  I have only used it in Java, however, so I'm not sure how capable it is in other languages that Cucumber supports.

Cheers,
Tim

[1] https://github.com/mojombo/toml

[2] https://gist.github.com/aslakhellesoy/5310313

[3] https://github.com/cucumber/cucumber-jvm/issues/330

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 

cheers,

Matt

--
http://mattwynne.net || https://twitter.com/mattwynne || http://the-cucumber-book.com || http://bddkickstart.com || http://www.relishapp.com

[Tim M]

On Thu, Apr 4, 2013 at 9:48 AM, Matt Wynne <ma...@mattwynne.net> wrote:

Fixing Rule (2)

Whoops, thanks Matt :)

 

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/CdO7fG1R0KQ/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Richard Paul]

This sounds like it would make it simpler for developers to run tests cross browser for the particular feature they are working on, although a bash script could accomplish this fairly simply (without the combination of reports mind you).

When it comes to the CI environment I imagine you'd want to run each browser in parallel across the entire suite of scenarios (to reduce the feedback time). If Cucumber were to support multiple environments via config would you envisage Cucumber handling the parallelisation itself across the configurations? Or would you still use parallel jobs on your CI server?

[Andrew Premdas]

On 4 Apr 2013, at 14:53, aslak hellesoy <aslak.h...@gmail.com> wrote:

Hi all,

I'm at CukeUp! (duh, obviously) and getting lots of great ideas talking to people.

Kirill from JetBrains told me they would like to run each Scenario with different configurations.

JetBrains uses Cucumber-JVM to test RubyMine - their excellent Ruby IDE. RubyMine needs to work with various versions of Ruby, and they would like to run each scenario once for each Ruby version they want to support.

This is similar to the situation where a team developing a web app uses a browser testing library and wants to run each Scenario once for each browser, say Firefox, Chrome and IE.

Currently, the only way to do this is to launch Cucumber once for each environment, setting the environment settings in environment variables or system properties.

I'm not sure this is necessarily true. This is a bit speculative, but I think you could use a profile (in cucumber.yml) to include a hook (via -f) that you would apply to all features. There was a post a while back about using hook to run a scenario twice (non-us and js), and that worked fine. So maybe Cukes already has all it needs.

Andrew

This is fine for a CI environment, but for a developer it becomes rather tedious to launch Cucumber N times.

I'm therefore considering adding a new feature to Cucumber that would make it able to run each Scenario N times within the same Cucumber execution. All the results would end up in the same report.

What I'm thinking is that the configuration of the various environments would be done in a configuration file. Configuration would be written in JSON or TOML [1]. See the example [2].

If we're introducing a configuration file, we might as well take the opportunity to add some more to it, like command line options.

You'd specify it with --config path/to/config.{json|toml}. For Cucumber-Ruby this would replace cucumber.yml. We also have a related ticket for this [3].

WDYT?

[1] https://github.com/mojombo/toml

[2] https://gist.github.com/aslakhellesoy/5310313

[3] https://github.com/cucumber/cucumber-jvm/issues/330

--

[aslak hellesoy]

On Thu, Apr 4, 2013 at 3:48 PM, Matt Wynne <ma...@mattwynne.net> wrote:

Fixing Rule (2)

On 4 Apr 2013, at 15:46, Tim Mertens <t...@filmchicago.org> wrote:

On Thursday, April 4, 2013 8:53:06 AM UTC-5, Aslak Hellesøy wrote:

Hi all,

I'm at CukeUp! (duh, obviously) and getting lots of great ideas talking to people.

Kirill from JetBrains told me they would like to run each Scenario with different configurations.

JetBrains uses Cucumber-JVM to test RubyMine - their excellent Ruby IDE. RubyMine needs to work with various versions of Ruby, and they would like to run each scenario once for each Ruby version they want to support.

This is similar to the situation where a team developing a web app uses a browser testing library and wants to run each Scenario once for each browser, say Firefox, Chrome and IE.

Currently, the only way to do this is to launch Cucumber once for each environment, setting the environment settings in environment variables or system properties.

This is fine for a CI environment, but for a developer it becomes rather tedious to launch Cucumber N times.

I'm therefore considering adding a new feature to Cucumber that would make it able to run each Scenario N times within the same Cucumber execution. All the results would end up in the same report.

What I'm thinking is that the configuration of the various environments would be done in a configuration file. Configuration would be written in JSON or TOML [1]. See the example [2].

If we're introducing a configuration file, we might as well take the opportunity to add some more to it, like command line options.

You'd specify it with --config path/to/config.{json|toml}. For Cucumber-Ruby this would replace cucumber.yml. We also have a related ticket for this [3].

WDYT?

Aslak,

A config file could certainly be useful, but specifying multiple environment configurations seems like a potential black hole as there are many different possible configurations a developer/tester might want to use when running tests.  It would need to be totally configuration agnostic and the heavy lifting of using the environment configuration settings at run-time should be done in the step definitions (@Before, @After, or in the stepdefs themselves) and implemented by the developer writing the test code.

Indeed. They way I imagine this would be used would be very similar to how we recommend using environment variables:

https://github.com/cucumber/cucumber/wiki/Environment-Variables

// java

Map<String,String> environment = Cucumber.env();

// ruby

environment = Cucumber.env

Regarding the configuration file format, have you considered YAML? I admittedly only glanced at TOML from the link in your post but it looks like it is fairly young and not nearly as mature as the YAML or JSON formats.  I would be concerned with the long term stability and support of the TOML API.  YAML has more flexibility and extensibility than JSON in my experience.  I used JSON initially for configuration files for a project I am currently working on but ultimately moved to YAML for its flexibility and readability, especially in Java using snakeyaml.  I have only used it in Java, however, so I'm not sure how capable it is in other languages that Cucumber supports.

Sure, we could easily support YAML as well. The markup is the least important part of this discussion I think.

Aslak

Cheers,
Tim

[1] https://github.com/mojombo/toml

[2] https://gist.github.com/aslakhellesoy/5310313

[3] https://github.com/cucumber/cucumber-jvm/issues/330

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 

cheers,

Matt

--
http://mattwynne.net || https://twitter.com/mattwynne || http://the-cucumber-book.com || http://bddkickstart.com || http://www.relishapp.com

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
---

You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

[Oleg Sukhodolsky]

am I right that you suggest to enhance our profiles configurations to allow user specify environment variables (settings) and provide a way to run features with different profiles by one command?

Regards, Oleg. 

[aslak hellesoy]

I'm not suggesting a new way to specify Shell/OS environment variables.

I'm suggesting a way to run scenarios more than once, each time with a user-defined "environment".

This environment is just a map/hash table that can be accessed globally.

 

and provide a way to run features with different profiles by one command?

Yes, the idea is to be able to run scenarios more than once within a single cucumber execution.

Aslak

[Oleg Sukhodolsky]

I see.  How this will corelate to profiles we already have?

 

 

and provide a way to run features with different profiles by one command?

Yes, the idea is to be able to run scenarios more than once within a single cucumber execution.

I see.

Regards, Oleg. 

[aslak hellesoy]

The current cucumber.yml provides a way to set command line options.

What I'm proposing also allows that (see the example gist), as well as optionally defining multiple execution environment.

My proposal would replace cucumber.yml

Aslak

[Matt Wynne]

On 4 Apr 2013, at 14:53, aslak hellesoy <aslak.h...@gmail.com> wrote:

I think I'd like to see a concrete PR to get an idea of how this would work and whether there are any alternatives.

FWIW, we're planning to clean up the configuration in Cucumber Ruby for 2.0, and that might involve replacing or at least deprecating Cucumber.yml, to replace it with something that directly configures Cucumber rather than just configuring CLI argument strings. So that might help / be relevant.

[Aslak Hellesøy]

On Saturday, 6 April 2013 at 20:37, Matt Wynne wrote:

On 4 Apr 2013, at 14:53, aslak hellesoy <aslak.h...@gmail.com> wrote:

Hi all,

I'm at CukeUp! (duh, obviously) and getting lots of great ideas talking to people.

Kirill from JetBrains told me they would like to run each Scenario with different configurations.

JetBrains uses Cucumber-JVM to test RubyMine - their excellent Ruby IDE. RubyMine needs to work with various versions of Ruby, and they would like to run each scenario once for each Ruby version they want to support.

This is similar to the situation where a team developing a web app uses a browser testing library and wants to run each Scenario once for each browser, say Firefox, Chrome and IE.

Currently, the only way to do this is to launch Cucumber once for each environment, setting the environment settings in environment variables or system properties.

This is fine for a CI environment, but for a developer it becomes rather tedious to launch Cucumber N times.

I'm therefore considering adding a new feature to Cucumber that would make it able to run each Scenario N times within the same Cucumber execution. All the results would end up in the same report.

What I'm thinking is that the configuration of the various environments would be done in a configuration file. Configuration would be written in JSON or TOML [1]. See the example [2].

If we're introducing a configuration file, we might as well take the opportunity to add some more to it, like command line options.

You'd specify it with --config path/to/config.{json|toml}. For Cucumber-Ruby this would replace cucumber.yml. We also have a related ticket for this [3].

WDYT?

[1] https://github.com/mojombo/toml

[2] https://gist.github.com/aslakhellesoy/5310313

[3] https://github.com/cucumber/cucumber-jvm/issues/330

I think I'd like to see a concrete PR to get an idea of how this would work and whether there are any alternatives.

I think it should wait until Cucumber-Ruby 2.0. Since it would break the 1-1 relationship between gherkin source and formatter output I think it's better to wait.

Aslak

FWIW, we're planning to clean up the configuration in Cucumber Ruby for 2.0, and that might involve replacing or at least deprecating Cucumber.yml, to replace it with something that directly configures Cucumber rather than just configuring CLI argument strings. So that might help / be relevant.

cheers,

Matt

--
http://mattwynne.net || https://twitter.com/mattwynne || http://the-cucumber-book.com || http://bddkickstart.com || http://www.relishapp.com

[Andrew Premdas]

Aslak 

Could you provide a link to the example gist you mention above - I'm struggling to find it in this thread

Cheers

Andrew

[Matt Wynne]

Fixing Rule 2. See footer for details.

On 11 Apr 2013, at 17:28, Adrian Ramses <ram...@tacitknowledge.com> wrote:

From my experience this would be very helpful,

We have been dealing with this too, in our framework we had to create yml files per environment so we could change data used in assertions depending on the environment.


So it would be nice that this configuration file whether its json, toml or yml, could grow as desired so we could link data to those configurations. For example:
{
    "options": {
        "glue": "classpath:com/company/app/stepdefs",
        "format": [
            "pretty",
            "json:target/cucumber.json"
        ]
    },
    "environments": {
        "firefox": {
            "driver": "org.openqa.selenium.firefox.FirefoxDriver"
        },
        "chrome": {
            "driver": "org.openqa.selenium.chrome.ChromeDriver"
        }
    },
    "contexts": {
        "prod": {
            "hostname": "prod.site.com",
            "propertyX": "expected_value_Y"
        },
        "integration": {
            "hostname": "mock.site.com",
            "propertyX": "expected_value_Z"
        }
    }
}

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 

[Kevin]

Aslak, Is there a timeline for this being built and is there documentation to achieve the same result (multiple browsers) in the meantime?  Either with maven or something else?

[aslak hellesoy]

On Thu, May 2, 2013 at 9:37 AM, Kevin <king....@gmail.com> wrote:

Aslak, Is there a timeline for this being built

No

 

and is there documentation to achieve the same result (multiple browsers) in the meantime?  Either with maven or something else?

Not that I can think of

 

[Mike Howells]

We're using Groovy glue and running the same scenarios with different configurations by adding a ConfigSlurper to a context object created on the Before hook. We can pass in the environment to use for each run as a system property. It doesn't run the same scenarios N times in one run, but we can easily do multiple runs with different environments.

This is another situation in which it would be useful if Cucumber-JVM passed a context variable into every step definition, like Behave for Python does. This context provides somewhere for steps to hang various values that can be used by later steps, which must surely be a pretty common practice. If the framework created a fresh context for each scenario it would save having to do this ourselves with hooks. If the framework included a configuration object in the context, and we could pass in a list of environments to use, it would be straightforward for the framework to repeat a feature for each of the configuration names passed in.

[Mike Howells]

If the framework were to repeat a scenario until the After hook evaluated to false, we could change the environment each time until we've gone through them all.

[Sasha]

I kind-off lost from this big thread so what should be doing to run the same scenario multiple times with in one one Cucumber execution , any documenation or example would be greatly appreciated.

Thanks,
-Sasha

[aslak hellesoy]

On Tue, Jul 9, 2013 at 3:49 PM, Sasha <swapna....@gmail.com> wrote:

I kind-off lost from this big thread so what should be doing to run the same scenario multiple times with in one one Cucumber execution

You can't. This thread is a discussion about adding that feature, something that hasn't been done.

Thanks,
-Sasha

[Ragavan Ambighananthan]

watirgrid does the same (execute the same scenario against multiple browsers + does this in parallel)

But will be nice to have this as part of cucumber

Regards

Ragavan

[Deepak Chennoria]

Hi Aslak Hellesøy,

Its very very good thought. I working to set an Automation framework using Capybar+Cucumber and Selenium Web driver. I stuck in the way, as I am successfully able to set one environment from config.yml for execution of scenarios. I want to mekw it happen in myltiple browsers using multiple environment in parallel.

This is how my config.yml looks:

QAEnvironment:
  default_driver: selenium
  javascript_driver: selenium
  default_selectore: css
  default_wait_time: 5
  base_url: "http://anywebsite/login"
  ignore_hidden_elements: false
  browser_name: firefox
  screenshots: yes
 
DevEnvironment:
  default_driver: selenium
  javascript_driver: selenium
  default_selectore: css
  default_wait_time: 10
  base_url: "http://anywebsite/login"
  ignore_hidden_elements: false
  browser_name: chrome
  screenshots: no
 
What I used to do just set my environment veriable from CMD as "set test_env=QAEnvironment" and then "cucumber src/" is used to execute the scenarios in the environment.

Here is my env.rb file:
require 'capybara'
require 'capybara/cucumber'
require 'capybara-screenshot'
require 'yaml'

require File.dirname(__FILE__) + '/lib/configuration';


Capybara.configure do |set|

  set.default_driver = :"#{Configuration['default_driver']}"
  set.javascript_driver = :"#{Configuration['javascript_driver']}"
  set.run_server = false
  set.default_selector = :"#{Configuration['default_selectore']}"
  set.default_wait_time = Configuration["default_wait_time"]
  set.app_host = Configuration["base_url"]
  set.match = :prefer_exact
  set.ignore_hidden_elements = false
  Capybara.register_driver :selenium do |app|
    Capybara::Selenium::Driver.new(app, :browser => :"#{Configuration['browser_name']}")
  end
browser = Capybara.current_session.driver.browser
browser.manage.window.resize_to(1680, 1050)
browser.manage.delete_all_cookies
end
if Configuration["screenshots"]
  After do |scenario|
    if(scenario.failed?)
      page.driver.browser.save_screenshot("reports/screenshots/#{scenario.__id__}.png")
      embed("#{scenario.__id__}.png", "image/png", "SCREENSHOT")
    end
  end
end

My config.yml reader file:
require 'yaml'

class Configuration
  def self.[] key
    @@config[key]
  end

  def self.load name
    @@config = nil
    io = File.open( File.dirname(__FILE__) + "/../config.yml" )
    YAML::load_documents(io) { |doc| @@config = doc[name] }
    raise "Could not locate a configuration named \"#{name}\"" unless @@config
  end

  def self.[]= key, value
    @@config[key] = value
  end
end

raise "Please set the TEST_ENV environment variable" unless ENV['TEST_ENV']
#Configuration.getEnvList(ENV['TEST_ENV'])
Configuration.load(ENV['TEST_ENV'])

Please let me know or suggest me, how can I trigger test cases in multiple environment with this setup.

Other issues:
Screenshot link is apper in htmlo reports of cucumber but not in json reports..



Thanks & Regards,
Deepak
-----------------------------------------------------------------------------------------------------------------