Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.
I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.
Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
On Wed, Nov 19, 2014 at 11:50 PM, Peter Wilson <peter....@gmail.com> wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I would be really scared working on such a monolithic software that requires tens of thousands of functional test. Or have you got too many tests? Sometimes there is little value in having such a massive number of scenarios, and maintaining them is going to be hell.
I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.Definitely a lot of work! For which flavour of Cucumber (Ruby, JVM, ...) are you planning to develop it?Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.On this point I totally disagree. The "expensive tools" for performance testing should put the application under load, and functional tests running definitely do not. In my experience there are a few scenarios that would never be addressed by running functional tests, to name a few...
- baseline - with reference production-like load, e.g. to test that the new release is performing as the old one- soak - running at medium load for several hours on the same running instance, to detect problems like memory leaks- special scenarios - from production load on specific events (promotional campaigns, etc.), to see if the next time it will break predict the breaking point, find bottlenecks, ...
On Wed, Nov 19, 2014 at 11:50 PM, Peter Wilson <peter....@gmail.com> wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I would be really scared working on such a monolithic software that requires tens of thousands of functional test. Or have you got too many tests? Sometimes there is little value in having such a massive number of scenarios, and maintaining them is going to be hell.
I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.Definitely a lot of work! For which flavour of Cucumber (Ruby, JVM, ...) are you planning to develop it?
Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.On this point I totally disagree. The "expensive tools" for performance testing should put the application under load, and functional tests running definitely do not. In my experience there are a few scenarios that would never be addressed by running functional tests, to name a few...- baseline - with reference production-like load, e.g. to test that the new release is performing as the old one- soak - running at medium load for several hours on the same running instance, to detect problems like memory leaks- special scenarios - from production load on specific events (promotional campaigns, etc.), to see if the next time it will break predict the breaking point, find bottlenecks, ...
On Thu, Nov 20, 2014 at 2:42 AM, Paolo Ambrosio <pa...@ambrosio.name> wrote:On Wed, Nov 19, 2014 at 11:50 PM, Peter Wilson <peter....@gmail.com> wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I would be really scared working on such a monolithic software that requires tens of thousands of functional test. Or have you got too many tests? Sometimes there is little value in having such a massive number of scenarios, and maintaining them is going to be hell.+1I have seen this done to a limited extent multiple times and it is a very brittle path that unnecessarily extends feedback loops much longer than is worth while IMO. (i.e. low ROI)
I would recommend making your systems much more component oriented and focus all of your testing at the lower levels.
I'm assuming at this scale, you're referring to several teams. The teams should be independently able to execute all tests necessary to validate their components are functional. Don't try to fight Conway's law.
I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.Definitely a lot of work! For which flavour of Cucumber (Ruby, JVM, ...) are you planning to develop it?Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.On this point I totally disagree. The "expensive tools" for performance testing should put the application under load, and functional tests running definitely do not. In my experience there are a few scenarios that would never be addressed by running functional tests, to name a few...+1I'm not necessarily advocating "expensive tools", just that one set of tests is a problematic idea. You'll likely want to have a focus at the component level if you have any sort of SOA. Even then, to load the system, to generate 1000s of users, items, etc is something you don't want to be part of your functional tests. You also shouldn't be focusing performance testing on edge case paths (it's debatable that you should even have functional cucumber tests for edge cases). And functional tests shouldn't be about code coverage or executing particular code paths like performance tests often are looking to profile specific conditions. Functional tests are more about behaviors... and should really be written before the functionality. Performance metrics should be thought about / discussed up front, but they often are not as binary (pass/fail) as functional tests. In other words, they are different scenarios for solving different problems.
Hi Rob, thanks for the response. Comments in line.
On Thursday, November 20, 2014 11:16:21 PM UTC+11, Rob Park wrote:On Thu, Nov 20, 2014 at 2:42 AM, Paolo Ambrosio <pa...@ambrosio.name> wrote:On Wed, Nov 19, 2014 at 11:50 PM, Peter Wilson <peter....@gmail.com> wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I would be really scared working on such a monolithic software that requires tens of thousands of functional test. Or have you got too many tests? Sometimes there is little value in having such a massive number of scenarios, and maintaining them is going to be hell.+1I have seen this done to a limited extent multiple times and it is a very brittle path that unnecessarily extends feedback loops much longer than is worth while IMO. (i.e. low ROI)Yeah. This is something I don't have a good handle on I suppose. We're in essence specifying by example and then executing our documentation. Some areas require a number of detailed examples to provide enough information to describe how it works. It's possible we're not pushing enough down to the unit level and leaving too much at the BDD level.
Can you give me any examples of how you have seen this done and fail before? This is exactly the kind of feedback I was hoping for. Is it a bad idea or was it just done badly before? Was it brittle because the automation was poorly done, the environment was unstable, the tests were not atomic enough? We have painstakingly gone down the path of making our test small and focused even though has meant we have had to overcome a large test data issue, so would our tests still be brittle?
Each team that makes a change would still be responsible for running relevant tests and fixing any tests that break. I guess if we have a lot of duplication in the tests or badly modularised automation code this would be a nightmare.
I would recommend making your systems much more component oriented and focus all of your testing at the lower levels.Thanks. We are actually doing that and also have tests at the integrated module levels too.I'm assuming at this scale, you're referring to several teams. The teams should be independently able to execute all tests necessary to validate their components are functional. Don't try to fight Conway's law.Yes. We are using scaled agile and have 7 to 8 agile teams in our program but at the organisation level we probably have more like somewhere between 15 to 30 teams... kind of a guess.I'm putting together solutions for the program that can also be used by the organisation.
Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.On this point I totally disagree. The "expensive tools" for performance testing should put the application under load, and functional tests running definitely do not. In my experience there are a few scenarios that would never be addressed by running functional tests, to name a few...+1I'm not necessarily advocating "expensive tools", just that one set of tests is a problematic idea. You'll likely want to have a focus at the component level if you have any sort of SOA. Even then, to load the system, to generate 1000s of users, items, etc is something you don't want to be part of your functional tests. You also shouldn't be focusing performance testing on edge case paths (it's debatable that you should even have functional cucumber tests for edge cases). And functional tests shouldn't be about code coverage or executing particular code paths like performance tests often are looking to profile specific conditions. Functional tests are more about behaviors... and should really be written before the functionality. Performance metrics should be thought about / discussed up front, but they often are not as binary (pass/fail) as functional tests. In other words, they are different scenarios for solving different problems.Yep. This is similar to what I said to Paolo. Definitely not looking at running wierdo edge cases for performance testing. Just picking representative user scenarios and then running them at prod like volumes.
Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.On this point I totally disagree. The "expensive tools" for performance testing should put the application under load, and functional tests running definitely do not. In my experience there are a few scenarios that would never be addressed by running functional tests, to name a few...+1I'm not necessarily advocating "expensive tools", just that one set of tests is a problematic idea. You'll likely want to have a focus at the component level if you have any sort of SOA. Even then, to load the system, to generate 1000s of users, items, etc is something you don't want to be part of your functional tests. You also shouldn't be focusing performance testing on edge case paths (it's debatable that you should even have functional cucumber tests for edge cases). And functional tests shouldn't be about code coverage or executing particular code paths like performance tests often are looking to profile specific conditions. Functional tests are more about behaviors... and should really be written before the functionality. Performance metrics should be thought about / discussed up front, but they often are not as binary (pass/fail) as functional tests. In other words, they are different scenarios for solving different problems.Yep. This is similar to what I said to Paolo. Definitely not looking at running wierdo edge cases for performance testing. Just picking representative user scenarios and then running them at prod like volumes.
Hi everyone,
In my company we use cucumber for REST APIs and this particular topic is very interesting for that. In our continuous delivery pipeline we would like to put each component under stress in isolation, just to make sure it still performs properly and the VM configuration is still up for the job. This is a step that happens separate from functional testing and separate from other performance testing (failover, chain, endurance, etc) so I think I have similar ideas as Peter. It's just meant as a fully automated regression test on performance, before the component goes to the next step in the pipeline. Developers/testers already spend time and effort in writing scenario's for functional testing and with most performance test tools, they would have to write very similar logic to do performance testing, meaning they'd be duplicating their work.
This is quite inefficient and if that logic is already specified somewhere, it would be nice to just reuse it.
It would be a great time-saver if we could for instance just add an annotation to specific scenarios to specify the amount of load it should be able to handle. I'm thinking about adding something like @performance(tps=10, p99=500ms) for a scenario that is called about 10 times each second in production and should respond in 500ms in 99% of the cases. In our 'functional testing environment' we would just ignore this annotation, but in the 'performance testing environment' we could use a different way of running the cucumber tests, selecting only those tests that have the @performance annotation and settings the load and verifying the performance as configured and the the API responses as specified in the feature files.
I am interested in hearing why this would or would not be a good idea. Also, if this can be built as a separate plug-in or something like that I would be very interested in a few pointers on how to do that. I am quite clueless on how to get hold of features and their annotations (including any information inside these annotations) in a location from which I can then invoke a stress test tool, so any help on that is appreciated.
Cheers,
Jordi
Hi Peter,I worked in an organisation that had 30000 scenarios last time I checked in so will be more than that today. At the time of there being 30000 scenarios they could all be executed in parallel around 15 at a time so that the total duration was about 1.5hrs. They were ALL being run after EVERY code check-in but they were starting to think about only running a sub-set after each code checkin as 1.5hrs was getting too long. Obviously could add more servers so they could run more than 15 in parallel or get faster servers, etc.The approach that was taken wasn't to parallelise by scenario but rather parallelise by feature file. They had an inhouse tool which would split up the around 10000 feature files and allocate 1/15th of each to a separate job, I cannot remember but I guess a separate bamboo job. I think each job was allocated a separate target environment to run the tests against. I don't recall we solved the problem of producing a single report.I think Scaling BDD is an important topic. Are you interested in pursuing this topic further privately?PS: Anyone who says to you that more than 10000 scenarios is wrong...well that person is wrong. ;-)
Regards,Stephen Kurlow
On Thursday, 20 November 2014 10:50:37 UTC+11, Peter Wilson wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.That's it.CheersPeter
Hi Peter,I worked in an organisation that had 30000 scenarios last time I checked in so will be more than that today.
At the time of there being 30000 scenarios they could all be executed in parallel around 15 at a time so that the total duration was about 1.5hrs. They were ALL being run after EVERY code check-in but they were starting to think about only running a sub-set after each code checkin as 1.5hrs was getting too long. Obviously could add more servers so they could run more than 15 in parallel or get faster servers, etc.The approach that was taken wasn't to parallelise by scenario but rather parallelise by feature file. They had an inhouse tool which would split up the around 10000 feature files and allocate 1/15th of each to a separate job, I cannot remember but I guess a separate bamboo job. I think each job was allocated a separate target environment to run the tests against. I don't recall we solved the problem of producing a single report.I think Scaling BDD is an important topic. Are you interested in pursuing this topic further privately?PS: Anyone who says to you that more than 10000 scenarios is wrong...well that person is wrong. ;-)
Regards,Stephen Kurlow
On Thursday, 20 November 2014 10:50:37 UTC+11, Peter Wilson wrote:Hi,We've been using cucumber for about 6 months now and have found it very beneficial.We're now looking to take it to a larger scale, with tens of thousands of tests and need a better test, infrastructure and data management system.I've put together a high level mud map of what I am thinking and would really value feedback from the group. For example, has this already been done, or thought about and discarded for some reason.To summarize:1. Feature file goes into the test manager (a server of some sort) which chops up the scenarios (being aware of the background) and farms it out to different cucumber nodes. It stores results of each execution of each step in the test management DB, along with how long each step takes and what test data it required2. The Cucumber Hub is like the Selenium Hub. Cucumber Nodes register with it and advertise test functions that it supports3. The Cucumber Node is a cucumber file (step definition and on, not the Gherkin) turned into a service. A scenarios is passed to the node, the node executes and returns the result4. The Data Manager looks after test data for different environments. It tracks daily usage and individual test usage and tries to create the necessary test data predictively ahead of time. The cucumber node will call the data manager as part of any data "given"5. The Selenium hub we already know about6. The Node Manager monitors the hubs looking at utilization levels and starts new cucumber and selenium nodes in a cloud service (such as Amazon) as required.Another important piece that I am heading towards here is using this model for performance testing. This would reduce the cost of performance testing because no expensive tools or specialized scripts are required. It would also mean performance testing can be done as soon as the functional scripts are complete, which means it can be done earlier. One set of scripts reduces maintenance costs as well.Having detailed information about scenario and step run times and data requirements would make performance testing pretty easy to setup too.That's it.CheersPeter
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/bh0EUhKHqCI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
On Tue, Mar 17, 2015 at 4:15 AM, Stephen Kurlow <sku...@gmail.com> wrote:Hi Peter,I worked in an organisation that had 30000 scenarios last time I checked in so will be more than that today.
Out of curiosity, roughly how many unit tests did this system have?
On Tue, Mar 17, 2015 at 12:15 AM, Stephen Kurlow <sku...@gmail.com> wrote:Hi Peter,I worked in an organisation that had 30000 scenarios last time I checked in so will be more than that today. At the time of there being 30000 scenarios they could all be executed in parallel around 15 at a time so that the total duration was about 1.5hrs. They were ALL being run after EVERY code check-in but they were starting to think about only running a sub-set after each code checkin as 1.5hrs was getting too long. Obviously could add more servers so they could run more than 15 in parallel or get faster servers, etc.The approach that was taken wasn't to parallelise by scenario but rather parallelise by feature file. They had an inhouse tool which would split up the around 10000 feature files and allocate 1/15th of each to a separate job, I cannot remember but I guess a separate bamboo job. I think each job was allocated a separate target environment to run the tests against. I don't recall we solved the problem of producing a single report.I think Scaling BDD is an important topic. Are you interested in pursuing this topic further privately?PS: Anyone who says to you that more than 10000 scenarios is wrong...well that person is wrong. ;-)Personally I think its up to the team, and product owner to decide on what scenarios should stay and which should go. The last place i was at the average number of scenarios per team was around 2500, and we managed to get the running at about 12 min on average.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/bh0EUhKHqCI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.