How to execute only few random Examples for Scenario Outline?

[Konstantin Trunin]

Hi. 

We have too big data set (examples section) in some test cases and we want to reduce execution time by checking only few random items for every test run.

Ideally it would be something like parametrized tag @Random(n) where is is % of examples that should be taken and executed.

E.g. if we have 100 examples and I set @Random(15) then 15 random examples should be taken and executed.

Could you please help me and clarify how to implement it?

I think I should create Hook, read tags.

If it is a first time we process Scenario then read examples and shuffle them, save in some global storage together with counter. 

If it is not a first time then check that counter is < required number - then find random item from global storage and remove it from there (to not repeat examples) and process execution with this example.

If we exceeded required number of executions then interrupt this Scenario Outline execution - and do it some how w/o error in report.

[Paolo Ambrosio]

On Mon, Feb 2, 2015 at 11:06 AM, Konstantin Trunin
<konst...@trunin.com> wrote:

> Hi.
> We have too big data set (examples section) in some test cases and we want
> to reduce execution time by checking only few random items for every test
> run.

The best way to do it is not to do it ;-)

If you are fine executing random examples, it means that the same
functionality is covered by more than one example. It is a test smell
and you should be thinking how you can split and group them. Examples
are fine to define rules and behaviour, but IMO examples should be
kept to a minimum in scenario outlines.

> Ideally it would be something like parametrized tag @Random(n) where is is %
> of examples that should be taken and executed.
> E.g. if we have 100 examples and I set @Random(15) then 15 random examples
> should be taken and executed.
> Could you please help me and clarify how to implement it?

If you want to execute part of the examples in a test suite that runs
on every checkin and a full suite overnight, you can have more than
one examples section annotated with different tags.

> I think I should create Hook, read tags.
> If it is a first time we process Scenario then read examples and shuffle
> them, save in some global storage together with counter.
> If it is not a first time then check that counter is < required number -
> then find random item from global storage and remove it from there (to not
> repeat examples) and process execution with this example.
> If we exceeded required number of executions then interrupt this Scenario
> Outline execution - and do it some how w/o error in report.

I don't see how this would work as Cucumber expands scenario outlines
in separate scenarios.


Paolo

[David Kowis]

On 02/02/2015 05:06 AM, Konstantin Trunin wrote:
> Hi.
> We have too big data set (examples section) in some test cases and we
> want to reduce execution time by checking only few random items for
> every test run.
> Ideally it would be something like parametrized tag @Random(n) where is
> is % of examples that should be taken and executed.
> E.g. if we have 100 examples and I set @Random(15) then 15 random
> examples should be taken and executed.

> *Could you please help me and clarify how to implement it?*

The best idea would be to push testing down a level or two if you can.
Highly exercise those things via unit tests, and reduce your sample size
for your Cucumber tests to a couple happy paths.

--
David

>
> I think I should create Hook, read tags.
> If it is a first time we process Scenario then read examples and shuffle
> them, save in some global storage together with counter.
> If it is not a first time then check that counter is < required number -
> then find random item from global storage and remove it from there (to
> not repeat examples) and process execution with this example.
> If we exceeded required number of executions then interrupt this
> Scenario Outline execution - and do it some how w/o error in report.
>

> --
> Posting rules: http://cukes.info/posting-rules.html
> ---
> You received this message because you are subscribed to the Google
> Groups "Cukes" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cukes+un...@googlegroups.com
> <mailto:cukes+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Konstantin Trunin]

The best way to do it is not to do it ;-)

If you are fine executing random examples, it means that the same
functionality is covered by more than one example. It is a test smell
and you should be thinking how you can split and group them. Examples
are fine to define rules and behaviour, but IMO examples should be
kept to a minimum in scenario outlines.

I really need it. This is a static data that should be processed in a specific way by the app. Because of huge amount of this data we can't check it every night. So we'd like to check it at least once per week or couple weeks. We can currently run it during weekend, but amount of such tests will grow and we need a way to randomize them.

How to make a feature request for this?

[Paolo Ambrosio]

For this kind of testing you shouldn't use Cucumber IMO. I would write JUnit/RSpec/... tests that interact with the application using the same code you have already written for Cucumber's glue code. If you write them properly (Dev + Tester pair would be great) you get the same readability of a Cucumber scenario but maintenance is massively reduced... and you can use whatever randomising logic you want because it's code!

How to make a feature request for this?

I don't think this should be implemented in Cucumber as it promotes bad practices. You can open a ticket in the issue tracker on GitHub (for the flavour of Cucumber you'd like to see it in) to start a discussion. If you do it, please reference this thread.

Paolo

[Eric Kessler]

Cucumber is a simple tool and you want to do a fancy thing. Instead of trying to make your simple tool do a fancy task, how about transforming your fancy task into a simple one that your tool can already do?

Consider breaking your test run down into two broad categories of tests:

• Your 'normal' tests
• Your 'randomized' tests

The first category is general and can be run in regular Cucumber fashion using whatever tag and path filters you desire (e.g. 'everything under this directory', 'everything with this tag'). The second category is specific and you need to be exact about which tests you want Cucumber to run (e.g. 'these three tests', 'those two outline rows').

 

Cucumber can run a specific set of tests in several ways:

• A list of specific tests (e.g. 'cucumber feature_a.feature:1 feature_a.feature:3 feature_b.feature:7')
• A file containing...wait for it...a list of specific tests (e.g. 'cucumber @my_test_file.txt')

So all you really need to do is come up with a way to generate a list of specific tests that is based on your tests that have a @Random_15 affecting them (this list will change every time that you generate it since you want tag based randomness). After that, your full test runs can be accomplished in two passes by first running your 'stable' tests and then running your 'random' tests.

Simple! ;)

As luck would have it, I generate said lists all of the time. I'll just go write up a Gist that demonstrates your particular case.

[Eric Kessler]

Two weeks of distraction and 15 minutes of actual code writing later and it is finished!

Enjoy: https://gist.github.com/enkessler/0042b9f0d3f7dce4eace

[Telmo Cardoso]

Hi Konstantin,

were you able to achieve this behavior?

Thanks

[Konstantin Trunin]

no :(

--
Posting rules: http://cukes.info/posting-rules.html
---

You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/gq1F7Tmd3Dk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.

[Eric Kessler]

On Friday, July 1, 2016 at 6:28:05 AM UTC-4, Konstantin Trunin wrote:

no :(

Not even with my handy example (https://gist.github.com/enkessler/0042b9f0d3f7dce4eace)? :(