custom scenario filtering

[Aleksandar Kostadinov]

Hello,

we want to track scenario selection based on a custom filter (that looks at an external system to get a list of scenarios). Is it possible to somehow hook our custom filter to cucumber ideally only changing `config/cucumber.yml`?

Any suggestions appreciated.

[Matt Wynne]

Hi,

I'm not quite sure what you're after but I think it's possible, yes, though you'll have to write some Ruby code.

This is a new feature in 2.0 and it's still really poorly documented at the moment.

Have a look at the acceptance test, then fire any questions you have back at me - we can use them to improve the docs!

http://www.relishapp.com/cucumber/cucumber/docs/extending-cucumber/custom-filter 

cheers,

Matt

[Aleksandar Kostadinov]

On Tue, Aug 4, 2015 at 12:52 AM, Matt Wynne <ma...@cucumber.pro> wrote:

Have a look at the acceptance test, then fire any questions you have back at me - we can use them to improve the docs!

http://www.relishapp.com/cucumber/cucumber/docs/extending-cucumber/custom-filter

This is awesome. I was asking for something that lets me select scenarios to run, but getting all scenarios one by one and filtering which one to run is still working for me. Fiddling a little bit with `pry` and reading the Cucumber::Core::Filter source got me going.

One thing I can't understand is why create a filter class by the expression `Filter.new do` instead of having Cucumber::Core::Filter just a regular class that one can extend. I think the conventional inheritance is much easier to understand than the current custom dynamic class generation.

That said, a few more examples would be IMHO the only needed thing to understand the feature. for example how to modify some particular step would be interesting.

I'll give you more details about my use-case and implementation so you have an idea what your users are doing.

  # ask test case manager if a scenario should run

  TestCaseManagerFilter = Cucumber::Core::Filter.new(:tc_manager) do
    # called upon new scenario
    def test_case(test_case)
      if tc_manager.should_run? test_case
        super
      else
        return self
      end
    end

    # called at end of execution to print summary
    def done
      tc_manager.all_test_cases_completed

      super
    end
  end

Then I have in AfterConfiguration hook:
config.filters << TestCaseManagerFilter.new(my_test_case_manager)

I am also passing to `tc_manager` other events like (in the appropriate hooks):
test_case_manager.before_scenario(scenario, err)
test_case_manager.after_scenario(scenario, err)
test_case_manager.at_exit

And my test case manager also supports attaching file artifacts to the external test management system. That's because cucumber reporting is nice, but if you run 2000 scenarios, you just can't debug them within a single log. We need logs and test artifacts split and attached to individual test cases.

Thank you a so much for the help! This implementation is so much easier than what we had to use with cucumber 1.3.x (namely external scripts to wrap around cucumber).

[Matt Wynne]

Hi Aleksandar,

On 5 Aug 2015, at 07:28, Aleksandar Kostadinov <akost...@gmail.com> wrote:

On Tue, Aug 4, 2015 at 12:52 AM, Matt Wynne <ma...@cucumber.pro> wrote:

Have a look at the acceptance test, then fire any questions you have back at me - we can use them to improve the docs!

http://www.relishapp.com/cucumber/cucumber/docs/extending-cucumber/custom-filter

This is awesome. I was asking for something that lets me select scenarios to run, but getting all scenarios one by one and filtering which one to run is still working for me. Fiddling a little bit with `pry` and reading the Cucumber::Core::Filter source got me going.

One thing I can't understand is why create a filter class by the expression `Filter.new do` instead of having Cucumber::Core::Filter just a regular class that one can extend. I think the conventional inheritance is much easier to understand than the current custom dynamic class generation.

It’s just a convenience macro, a bit like Struct.new in Ruby. It seemed like the best way to reduce boilerplate, but I appreciate it’s a bit unusual.

It’s really easy to just create a filter by hand. The protocol is just three methods, which are documented here: http://www.rubydoc.info/github/cucumber/cucumber-ruby-core/Cucumber/Core/Filter

That said, a few more examples would be IMHO the only needed thing to understand the feature. for example how to modify some particular step would be interesting.

There are quite a few examples in the codebase:

https://github.com/cucumber/cucumber-ruby-core/tree/master/lib/cucumber/core/test/filters

https://github.com/cucumber/cucumber-ruby/tree/master/lib/cucumber/filters

I'll give you more details about my use-case and implementation so you have an idea what your users are doing.

  # ask test case manager if a scenario should run

  TestCaseManagerFilter = Cucumber::Core::Filter.new(:tc_manager) do
    # called upon new scenario
    def test_case(test_case)
      if tc_manager.should_run? test_case
        super
      else
        return self
      end
    end

You could shorten the implementation of method to this:

    def test_case(test_case)

      test_case.describe_to receiver if tc_manager.should_run?(test_case)

      self

    end

    # called at end of execution to print summary
    def done
      tc_manager.all_test_cases_completed

      super
    end
  end

Then I have in AfterConfiguration hook:
config.filters << TestCaseManagerFilter.new(my_test_case_manager)

I am also passing to `tc_manager` other events like (in the appropriate hooks):
test_case_manager.before_scenario(scenario, err)
test_case_manager.after_scenario(scenario, err)
test_case_manager.at_exit

And my test case manager also supports attaching file artifacts to the external test management system. That's because cucumber reporting is nice, but if you run 2000 scenarios, you just can't debug them within a single log. We need logs and test artifacts split and attached to individual test cases.

I’d love to hear more about this. I’m sure you’re not the only one with needs like this.

Thank you a so much for the help! This implementation is so much easier than what we had to use with cucumber 1.3.x (namely external scripts to wrap around cucumber).

I’m glad to hear it. The filters protocol was actually an happy accident that fell out of the internal redesign work we did for 2.0, but I’m really pleased with it. Thank GOOS for inspiring me with “your domain model is in the protocols, not in your classes.” :)

cheers,

Matt

—

+44 20 8720 6842

Co-founder, Cucumber Limited

ma...@cucumber.io

[Aleksandar Kostadinov]

On Wed, Aug 12, 2015 at 2:44 PM, Matt Wynne <ma...@cucumber.io> wrote:
<...>

>> And my test case manager also supports attaching file artifacts to the
>> external test management system. That's because cucumber reporting is nice,
>> but if you run 2000 scenarios, you just can't debug them within a single
>> log. We need logs and test artifacts split and attached to individual test
>> cases.
>
>
> I’d love to hear more about this. I’m sure you’re not the only one with
> needs like this.

Well, there are some test case management systems like:
https://fedoraproject.org/wiki/Nitrate
https://github.com/Nitrate/Nitrate

Where you can define plans, cases, runs, etc. So you have the testing
process more organized. You run different set of tests at different
times against different environments and builds. There are commercial
solutions too. Good reporting and tracking of test execution is
needed. Now such test case manager can hook to the external system to
select scenarios to be run and update the test case runs with the
result of run and any test artefacts. It is much easier to do whit a
filter than before (although still custom logger an hooks are needed).
The challenge is in finding the most reliable places for processing
cucumber results.
I hope that we open source the test project we have when time permits.

>> Thank you a so much for the help! This implementation is so much easier than
>> what we had to use with cucumber 1.3.x (namely external scripts to wrap
>> around cucumber).
>
>
> I’m glad to hear it. The filters protocol was actually an happy accident
> that fell out of the internal redesign work we did for 2.0, but I’m really
> pleased with it. Thank GOOS for inspiring me with “your domain model is in
> the protocols, not in your classes.” :)

I must admit I don't know what you are referring to, so I'll be happy
to see any articles about this design approach.

Thank you!

[Matt Wynne]

Hi Aleksandar,

On 12 Aug 2015, at 13:55, Aleksandar Kostadinov <akost...@gmail.com> wrote:

On Wed, Aug 12, 2015 at 2:44 PM, Matt Wynne <ma...@cucumber.io> wrote:
<...>

And my test case manager also supports attaching file artifacts to the
external test management system. That's because cucumber reporting is nice,
but if you run 2000 scenarios, you just can't debug them within a single
log. We need logs and test artifacts split and attached to individual test
cases.

I’d love to hear more about this. I’m sure you’re not the only one with
needs like this.

Well, there are some test case management systems like:
https://fedoraproject.org/wiki/Nitrate
https://github.com/Nitrate/Nitrate

Where you can define plans, cases, runs, etc. So you have the testing
process more organized. You run different set of tests at different
times against different environments and builds. There are commercial
solutions too. Good reporting and tracking of test execution is
needed. Now such test case manager can hook to the external system to
select scenarios to be run and update the test case runs with the
result of run and any test artefacts. It is much easier to do whit a
filter than before (although still custom logger an hooks are needed).
The challenge is in finding the most reliable places for processing
cucumber results.

Yay! This is exactly the kind of use-case I was hoping formatters would help with. Great!

I hope that we open source the test project we have when time permits.

Thank you a so much for the help! This implementation is so much easier than
what we had to use with cucumber 1.3.x (namely external scripts to wrap
around cucumber).

I’m glad to hear it. The filters protocol was actually an happy accident
that fell out of the internal redesign work we did for 2.0, but I’m really
pleased with it. Thank GOOS for inspiring me with “your domain model is in
the protocols, not in your classes.” :)

I must admit I don't know what you are referring to, so I'll be happy
to see any articles about this design approach.

http://www.growing-object-oriented-software.com/

Thank you!

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/SiCNr8cTFsE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.