How to write a plugin to handle property testing?

[Wiebe-Marten Wijnja]

Greetings, dear Rspec developers and users!

I am currently in the process of writing a Property Testing ('QuickCheck'-like) library, because while there exist a couple of gems that allow this,

they are either (a) very feature-incomplete and do not properly support shrinking, and/or (b) unmaintained for a very long time.

The basics of the library work, but I want to be able to integrate it with Rspec.

Essentially, this means that I want to introduce an alternate keyword instead of 'it', (named 'forall') which would call the block passed to it many times (a thousand by default) with different values of increasing complexity.

As soon as one call/set of inputs  is found to cause a failure, the library will then shrink this set of inputs to be the simplest set of inputs that still causes a failure.

The current implementation of this works, but it assumes that all failures will be raised as exceptions, and that failures will not be captured/logged by some reporting mechanism in-between.

w.r.t. Rspec this means that I have the following questions:

1. How can I/should I wrap the test execution context?

3. Most importantly, I want to report the initial failure that happens (with the set of inputs at that time), as well as the failure that happens on the simplest shrunken set of inputs.

    I do not want Rspec to report on the fifty failures that happened while shrinking.

Thank you,

~Qqwy/ W-M

[Jon Rowe]

Hi

The easiest way to do this will be to consider your “keyword” as a DSL that creates examples. Then within the examples you create handle the logic of running the block provided to perform the work you want. You will need to swallow the errors yourself if you don’t want RSpec to perform them. This is very much pseudo code but might help point you in the right direction...

module PropertyTesting

  module_function

  def test_properties(properties, &block)

    # for simplicity use the existing DSL to create examples

    it “tests the properties” do

      begin

        properties.each do |property|

          block.call(property)

        end

      # its up to you what you rescue here, either all errors or just ExpectationNotMet

      rescue

        shrink

      ensure

        # You can create either a new ExpectationNotMet error with whatever you want

        # or you can use standard ruby errors, or you could create an AggregateFailures 

        # error with the original and/or the new error, or any other errors. Up to you.

        raise a_final_rspec_error if errors

      end

    end

  end

end

Let me know if I can help further 

Cheers

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

[Wiebe-Marten Wijnja]

Thank you for your response! :D

In the meantime I also found an old (2015) discussion on the Rspec google groups about virtually the same topic:

https://groups.google.com/forum/#!msg/rspec/XgY5wHI-S_c/whRRWbwDBwAJ

The main issue that was mentioned there is that running all examples of a property test inside `it` (using the technique of e.g. your example pseudo-code)

means that before/after(:each) hooks are only executed once for the entire run, rather than once per generated input value-set.

I will probably end up using the approach proposed by you for now for simplicity's sake,

but if there exists a way that allows users to leverage the existing test lifecycle hooks that RSpec provides, that would of course be much nicer :-).

Sincerely,

~W-M/Qqwy

[Jon Rowe]

The problem is that hooks are designed for a complete test lifecycle, and you want to surpress that lifecycle (the reporting of failures). There is a way around this but it’s more complicated.

If you want to use all of RSpecs config / lifecycle per run of a property you are going to have run each as a seperate example. To then achieve the suppression of spurious failures / the correct output you want you’re going to have build a custom formatter that knows about what your doing and can ignore those failures and take account of them in the final output. To be compatible with normal tests it will have to also be able to act as a normal formatter and differentiate the behaviour. You’ll also have to correct the spurious run results.

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

On 24 June 2019 at 15:01, Wiebe-Marten Wijnja wrote:

Thank you for your response! :D

In the meantime I also found an old (2015) discussion on the Rspec google groups about virtually the same topic:

[Wiebe-Marten Wijnja]

Thank you for your response. This seems like quite a big undertaking. Also, would having a custom formatter mean that it would be impossible for people to use their own custom formatter?

I think for now I'll choose the easier (but slightly less satisfying) approach of having the user specify lifecycle methods in the property itself explicitly. (And maybe have a custom way to run the same before/after actions for a whole group of property tests.)

[Jon Rowe]

Yes it would be fairly involved. The most seamless way would be to create and run example groups and examples yourself using that custom formatter, then feeding back into the existing run cycle. You can see why this hasn’t been tackled to satisfaction before!

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

[Wiebe-Marten Wijnja]

All right, I am currently considering an alternate approach, in which someone can

Rspec.describe MyFancyStuff do
  ...
  # In any (nested or not) ExampleGroup:
  require 'prop_check/rspec'
  extend PropCheck::RSpec
  ...
end

to bring not only `forall` into scope, but also override `before/after/around` to handle one extra context, which I have tentatively called `:each_prop_check_iteration`.

The thing I am currently struggling with however, is how to correctly keep track of the blocks that someone wants to add as hooks, and then to call them correctly from within an example context

(Or, to be precise: From within a block within an `it "description" do ... end`)

At first I attempted adding instance methods, which did not work because each example runs its own instance.

Then I attempted working with class-methods on the ExampleGroup, which did not work because by default when an example runs it prevents methods that are not part of the `ExampleGroup` class itself from being run, raising a WrongScopeError (see e.g. https://github.com/rspec/rspec-core/blob/2a62a644b52536d44a7969b5c5b69077a35687ca/lib/rspec/core/example_group.rb#L730).

Then I tried adding it to the `metadata`, but you do not have access to it from within a running example unless you have full control over the block that `it` is being called with (the only way to access the example's metadata is if you specify a parameter to the block you pass to `it`).

--

What would be the way to accomplish this?

[Jon Rowe]

Each `describe` / `context` creates a class, each test is an instance. You can create (with your own DSL) a module thats mixed into the class to provide the functionality you want.

I’d be creating each additional hook as a non memoized method, then having a `propcheck` or similar DSL which is invoked with test that calls these methods each time the block is invoked.

Apologies for the delayed response, under a pile of mail atm!

Cheers

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

On 30 June 2019 at 09:11, Wiebe-Marten Wijnja wrote:

All right, I am currently considering an alternate approach, in which someone can

Rspec.describe MyFancyStuff do

 

...

 

# In any (nested or not) ExampleGroup:

 

require 'prop_check/rspec'

  extend

PropCheck::RSpec

 

...
end

to bring not only `forall` into scope, but also override `before/after/around` to handle one extra context, which I have tentatively called `:each_prop_check_iteration`.

The thing I am currently struggling with however, is how to correctly keep track of the blocks that someone wants to add as hooks, and then to call them correctly from within an example context

(Or, to be precise: From within a block within an `it "description" do ... end`)

At first I attempted adding instance methods, which did not work because each example runs its own instance.

Then I attempted working with class-methods on the ExampleGroup, which did not work because by default when an example runs it prevents methods that are not part of the `ExampleGroup` class itself from being run, raising a WrongScopeError (see e.g. github.com/rspec/rspec-core/blob/2a62a644b52536d44a7969b5c5b69077a35687ca/lib/rspec/core/example_group.rb#L730).

Then I tried adding it to the `metadata`, but you do not have access to it from within a running example unless you have full control over the block that `it` is being called with (the only way to access the example's metadata is if you specify a parameter to the block you pass to `it`).

What would be the way to accomplish this?

[Wiebe-Marten Wijnja]

Thank you for your 'late' reply... allow me to reply even later (10 months after your reply ^_^').

One problem I encountered, is that RSpec seems to require that examples are first created, and only then they are all run. Is this true?

For property-testing it makes sense to run the tests for a particular property only until the first failure occurs (and at that time we'd want to shrink the input back to the most basic input that creates the same error).

Is this approach fundamentally incompatible with RSpec's architecture, or is there some way to do this? ('this' being to define more tests on the fly.)

~Marten/Qqwy

[Jon Rowe]

So it is possible, but difficult, to register new tests within an RSpec suite during a run.

However I’m not sure thats what you want, and its not what I was suggesting.

I was suggesting that a `prop_check` alias, or similar, creates one test, but that within that test it runs its block multiple times for each property.

Something along the lines of:

def self.prop_check(string, &block)

  it(string) do

    generate_properties.each(&block)

  end

end

Which is pseudo code, but you can see if you had a property generator that kept running the “test block”until it satisfied conditions, or failed once, you could achieve what you want.

RSpec supports wrapping exceptions into one (it powers our aggregate failures functionality) so you could have one test representing multiple values like this easily enough.

Cheers

Jon Rowe

---------------------------

ma...@jonrowe.co.uk

jonrowe.co.uk

[Wiebe-Marten Wijnja]

Thank you for your reply!

I would like to do both: Create an alias that under the hood registers new tests at runtime.

The current implementation does the former (the alias) successfully, based on the tips you and others gave earlier in this thread :-).

I am still looking for how to register new tests while the system is running. This is the reason why:

Running all tests for a single property within a single 'it' essentially clusters e.g. 1000 tests into a single one.

This poses a problem for tools that notify users on 'slow tests' and similar metrics, because these metrics won't be meaningful anymore.