Generating Examples on the Fly

[Rob Howard]

Hello,

I'm looking to write a library that adds property-based testing to RSpec, in the style of QuickCheck[1]. 

The particular difficulties of this integration is that there there are a number of repeated tests, these tests being generated or cut short on the fly, as opposed to being all being generated in advance.

For example, when testing that a sorting function works correctly (eg. [3,1,2].my_sort == [1,2,3]), I'd generate 100 tests with different arrays of items in ever-increasing sizes. If one of the tests were to fail, I'd want to stop running further tests, and then swap over to a separate "shrinking" phase where I generate further tests with the failing output until I can get the smallest input I can generate to fail the example.

The test runs could look like (each line being an input, and an output of a test run with its result):

  Initial run (up to 100 attempts):

  [] == [], Passed

  [1] == [1], Passed

  [2,1] == [1,2], Passed

  [2,1,9,4,5,5,6,7] == [1,2,4,5,6,7,9], Failed! Stopping here; no further data generation to be done.

  

  Shrinking failed value...

  [2,1,9,4,5,5,6] == [1,2,4,5,6,9], Failed

  [2,1,9,4,5,5] == [1,2,4,5,9], Failed

  [2,1,9,4,5] == [1,2,4,5,9], Passed

  Failed, with smallest known failing value: [2,1,9,4,5,5]

To do the above with RSpec means being able to, in either an around() block or in an example itself, add further examples to an example group to run, on the fly. This doesn't appear to be possible. Even directly monkeying with an example.example_group.examples list is already too late; nothing picks up the change.

I've had the suggestion of having the ExampleGroup examples be an iterator instead of a plain list. Would I be on the right track there, or does anyone have any other suggestions?

(I'm particularly interested in an RSpec integration so as to take advantage of the existing test integrations people have, eg. database resets for ActiveRecord models, Capybara Rack-based tests, etc. There are other Quickcheck ports/adaptations (eg. Rantly[2], Rubycheck[3], Queencheck[4]), yet none of them have this RSpec lifecycle integration; the only thing that *does* is Generative[5], which just replicates single examples a certain number of times, with none of the early-stop or shrinking smarts that I'm trying to implement.)

Thanks,

Rob

1. https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing

2. https://github.com/hayeah/rantly

3. https://github.com/mcandre/rubycheck

4. https://github.com/rosylilly/QueenCheck

5. https://github.com/justincampbell/generative

[Myron Marston]

Hey Rob,

Sounds like an interesting, useful project!

Dynamically generating examples works fine as long as they are all generated during the “load spec files” phase, before the first spec runs. Defining additional examples while the examples run isn’t going to work very well. A lot of RSpec’s features (e.g. randomization and metadata filtering) need to be able to get the complete list of examples before the first one runs so that these features can do their thing. In addition, other features in 3.3 (bisect and —only-failures) rely the example IDs being stable — that is, spec/foo_spec.rb[1:10] needs to consistently reference the same example (in this case, the 10th example in the 1st example group in spec/foo_spec.rb) on each run of the suite to work properly. If you are generating additional examples as the specs run the ids may not be consistent and you’re going to break some of RSpec’s features.

My suggestion is to change how you model the problem. From what you described, you are modeling each assertion as a separate RSpec example. This is natural, given that each assertion represents an example of the property being verified and RSpec has something called an example as well. However, I think you’ll have better success if you model each property as a separate RSpec example, and then have that RSpec example delegate to a bit of code you’ve written that dynamically generates and runs as many assertions as necessary against as many input examples as necessary — and then reports the results back to RSpec in the form of no exception (if the property passed all checks) or an exception with a good, detailed message if the property failed a check.

Does that make sense?

BTW, I’m leaving for a long vacation tonight and will have sporadic access to email so I may not see or respond to replies for quite some time. I’m sure others can help out, though :).

HTH,
Myron

​

--
You received this message because you are subscribed to the Google Groups "rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rspec+un...@googlegroups.com.
To post to this group, send email to rs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/CALx9tg3ixVUQOrnu6yCdpr1TSWQhNRW_ohOJn8p0X0LCPNCE0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Howard]

Hello,

Thanks for the reply. :-)

Specifically regarding:

> However, I think you’ll have better success if you model each property as a

> separate RSpec example, and then have that RSpec example delegate to a bit of

> code you’ve written that dynamically generates and runs as many assertions as

> necessary against as many input examples as necessary — and then reports the

> results back to RSpec in the form of no exception (if the property passed all

> checks) or an exception with a good, detailed message if the property failed a

> check.

Understood. That's pretty much how you'd use, say, Rantly. For example:

  specify "+ and - reverse each other" do

    property_of {

      integer

    }.check { |n|

      expect(n + 1 - 1).to eq n

    }

  end

That's fine and all (neatened up so you have a shortcut with, say, property "+ and - ..."  or something), but the thing I was really hoping to keep was the running of the before and after hooks after each property test.

At the moment, as soon as you try something with ActiveRecord in a way that hits the database, you then only get the benefit of your existing "RSpec Example lifecycle" before and after the entire run of properties. Same with any setup for Rack testing, for example.

(I *could* come up with my own mini-DSL that'd let you specify before/after property runs, with the existing before/after(:each) acting effectively as before(:all) for the properties, but I was very much hoping to avoid something like that.)

Thanks again,

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/CADUxQmuQK9riLFoZ50pEJpERaY97FU7Yuza9mkmzbNw%2BR9uMSg%40mail.gmail.com.

[Allen Madsen]

This may not be exactly what you want, but you could generate the spec
files and then shell out to rspec to run it. You'd probably then tell
rspec to use a custom formatter and have it fail on first failure. The
exit code would tell you if it passed and the output to stdout could
be captured to read in any metadata necessary to decide what to do
next.
Allen Madsen
http://www.allenmadsen.com

> https://groups.google.com/d/msgid/rspec/CALx9tg3DaWNt%2BsG6-5uxtQ_cVxna6DB4uxCORRPhqT8gF4E5ew%40mail.gmail.com.

[Myron Marston]

If you just want to leverage RSpec's example lifecycle, and don't care to make each property example tracked by RSpec as a separate example, I think there's a pretty simple way to do this.  RSpec's example lifecycle is primarily implemented simply by running each example in the context of a new instance of the example group class -- that way, each example has a clean slate to work with and instance variables from one example don't leak into another.  Constructs like `let` in turn simply use an instance variable under the covers to provide a per-example lifecycle.  So, here's what you can do...

Instantiate a new instance of the example group class.  Here's where we do it:

https://github.com/rspec/rspec-core/blob/v3.3.2/lib/rspec/core/example_group.rb#L557

Then pass it (along with the reporter) to `Example#run` to run the example in the context of the provided instance:

https://github.com/rspec/rspec-core/blob/v3.3.2/lib/rspec/core/example.rb#L189

Bear in mind that `Example#run` is not a public API so per our current versioning policy it could be changed in any release (even a patch release).  However, if you try building something with that API, find it useful, and would like us to declare it public (such that it would only change in major versions if ever), let us know -- I think we could do that.

HTH,

Myron

To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/CAK-y3Cthyi-CYQsvL3zk8_ZaxuA%3DcUS7%3DBoMdb50Y5KELrDGfA%40mail.gmail.com.