Stateful testing and pytest fixtures

[Tin Tvrtković]

Hello, me again.

Trying to get Hypothesis stateful testing to play nice with pytest fixtures. I'm trying different things but nothing seems to work. Maybe the best way forward would be to just have a normal pytest test function, receive the fixtures as arguments, set the fixtures as class attributes on my rule machine subclass and invoke run_state_machine_as_test directly.

I'm totally willing to accept "pytest doesn't really support this" as a valid response, but if there's a nice way I'd be interested to know. :)

[Tin Tvrtković]

Also, can the data strategy be used with stateful testing? I seem to be getting "hypothesis.errors.InvalidArgument: Cannot use strategy data() within a call to find (presumably because it would be invalid after the call had ended)." when trying to do so. Presumably because find_breaking_runner uses find first?

[David MacIver]

On 2 May 2016 at 13:31, Tin Tvrtković <tinch...@gmail.com> wrote:

Also, can the data strategy be used with stateful testing? I seem to be getting "hypothesis.errors.InvalidArgument: Cannot use strategy data() within a call to find (presumably because it would be invalid after the call had ended)." when trying to do so. Presumably because find_breaking_runner uses find first?

If you upgrade to the latest version (3.1.3) that should work now.

 

On Monday, May 2, 2016 at 1:58:47 PM UTC+2, Tin Tvrtković wrote:

Hello, me again.

Trying to get Hypothesis stateful testing to play nice with pytest fixtures. I'm trying different things but nothing seems to work. Maybe the best way forward would be to just have a normal pytest test function, receive the fixtures as arguments, set the fixtures as class attributes on my rule machine subclass and invoke run_state_machine_as_test directly.

That's probably the best way for now, though you can simplify it slightly: run_state_machine_as_test can take any function that takes zero arguments and returns a state machine object. So instead of messing around with class attributes you can just give it a normal __init__ (don't forget to call super!) and call run_state_machine_as_test with a lambda that passes the fixtures to that init.

I don't think there's any intrinsic difficulty with supporting fixtures in state machines to the same degree that they're supported in given, but there's not currently any API in Hypothesis that has that work out of the box.

 

I'm totally willing to accept "pytest doesn't really support this" as a valid response, but if there's a nice way I'd be interested to know. :)

--
You received this message because you are subscribed to the Google Groups "Hypothesis users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypothesis-use...@googlegroups.com.
To post to this group, send email to hypothes...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/hypothesis-users/9738e0b5-7d1f-499b-b67e-598e37918f4b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tin Tvrtković]

Thanks for the tips. Here's what I ended up doing:

@pytest.mark.usefixtures('dummy_profiles')
def test_level_obj_bought(fake_user: User, fake_profile: GameProfile,
                          random_profileset: ProfileSet):
    factories = sampled_from(random_profileset.factories)
    ovens = sampled_from(random_profileset.ovens)
    uids = text(min_size=8, max_size=8, alphabet=string.ascii_uppercase + string.digits)

    class LevelObjectBoughtMachine(RuleBasedStateMachine):
        def __init__(self, fake_user: User, fake_profile):
            super().__init__()
            self.fake_user = fake_user
            self.fake_profile = fake_profile
            # Clear stuff
           
        @rule(factory=factories, uid=uids)
        def test_factory_bought(self, factory: Factory, uid: str):
            # apply events

        @rule()
        @precondition(lambda self: self.fake_user.savegame.restaurant.levelObjects)
        def test_unique_uids(self):
            # assert properties

    factory = partial(LevelObjectBoughtMachine, fake_user, fake_profile)
    factory.__name__ = LevelObjectBoughtMachine.__name__
    run_state_machine_as_test(factory, settings(max_examples=1000))

So I'm defining the rule machine inside the function. I'm actually totally ok with this, since it means I can use strategies that depend on my fixtures. Technically I don't even need the partial thing there (or to pass stuff to __init__), I can just access the fixtures from the test function scope. In fact that might even be better since it hammers home the fact these are global, mutable things we're using.

Interestingly Hypothesis needs the __name__ of the factory, for its database I'm guessing?

Anyway, maybe this pattern should be documented somewhere? Feels useful. Thanks for the help.