Getting logging from stateful tests

20 views
Skip to first unread message

Joey Tran

unread,
Nov 19, 2023, 4:03:43 PM11/19/23
to Hypothesis users
I have a stateful machine set up and it runs and passes but it uses way fewer examples than I expect. I've set the settings to have a max number of examples to more than 1,000 but I'm only seeing around ~80 examples get run (based on just manually incrementing a file). I'm tying to understand why but I'm not sure how to get started. I've set the verbosity of the test using `settings` to verbose/debug but when I run the test (with pytest -s to allow the stdout through), I don't see any output.

Also interested in at a high level what reasons a stateful test might finish early. I'd think it might be because it exhausted the search space but in this particular machine there are 10 rules all of which have no preconditions, e.g. it should be an infinite-sized space

Joey Tran

unread,
Nov 19, 2023, 4:33:00 PM11/19/23
to Hypothesis users
I figured it out.

I didn't pay close enough attention to the docs. I had my own testcase...

```
class TestMyStatefulMachine(MyStatefulMachine.TestCase):
    settings = settings(max_examples=10000)
```

But the settings need to be set directly on `MyStatefulMachine.TestCase`, i.e.
```
MyStateMachine.TestCase.settings = settings(max_examples=10000)
class TestMyStatefulMachine(MyStatefulMachine.TestCase):
    pass
```

Zac Hatfield-Dodds

unread,
Nov 19, 2023, 10:58:10 PM11/19/23
to Hypothesis users
We already had an explicit error if you tried to assign `MyStateMachine.settings = settings(...)` after definition, and in [1] I've added a new error if you do so as a class attribute too.  Hopefully that will save future users the confusion - thanks for reporting it!

Reply all
Reply to author
Forward
0 new messages