You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to Hypothesis users
I have a stateful machine set up and it runs and passes but it uses way fewer examples than I expect. I've set the settings to have a max number of examples to more than 1,000 but I'm only seeing around ~80 examples get run (based on just manually incrementing a file). I'm tying to understand why but I'm not sure how to get started. I've set the verbosity of the test using `settings` to verbose/debug but when I run the test (with pytest -s to allow the stdout through), I don't see any output.
Also interested in at a high level what reasons a stateful test might finish early. I'd think it might be because it exhausted the search space but in this particular machine there are 10 rules all of which have no preconditions, e.g. it should be an infinite-sized space
Joey Tran
unread,
Nov 19, 2023, 4:33:00 PM11/19/23
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to Hypothesis users
I figured it out.
I didn't pay close enough attention to the docs. I had my own testcase...
``` class TestMyStatefulMachine(MyStatefulMachine.TestCase): settings = settings(max_examples=10000) ```
But the settings need to be set directly on `MyStatefulMachine.TestCase`, i.e. ``` MyStateMachine.TestCase.settings = settings(max_examples=10000) class TestMyStatefulMachine(MyStatefulMachine.TestCase): pass ```
Zac Hatfield-Dodds
unread,
Nov 19, 2023, 10:58:10 PM11/19/23
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to Hypothesis users
We already had an explicit error if you tried to assign `MyStateMachine.settings = settings(...)` after definition, and in [1] I've added a new error if you do so as a class attribute too. Hopefully that will save future users the confusion - thanks for reporting it!