How can I specify choice sequences to run for a test?

[Jacob Yu]

Hi all!

I am interested in using the "generators are parsers for random sequences" property to do some machine-learning experiments. For instance, I'm curious if it can infer some strategies used in test case reduction.

For this, I am trying to expose the random number sequence used by the generators directly for the machine learning algorithm to optimise. When a test is run, I'd like to recover the sequence, as well as the tree structure that arose from using the input sequence.

More specifically, for a Strategy that generates values of type T, I'd like to obtain from it a function of type:

- RandomSequence -> (T, SequenceTree)

where SequenceTree has the call-stack structure attached to it, based on what parts of the sequence were drawn by which generators.

I'm wondering what are the ways to obtain something like this. I am quite new to Hypothesis itself, so apologies if this is a basic question! :)

Best regards,

Jacob

[Liam DeVoe]

Hi Jacob!

I think the best way to go about this is with a custom backend, which is called each time Hypothesis generates a new test case and provides overridable functions like `span_start`. I recommend subclassing HypothesisProvider, and overriding span_start, span_end, and the five draw_* functions which correspond to the five types in the choice sequence; the draw_* functions let you recover the linear choice sequence, and imposing the spans on top of that lets you recover the choice sequence tree. You'll want to return `super().foo` for each function you override, unless you want to change the generated distribution.

(The choice sequence in Hypothesis used to be a sequence of bytes, which may be what you're referring to by the "random number sequence". Nowadays, the choice sequence has five types: boo, int, str, bytes, and float. You can read more about that on this issue).

Here's a starting example: https://gist.github.com/tybug/c247ae7fb14d5c5d920653440d736a63. The backend docs on the website aren't quite filled out yet, but you can check the PrimitiveProvider class in providers.py for docstrings, or hypothesis-crosshair for an implementation example. 

The caveat of the backend approach is that we do not currently use custom backends for Phase.shrink (or Phase.reuse), so you won't get insight into what choice sequences are generated during shrinking. I've submitted a PR which uses alternative backends for all phases, but the interactions are complicated, and it may not get merged as-is.

--
You received this message because you are subscribed to the Google Groups "Hypothesis users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypothesis-use...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/hypothesis-users/5d6f1b56-850d-497f-81b3-b1e81c35f621n%40googlegroups.com.

[Jacob Yu]

Dear Liam,

Thank you so much for the detailed explanation, code example and the PR! It's super useful and clarified a lot :D

I read through the discussion of the typed sequences PR, it's a very cool change :) Currently though I am have a little trouble understanding the new typed choice sequences, and its use in mutating the input. 

I understood that in the old bytes based choice sequence, _any_ sequence would parse into a valid output by the strategy. However, it doesn't seem like we have this property in the typed sequence - might we run into type mismatch errors (like, suplying a string when the strategy expects an int) for some mutations of the sequence? 

If I had to guess, do we have separate sequences for each type supplied, or perhaps are reductions done in a controlled way to maintain the typing of the original sequence?

Best regards,

Jacob 

---

发件人: Liam DeVoe <orion...@gmail.com>
发送时间: 星期二, 五月 6, 2025 6:25:37 上午
收件人: Jacob Yu <ping...@gmail.com>
抄送: Hypothesis users <hypothes...@googlegroups.com>
主题: Re: How can I specify choice sequences to run for a test?

[Liam DeVoe]

> I understood that in the old bytes based choice sequence, _any_ sequence would parse into a valid output by the strategy. However, it doesn't seem like we have this property in the typed sequence - might we run into type mismatch errors (like, suplying a string when the strategy expects an int) for some mutations of the sequence?

Correct! We call this a "misalignment". You can search for "misalign" in the Hypothesis codebase for more context, and particularly read this comment. The short answer is that while the typed choice sequence loses the "all choice sequences parse to a valid input" property in theory, we address this in practice with clever shrinking tricks, and by supplying the "0-index" choice as a default when we run into a misalignment (aka the simplest possible choice, given the available constraints - so the index 0 choice of st.integers() is 0, and the index 0 choice of st.integers(min_value=42) is 42). Note that misalignment can only happen during mutation, not during generation.

By the way, after discussion on the above PR, I now think that our observability feature is more suited to your use case than a backend. We don't yet expose the choice sequence in observability, but I will submit a PR soon and let you know when we do!

[Liam DeVoe]

We now expose the choice sequence (and choice sequence spans; see PrimitiveProvider.span_start for details about that) in observability, when the OBSERVABILITY_CHOICES flag is set! See also this PR. 

As noted in the docs, this parameter is experimental. It should get you the choice sequence information you need, but we hope to move towards a more stable interface in the future.

I'd be excited to hear how it goes!