FlakyStrategyDefinition advice

[Evgeni Burovski]

Hi,

What are best ways to debug / fix `FlakyStrategyDefinition` errors? 

In array-api-tests we have a long-standing flake, which we failed to nail down so far. 

The strategy itself is https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_statistical_functions.py#L341-L366 --- the only somewhat unusual part is `.filter`-ing to require that the generated array has at least two elements, but this shouldn't be a problem, should it.

I recently caught the failure, and got a hypothesis-statistics log, https://github.com/data-apis/array-api-tests/issues/300#issuecomment-3944009611, but I've to admit I'm not sure what it tells me.

Would appreciate any pointers.

Cheers,

Evgenu 

[Liam DeVoe]

Hey Evgenu! Hypothesis does a pretty poor job of providing useful information to debug flaky errors. In practice, I've had success with simplifying the test definition until it's no longer flaky up to some probability, then staring at it very hard. This is helped by the only cause of FlakyStrategyDefinition being drawing values from strategies, so all unrelated test code can be removed. I recognize this is an unsatisfying answer!

For those following along, there's now an open issue for this with further discussion: https://github.com/HypothesisWorks/hypothesis/issues/4673

--
You received this message because you are subscribed to the Google Groups "Hypothesis users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypothesis-use...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/hypothesis-users/36b555da-7b27-4f72-839c-7db627c0b8fdn%40googlegroups.com.