Option to keep generating examples even if < 10% satisfy assumptions

[ajdavis]

I have a situation where my assume() call rejects more than 90% of example inputs. It's hard to randomly generate inputs that pass assume() in my case, so I'd like Hypothesis to just keep trying. This code stops iterating after max_examples * 10 tries, however, with the message "settings.max_examples={s.max_examples}, but < 10% of examples satisfied assumptions". The logic is not configurable. I propose to make it configurable, if that's something you're interested in? I don't yet have a proposal for what the option will be called.

(Background: I'm generating inputs to test a program which uses a SAT solver. It's hard to predict which combinations will lead to a solution or no-solution. I don't want to figure that out, especially since whatever logic I use to make solvable inputs might be overly restrictive and not fully test the program. I'd rather keep making random inputs, assume() that the SAT solver finds a solution, then assert things about the solution.

[Liam-DeVoe]

Thanks for raising this! I'd definitely like to support this use case if possible.

One good reason for this conditional to exist is that it prevents Hypothesis from running forever when a assume truly is unsatisfiable.

Given that users probably already had to suppress HealthCheck.filter_too_much to hit this conditional, requiring 10% valid test cases out of 1000 seems high. I'm thinking of requiring 0.5-1% valid test cases instead, and making this a continuous check rather than one-time.

I'm hesitant to expose yet more configuration options for Hypothesis.

[ajdavis]

Thanks @Liam-DeVoe. Would you like me to make a PR? And if so, could you explain more about "making this a continuous check rather than one-time" so I know what kind of change you have in mind? I don't understand yet.

[Liam-DeVoe]

I suspect @Zac-HD will have some opinions on what the Hypothesis behavior should be here first ;)

Re continuous: if we want to bump down to 1%, in practice we'd change the check from happening at max_examples * 10 to max_examples * 100. 100 times max_examples feels like too long for us to exit for actually-unsatisfiable tests, so we probably want to continuously check for "if less than 1% satisfiability, exit" at every test case instead.