Allow polling until a test succeeds #201
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexfmpe
commented
Nov 2, 2022
| where | ||
| go = do | ||
| threadDelay pollInterval | ||
| catch test $ \(e :: HUnitFailure) -> go |
There was a problem hiding this comment.
Note this type is stated to be internal only:
https://hackage.haskell.org/package/HUnit-1.6.2.0/docs/Test-HUnit-Lang.html#g:1
I didn't find a suitable public API equivalent though
alexfmpe
force-pushed
the
test-delay-poll
branch
from
4066ee2
to
0a74513
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#200 turned out to be caused by the test not being yet finished in time. Duplicating the delay here
rhyolite/beam/task/backend/test/Test.hs
Line 254 in 151649c
FWIW, I'm running Ryzen 1800X, meaning 8 cores 16 threads. I suspect this test might need even more time on machines with less cores, unless it's IO bound?
Another downside is this makes the test duration increase from around 26s to around 34s on my machine.
This PR suggests an alternative: to keep running the test until success, subject to a generous timeout, rather than guessing delays and coarsely trade-offing false negatives against test duration.
I've converted only the offending test to validate the approach, but AFAICT the other tests would also benefit.