Skip to content

Commit 799a304

Browse files
authored
test(ci-insights): Improve test for unhealthy tests detection (#281)
The goal of this change is to improve the test to make sure the flaky detection is not breaking CI of our users by asserting the exit code of the pytest process. References: INC-402
1 parent b7fdaaf commit 799a304

File tree

1 file changed

+6
-4
lines changed

1 file changed

+6
-4
lines changed

tests/test_ci_insights.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ def test_bar():
252252
global execution_count
253253
execution_count += 1
254254
255-
if execution_count == 1:
255+
if execution_count == 2:
256256
pytest.fail("I'm flaky!")
257257
258258
def test_baz():
@@ -266,10 +266,12 @@ def test_quux():
266266
"""
267267
)
268268

269+
# The goal is to make sure the failed rerun of the flaky test does not
270+
# impact the results to avoid failing the CI of our users.
271+
assert result.ret == 0
269272
result.assert_outcomes(
270-
failed=1, # Only the first execution of the flaky test.
271-
passed=3, # Do not log the reruns so the CI doesn't fail. The goal is to send data to Mergify, not lock the CI of our users.
272-
skipped=1, # The skipped test is tested only once because skipped tests are excluded from the flaky detection.
273+
passed=4, # Initial run of each test.
274+
skipped=1,
273275
)
274276

275277
assert re.search(

0 commit comments

Comments
 (0)