Failed tests showing yellow square instead of red

[architkulkarni]

Unfortunately don't have a stable reproduction of this, but for now, check https://flaky-tests.ray.io/#owner=serve

test_job_submission only has one red square:

However it actually should have three red squares. It's correctly reported by the percent on the right

and also by the links

which indeed show the test failing ("FAILED in 3 out of 3 tries") in CI if you click through.