fix: gradle test might give false green

[peterhaochen47]

• when failFast = true AND retry is on, there is a serious issue: gradle might stop the test run due to the failFast but still concludes with BUILD SUCCESSFUL (if the retry is successful)
  • we have observed this "false green" and this commit fixes this.
• impact: this might cause builds with many failing tests to take longer to finish.

[#186986697]

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/187318075

The labels on this github issue will be updated when the story is started.

[strehle]

• when failFast = true AND retry is on, there is a serious issue: gradle might stop the test run due to the failFast but still concludes with BUILD SUCCESSFUL (if the retry is successful)

  • we have observed this "false green" and this commit fixes this.

• impact: this might cause builds with many failing tests to take longer to finish.

[#186986697]

Ok, I run tests in a loop and yes there are some tests which are flaky and these tests will appear to be red then more often.

e.g. Ldap test, e.g. #2738 or #2802

[swalchemist]

@strehle There seems to be a bad interaction between the retry feature and failFast, where a test fails once and then passes on retry - failFast sees that as a failure and stops the run, and retry properly counts it as a passing test and the overall test run is reported as passing. But if there are other tests that are failing but didn't run, then we have a false green, and the test run should have been reported as failing overall.

If this means we report failures more often, that's a good thing.