Infinite loop of retries for RetryFailed listener

[olegosh]

I have a test run set up in GCP where in options I included RetryFailed listener and skips for tags "skip":

substitutions
...
_ROBOT_OPTIONS: --listener RetryFailed --exclude skip
_HEADLESS: 'True'
...

Inside the test case I use tag:

[Tags]    test:retry(1)

I faced an issue that test run is stuck in an infinite loop. In the logs I can see messages:

...and again, and again: PASS skip FAIL skip. And it retries like this for hours until the job stops by timeout. I have a scheduled run.

I ran the same exact configuration tests run in Docker container locally and RetryFailed listener works as expected, without this infinite loop of retries.

I don't know if anyone stumbled upon this issue, but I found no similar information as to why it might happen.

I'm not sure, but I think the issue might happen because of the thing that in this block of code it sets up the result.status = "SKIP", sets the retry count back to 0, and on the next retry, which shouldn't happen, it loops back again:

def end_test(self, test, result):
        if self.retries and self._original_log_level is not None:
            BuiltIn()._context.output.set_log_level(self._original_log_level)
        if not self.max_retries:
            self.retries = 0
            return
        if result.status == "FAIL":
            if self.retries < self.max_retries:
                index = test.parent.tests.index(test)
                test.parent.tests.insert(index + 1, test)
                result.status = "SKIP"
                result.message += "\nSkipped for Retry"
                self.retried_tests.append(test.longname)
                self.retries += 1
                return
            else:
                result.message += (
                    f"{linebreak * bool(result.message)}[RETRY] FAIL on {self.retries}. retry."
                )
        else:
            if self.retries:
                result.message += (
                    f"{linebreak * bool(result.message)}[RETRY] PASS on {self.retries}. retry."
                )
        self.retries = 0
        return

The expected behavior is that it should mark test as FAIL after defined number of retries, and then skip to the next test run

[zawazingo]

Hi,
I'm encountering the same problem where our Jenkins pipeline experiences prolonged delays whenever a test fails during execution.
Any assistance you can provide would be greatly appreciated.

[olegosh]

I think I've solved this issue:

The issue was that I configured tests run the way the --listener RetryFailed was in 2 different places:

1. in .sh test run script

xvfb-run \
    ...
    --listener RetryFailed \
    ...

2. in substitutions in cloudbuild.yaml

substitutions:
    ...
    ROBOT_OPTIONS: --listener RetryFailed --exclude skip
    ....

I removed --listener RetryFailed from cloudbuild.yaml and now it runs as expected, without the infinite loop of retries for the same failed test