Bugfix MTE-4133 Enable retries in fastlane #24251
Conversation
| @@ -46,7 +46,7 @@ for lang in $LOCALES; do | |||
| if [ "$?" != "0" ]; then | |||
| echo "Fastlane exited with code: $?" | |||
| exit $? | |||
| elif grep -q "** TEST FAILED **"; then | |||
| elif grep -q "TEST FAILED" "output.txt"; then | |||
There was a problem hiding this comment.
"output.txt" was missing previously, so this line might not be working properly before.
firefox-ios/l10n-screenshots.sh
Outdated
| @@ -34,7 +34,7 @@ for lang in $LOCALES; do | |||
| echo "$(date) Snapshotting $lang" | |||
| mkdir "l10n-screenshots/$lang" | |||
| fastlane snapshot --project firefox-ios/Client.xcodeproj --scheme L10nSnapshotTests \ | |||
| --number_of_retries 0 \ | |||
| --number_of_retries 3 \ | |||
There was a problem hiding this comment.
I can't tell if the default value has changed or not by reading the documentation. Let's hard code this value.
There was a problem hiding this comment.
👀 testHistoryTableContextMenu failed. The whole list of tests are rerun. This may not be what we want.
https://app.bitrise.io/build/bc7e1954-3780-4157-afeb-51e10ad9ff3d?tab=log
There was a problem hiding this comment.
no, we don't want retries. Just one execution per locale no matter the reesult. Please see explanation here:https://mozilla.slack.com/archives/C02KSH6QNBS/p1737374666180979?thread_ts=1737374476.241319&cid=C02KSH6QNBS
There was a problem hiding this comment.
This option is ignored now by fastlane. We should use set the retry opion to off from xcode and to do that, we need to create a test plan for the L10n schema where we can configure that. I have done that locally, forced a test to fail, and there are no retries.
Please try that and let me know in case you want a hack session to do that together.
There was a problem hiding this comment.
I have created a L10nSnapshotTests test plan and have the schema to use the test plan.
| --skip_open_summary \ | ||
| --xcargs "-maximum-parallel-testing-workers 2" \ | ||
| --derived_data_path l10n-screenshots-dd \ | ||
| --ios_version "18.2" \ | ||
| --erase_simulator --localize_simulator \ | ||
| --devices "iPhone 16" --languages "$lang" \ | ||
| --output_directory "l10n-screenshots/$lang" \ | ||
| --xcodebuild_formatter xcbeautify \ |
There was a problem hiding this comment.
The default formatter on Bitrise is now xcbeautify. I have intermittent issues on xcpretty currently with the latest fastlane. Let me ensure that I use the same formatter locally.
| } | ||
| ], | ||
| "defaultOptions" : { | ||
| "testRepetitionMode" : "retryOnFailure", |
There was a problem hiding this comment.
I have enabled "Retry on Failure" for the test plan's config. The default value is 3.
There was a problem hiding this comment.
When I run the test on Xcode 16, I see the following line in the logs:
Test Case '-[L10nSnapshotTests.L10nSuite1SnapshotTests test3ReloadButtonContextMenu]' started (Iteration 1 of 3).
In addition, a test retries 3 times when I made it to fail on purpose.
There was a problem hiding this comment.
we should NOT retry on failure. Please read my comments above and on slack about that.
The option --number_of_retries 0 is being ignored. By adding a test plan what we want is to have the option to set the retry to off in xcode. It is being ignored with the fastlane option but looks like if we have it enabled via xcode it works.
We don't want tests to retry because we run 6 locales per build, if there are retries, it takes too long to run each set of locales that builds timeout in Bitrise. If a test fail, we only miss that screenshoot, the rest are generated
📜 Tickets
Jira ticket
💡 Description
Let's enable retries on the L10n test plan so that we will get some screenshots in just in case of flaky tests.
Bitrise L10nBuild workflow: https://app.bitrise.io/build/f078a5ac-5f28-4a23-b444-6fd7238d012f?tab=log
📝 Checklist
You have to check all boxes before merging
@Mergifyio backport release/v120)