Bugfix MTE-4133 Disable retries in fastlane #24251

clarmso · 2025-01-20T19:46:47Z

📜 Tickets

Jira ticket

💡 Description

Let's enable retries on the L10n test plan so that we will get some screenshots in just in case of flaky tests.

Bitrise L10nBuild workflow: https://app.bitrise.io/build/f078a5ac-5f28-4a23-b444-6fd7238d012f?tab=log

📝 Checklist

You have to check all boxes before merging

Filled in the above information (tickets numbers and description of your work)
Updated the PR name to follow our PR naming guidelines
Wrote unit tests and/or ensured the tests suite is passing
When working on UI, I checked and implemented accessibility (minimum Dynamic Text and VoiceOver)
If needed, I updated documentation / comments for complex code and public methods
If needed, added a backport comment (example @Mergifyio backport release/v120)

clarmso · 2025-01-20T19:47:16Z

firefox-ios/l10n-screenshots.sh

@@ -46,7 +46,7 @@ for lang in $LOCALES; do
    if [ "$?" != "0" ]; then
        echo "Fastlane exited with code: $?"
        exit $?
-    elif grep -q "** TEST FAILED **"; then
+    elif grep -q "TEST FAILED" "output.txt"; then


"output.txt" was missing previously, so this line might not be working properly before.

clarmso · 2025-01-20T19:48:25Z

firefox-ios/l10n-screenshots.sh

@@ -34,7 +34,7 @@ for lang in $LOCALES; do
    echo "$(date) Snapshotting $lang"
    mkdir "l10n-screenshots/$lang"
    fastlane snapshot --project firefox-ios/Client.xcodeproj --scheme L10nSnapshotTests \
-        --number_of_retries 0 \
+        --number_of_retries 3 \


I can't tell if the default value has changed or not by reading the documentation. Let's hard code this value.

👀 testHistoryTableContextMenu failed. The whole list of tests are rerun. This may not be what we want.
https://app.bitrise.io/build/bc7e1954-3780-4157-afeb-51e10ad9ff3d?tab=log

no, we don't want retries. Just one execution per locale no matter the reesult. Please see explanation here:https://mozilla.slack.com/archives/C02KSH6QNBS/p1737374666180979?thread_ts=1737374476.241319&cid=C02KSH6QNBS

This option is ignored now by fastlane. We should use set the retry opion to off from xcode and to do that, we need to create a test plan for the L10n schema where we can configure that. I have done that locally, forced a test to fail, and there are no retries.
Please try that and let me know in case you want a hack session to do that together.

I have created a L10nSnapshotTests test plan and have the schema to use the test plan.

clarmso · 2025-01-20T20:50:03Z

firefox-ios/l10n-screenshots.sh

        --skip_open_summary \
        --xcargs "-maximum-parallel-testing-workers 2" \
        --derived_data_path l10n-screenshots-dd \
        --ios_version "18.2" \
        --erase_simulator --localize_simulator \
        --devices "iPhone 16" --languages "$lang" \
        --output_directory "l10n-screenshots/$lang" \
+        --xcodebuild_formatter xcbeautify \


The default formatter on Bitrise is now xcbeautify. I have intermittent issues on xcpretty currently with the latest fastlane. Let me ensure that I use the same formatter locally.

clarmso · 2025-01-22T06:06:03Z

firefox-ios/firefox-ios-tests/Tests/L10nSnapshotTests.xctestplan

+    }
+  ],
+  "defaultOptions" : {
+    "testRepetitionMode" : "retryOnFailure",


I have enabled "Retry on Failure" for the test plan's config. The default value is 3.

When I run the test on Xcode 16, I see the following line in the logs:

Test Case '-[L10nSnapshotTests.L10nSuite1SnapshotTests test3ReloadButtonContextMenu]' started (Iteration 1 of 3).

In addition, a test retries 3 times when I made it to fail on purpose.

we should NOT retry on failure. Please read my comments above and on slack about that.
The option --number_of_retries 0 is being ignored. By adding a test plan what we want is to have the option to set the retry to off in xcode. It is being ignored with the fastlane option but looks like if we have it enabled via xcode it works.
We don't want tests to retry because we run 6 locales per build, if there are retries, it takes too long to run each set of locales that builds timeout in Bitrise. If a test fail, we only miss that screenshoot, the rest are generated

clarmso · 2025-01-22T15:59:31Z

Note that the formatting tests for focus-ios are failing. Let me recreate on another branch to see if anything from main fixes the issue.

…l to use nested configurations (#24219) * Merge exclude paths from focus-ios to root config * Disable rules that trigger violations in focus-ios using a nested configuration * Updated Swiftlint run phases of each subprojects to use nested configs * Update Bitrise configuration to make use of nested configurations --------- Co-authored-by: tonell_m <[email protected]>

Adding logs

… Widget (#24244) * Completed Ticket * Linter error fixed

* Fix closure length violation in Focus. * Fix to properly swiftlint ignore nimbus generated files in Focus.

… setting's "Learn more" link (#24205)

* Add FXIOS-10902 Firefox iOS: Send data deletion request when DAU ping is toggled off * Update data review link * Fix PR comments

…efoxBeta (#24196) Remove universal links logic

…0118050318 (#24230) * Auto update SPM with latest rust-component release 136.0.20250118050318 * Fix build issue with new ftsMatchInfo type added to Suggestion.amp case. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: ih-codes <[email protected]>

…isabled when tapping on secure connection and returning to panel (#24264) FXIOS-11127 #24262 ⁃ [Felt privacy - unified panel] - ETP toggle is disabled when tapping on secure connection and returning to panel

…ion for context menu after row reorder (#24181)

…tint color (#24183)

[FXIOS-11111] Make Swiftlint pre-push hook quiet

…n endpoint in beta (#24255) Ensure we use production endpoint in beta

…es (#24253) * - Moved default browser user default to default browser util - Change logic of rating prompt, remove cumulative days of use counter - Call rating prompt on init of BVC - Remove rating prompt from dependency container * Fix tests * Use time interval instead of calendar

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…r telemetry (#24257) * Ensure we send to both places for telemetry * Add tests, remove static from SponsoredTileTelemetry todo so * Fix * Remove savedEvent and replace with SavedEvents * Add event check on array * Adjust for latest main

…Terms of Service' card to maintain consistency with the other Onboarding Cards (#24280) * FXIOS-11138 [Terms of service] Landscape mode should not be supported for 'Terms of Service' card to maintain consistency with the other Onboarding Cards * Moved override methods inside the main class

FXIOS-11128 [Terms of service] - "Continue" is written with lowercase

clarmso · 2025-01-22T19:17:33Z

Cherry pick to a new branch. #24285

mobiletest-ci-bot · 2025-01-22T19:30:00Z

	Warnings
⚠️	Pull Request size seems relatively large. If this Pull Request contains multiple changes, please split each into separate PR will helps faster, easier review. Consider using epic branches for work that would affect main.

	Messages
📖	Project coverage: 33.95%
📖	Edited 121 files
📖	Created 3 files

Client.app: Coverage: 32.37

File	Coverage
TrackingProtectionViewController.swift	12.14%	⚠️
EditBookmarkViewController.swift	36.59%	⚠️
BrowserViewController.swift	3.53%	⚠️
LegacyFeatureFlagsManager.swift	95.6%	✅
BookmarksCoordinator.swift	73.68%	✅
TermsOfServiceViewController.swift	81.42%	✅
HomepageContextMenuHelper.swift	3.99%	⚠️
GleanUsageReporting.swift	67.07%	✅
GleanPlumbContextProvider.swift	100.0%	✅
LegacyBookmarksPanel.swift	39.7%	⚠️
AppDelegate.swift	0.0%	⚠️
BookmarksViewController.swift	8.91%	⚠️
TopSitesViewModel.swift	33.48%	⚠️
RouteBuilder.swift	88.8%	✅
DefaultBrowserUtil.swift	88.06%	✅
UnifiedAdsCallbackTelemetry.swift	86.57%	✅
BaseCoordinator.swift	93.94%	✅
TopSite.swift	46.67%	⚠️
ActionProviderBuilder.swift	0.0%	⚠️
BookmarksTelemetry.swift	100.0%	✅
EditBookmarkCell.swift	0.0%	⚠️
NimbusFlaggableFeature.swift	99.3%	✅
TopSiteState.swift	54.93%	✅
SponsoredTileTelemetry.swift	100.0%	✅
EditFolderViewController.swift	43.56%	⚠️
DependencyHelper.swift	0.0%	⚠️
NimbusFeatureFlagLayer.swift	63.39%	✅
OneLineTableViewCell.swift	0.0%	⚠️
TelemetryWrapper.swift	66.9%	✅
SceneDelegate.swift	0.0%	⚠️
LegacyBookmarkDetailPanel.swift	17.98%	⚠️
BookmarksFolderCell.swift	0.0%	⚠️
BookmarksSaver.swift	97.35%	✅
RatingPromptManager.swift	97.58%	✅
PrivacyPreferencesViewController.swift	0.0%	⚠️
GleanWrapper.swift	16.07%	⚠️
CoreFlaggableFeature.swift	100.0%	✅

WidgetKitExtension.appex: Coverage: 7.05

File	Coverage
OpenTabsWidget.swift	0.0%	⚠️

libStorage.a: Coverage: 55.32

File	Coverage
RustFirefoxSuggestion.swift	23.53%	⚠️

Generated by 🚫 Danger Swift against 46fc441

Enable retries in fastlane

39e6d4a

clarmso requested review from isabelrios and mdotb-moz January 20, 2025 19:46

clarmso requested a review from a team as a code owner January 20, 2025 19:46

clarmso commented Jan 20, 2025

View reviewed changes

Hardcode xcbeautify as the formatter. It's the default on Bitrise

c5635bf

clarmso commented Jan 20, 2025

View reviewed changes

clarmso added 2 commits January 20, 2025 16:24

Sometimes we can open the tab tray

9a554a1

Create L10nSnapshotTests test plan. Enable test retries.

c2ab494

clarmso commented Jan 22, 2025

View reviewed changes

clarmso added 2 commits January 22, 2025 01:11

Use test plan for L10n

c54a8e1

Use old wait for now.

c61388c

isabelrios changed the title ~~Bugfix MTE-4133 Enable retries in fastlane~~ Bugfix MTE-4133 Disable retries in fastlane Jan 22, 2025

Disable retries for L10n

5638e8f

Marceau TONELLI and others added 14 commits January 22, 2025 14:16

Add FXIOS-10386 #22756 Add debug logs for handling route (#24218)

c536cc0

Adding logs

Bugfix FXIOS-10601 - User Not Directed To Correct Website Firefox-iOS…

ce86e9c

… Widget (#24244) * Completed Ticket * Linter error fixed

Bugfix FXIOS-7301 Fix swiftlint build errors on main (#24259)

dc718ff

* Fix closure length violation in Focus. * Fix to properly swiftlint ignore nimbus generated files in Focus.

Add FXIOS-11098 (Focus iOS) Update "Automatically Send Crash Reports"…

1f0779a

… setting's "Learn more" link (#24205)

Add FXIOS-10666 Implement iOS "onboarding-opt-out" ping (#24101)

42dd48f

* Add FXIOS-10902 Firefox iOS: Send data deletion request when DAU ping is toggled off * Update data review link * Fix PR comments

Remove [FXIOS-10299] remove universal link logic from Firefox and Fir…

0b6048f

…efoxBeta (#24196) Remove universal links logic

Bugfix FXIOS-11078 [Bookmarks Evolution] Incorrect index path resolut…

6a28ae6

…ion for context menu after row reorder (#24181)

Bugfix FXIOS-11081 [Bookmarks Evolution] Bookmarks disclosure button …

7588b5a

…tint color (#24183)

Refactor FXIOS-11111 Make SwiftLint pre-push use --quiet (#24229)

c199e00

[FXIOS-11111] Make Swiftlint pre-push hook quiet

Refactor FXIOS-11120 #24247 [Sponsored tiles] Ensure we use productio…

edc56e7

…n endpoint in beta (#24255) Ensure we use production endpoint in beta

Refactor FXIOS-11071 bookmarks telemetry into it's own struct (#24227)

3312fd5

lmarceau and others added 6 commits January 22, 2025 14:16

focus-string-import-01-20-2025 (#24238)

ee1ea73

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Add MTE-4138 - long press share actions tests (#24242)

0f19ac3

Bugfix FXIOS-11128 [ToS] - "Continue" is written with lowercase (#24281)

46fc441

FXIOS-11128 [Terms of service] - "Continue" is written with lowercase

clarmso requested a review from a team as a code owner January 22, 2025 19:16

clarmso closed this Jan 22, 2025

rvandermeulen deleted the cs/MTE-4133-bitrise-retries branch February 11, 2025 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bugfix MTE-4133 Disable retries in fastlane #24251

Bugfix MTE-4133 Disable retries in fastlane #24251

Uh oh!

clarmso commented Jan 20, 2025 •

edited

Loading

Uh oh!

clarmso Jan 20, 2025

Uh oh!

clarmso Jan 20, 2025

Uh oh!

clarmso Jan 20, 2025 •

edited

Loading

Uh oh!

isabelrios Jan 21, 2025 •

edited

Loading

Uh oh!

isabelrios Jan 21, 2025

Uh oh!

clarmso Jan 22, 2025

Uh oh!

clarmso Jan 20, 2025

Uh oh!

clarmso Jan 22, 2025

Uh oh!

clarmso Jan 22, 2025

Uh oh!

isabelrios Jan 22, 2025 •

edited

Loading

Uh oh!

clarmso commented Jan 22, 2025

Uh oh!

clarmso commented Jan 22, 2025 •

edited

Loading

Uh oh!

mobiletest-ci-bot commented Jan 22, 2025

Uh oh!

Uh oh!

Bugfix MTE-4133 Disable retries in fastlane #24251

Bugfix MTE-4133 Disable retries in fastlane #24251

Uh oh!

Conversation

clarmso commented Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Tickets

💡 Description

📝 Checklist

Uh oh!

clarmso Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isabelrios Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isabelrios Jan 21, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

clarmso Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

isabelrios Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clarmso commented Jan 22, 2025

Uh oh!

clarmso commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mobiletest-ci-bot commented Jan 22, 2025

Client.app: Coverage: 32.37

WidgetKitExtension.appex: Coverage: 7.05

libStorage.a: Coverage: 55.32

Uh oh!

Uh oh!

clarmso commented Jan 20, 2025 •

edited

Loading

clarmso Jan 20, 2025 •

edited

Loading

isabelrios Jan 21, 2025 •

edited

Loading

isabelrios Jan 22, 2025 •

edited

Loading

clarmso commented Jan 22, 2025 •

edited

Loading