Skip to content

Retry Bitrise Pipeline

Jump to bottom
Clare So edited this page Apr 17, 2024 · 9 revisions

Summary

The purpose of this document is to show anyone who works within Bitrise to do the following common tasks:

  • identify issues reported,
  • (re)trigger builds
  • unblock common issues. Bitrise grants us the ability to identify common causes of errors and permits reruns targeting failing workflows.

What does a workflow failure look like?

This section illustrates the common failures that could occur at different stages in the pipeline and how we could investigate those failures. Every PR must pass Bitrise before merging.

build_applications

This stage performs the following:

  • Builds the app (Firefox or Focus)
  • Identifies lint issues
  • Finds warnings
  • Runs unit tests Here is an example pipeline that fails at build_applications and no workflows from run_tests are run. (picture here)

Investigating the Failure

This stage could fail due to one of the steps or Bitrise related issues.

You can determine the cause of the failure by opening the logs:

  1. Click the workflow’s name (firefox_configure_build or focus_configure_build).
  2. Open the Tests tab for any unit test failures.
  3. Click Build Log for information for build failures, Swiftlint failures and any Bitrise errors. The same steps could be applied for focus_configure_build.

Common workflow remediation

Build Errors

Ensure the supported version of Xcode is installed and invoked. See README of the repository for the currently supported version.

$ xcodebuild -version
Build the app using the command line.
$ cd firefox-ios/
$ xcodebuild build-for-testing -project Client.xcodeproj -scheme Fennec -configuration Fennec -sdk iphonesimulator -destination 'platform=iOS Simulator,name=<device>,OS=<iOS version>'

Swiftlint/Warnings

  1. Run swiftlint on the command line.

Unit Tests

  1. Run the unit tests in question using the command line. 
    $ cd firefox-ios/
$ xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -only-testing:<test>
  2. If the unit tests in question passes, run the same tests repeatedly. 
    $ xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -test-iterations 100 -only-testing:<test>

If the unrelated unit tests fail intermittently, please retrigger the pipeline (see Retrigger the complete pipeline).

For any issues that are not covered above, please message @mobiletest on Slack.

run_tests workflow

run_tests consists of the XCUITest in the Smoke Tests suite running in parallel. The smoke tests consist of the key functionalities of the app.

How to remediate the failures?

  1. Click the failed workflow’s name under run_tests from the pipeline’s page.
  2. Identify the test’s name.
  3. Run the particular test using command line command locally. 
    xcodebuild test -target Client -scheme Fennec -destination 'platform=iOS Simulator,name=<device>,OS=<version>' -only-testing:XCUITests/<file>/<test> -test-plan Smoketest<number>
    For this example, the file would be CreditCardsTests and the test would be testAutofillCreditCardsToggleOnOff.
  4. For more information on the test failure reported by Bitrise, you may check the Build log. Scroll down to read the relevant section of the log.
  5. If the smoke test passes locally on your computer, please rebuild the unsuccessful workflow instead of rebuilding the whole pipeline.

In the event that the issue persists, please ping @mobiletest on Slack if you are unsure.

Rerun the pipeline and workflows

We can re-run the failing pipeline or workflow without committing a change to the branch.

Rebuild only unsuccessful workflows from Bitrise

If one of the workflows for smoke tests fails, you may run just the failed workflow. (See Test Retries for an explanation on how we handle intermittently failing tests.)

  1. Open the pipeline’s page on Bitrise.
  2. Open the Rebuild dropdown menu and select Rebuild unsuccessful Workflows.
  3. Open Bitrise Firefox iOS page. Locate the Manual Approval Builds section.
  4. Click Approve and run Build button for your PR.

⛔Important: Please do not re-run any part of the builds via Github. These links from Github do not invoke Bitrise properly.

Retrigger the complete pipeline

If the pipeline fails on build_applications or earlier, you may retrigger the whole pipeline. An example use would be when a unit test fails intermittently.

  1. Open the pipeline’s page on Bitrise.
  2. Open the Rebuild dropdown menu and select Rebuild entire Pipeline.
  3. Go back to Bitrise Firefox iOS page. Locate the _Manual Approval Builds section. Click Approve and run Build button for your PR.

⛔Important: Please do not re-run any part of the builds via Github. These links do not invoke Bitrise properly.

Tips and Tricks

Ensure bitrise.yml is Valid

If the Bitrise pipeline fails before the stages are run, there may be a syntax error in the bitrise.yml file. We can do a sanity check on this file by using Bitrise CLI.

  1. Install bitrise using brew: brew update && brew install bitrise.
  2. Run bitrise validate bitrise.yml.

Test Retries

It’s possible for the failing smoke tests to pass on retry because the smoke tests have a limit of 3 retries. In other words, if a smoke test fails initially, the test runner gives the smoke test another chance to pass. If the smoke test fails even after the retries, please debug the failing test locally.

Note that only Bitrise is needed to be green ✅ before the PR lands. Bitrise reports to Github as red ❌ if a smoke test did not pass initially. In the following example report from Github, this PR is ready to land.

Enable Manual Approval Builds Button

If the Manual approval builds button does not appear or does not work for some reason, you may close and reopen the PR or fetch the changes from main to the branch.

Close and Reopen PR

  1. Go to the PR’s page on Github.
  2. Write a brief comment and click Close pull request.
  3. Write a brief comment and click Reopen pull request.
  4. Go to Bitrise Firefox iOS page on Bitrise. Scroll down to the list of running and recent pipelines to ensure that the pipeline is running.

Fetch changes from main

  1. Open a terminal.
  2. cd to the Firefox iOS repository.
  3. Merge the latest changes from main to your branch.
  4. Go to the Bitrise Firefox iOS page on Bitrise.
  5. Scroll down to the list of running and recent pipelines to ensure that the pipeline is running.
Clone this wiki locally