Azure DevOps Test Execution

Azure DevOps Test Execution

Before starting

1. Ensure all tests are working locally prior to setting them up to run from Azure DevOps.

2. Consider how you will store credentials required to connect to the environment the tests will be performed on. Possible options:

   • Update your EasyRepro project to dynamically retrieve credentials from an external source like Azure Key Vault
   • Store credentials as pipeline variables and use PowerShell or another mechanism to update the .config file prior to test execution

Avoid hard coding credentials so they are not checked into source control

Scenario #1: Executing tests inside an Azure DevOps build

This will compile and run tests inside of an Azure DevOps build pipeline using a Microsoft hosted build agent.

Build Setup

1. Set Pipeline: Agent Pool to 'Hosted VS2017'
2. Set Get sources to be the repository containing the EasyRepro solution which contains the tests to be run
3. Add a NuGet task (2.*)
   1. Execute the restore command
   2. Set Path to solution, packages.config, or project.json to the location of the .sln, packages.config, or project.json file
4. Add a Visual Studio Build task (1.*)
   1. Set Solution to the path of the solution or project to be built
   2. Set the Visual Studio Version to 'Visual Studio 2017'
   3. Set the Configuration to 'release'
5. Add a Visual Studio Test Platform Installer (1.*) task
6. Add a Visual Studio Test (2.*) task
   1. Set Select tests using to 'Test assemblies'
   2. Update Test files to include a pattern that will find your project's compiled assembly
   3. Ensure the Search folder path corresponds to the value in Test files
   4. Check Test mix contains UI tests
   5. Set Select test platform using to 'Version'
   6. Set Test platform version to 'Installed by Tools Installer'

This will run all tests found in the Search Folder matching the pattern set under Test files during a build.

Scenario #2: Executing tests inside an Azure DevOps release

This will compile and run tests inside of an Azure DevOps release pipeline using a Microsoft hosted build agent.

Build Setup

1. Add steps 1-4 from Scenario #1
2. Add a Publish Build Artifacts task
   1. Set Path to publish to the 'bin/release' folder of the project
   2. Set Artifact publish location to 'Azure Pipelines/TFS'

Release Setup

1. Add an Artifact that references the Project and Source (build pipeline) created above
   1. Set Default version to 'Latest'
2. Add a Stage to the pipeline if one does not exist
3. Add steps 5-6 from Scenario #1

This will run all tests found in the Search Folder matching the pattern set under Test files during a release pipeline.

Scenario #3: Executing tests from the Azure DevOps Test Plans

This will allow users to run individual or groups of tests ondemand from existing Test Plans, Test Suites, and/or, Test Cases found in the Azure DevOps.

Project Set

1. Ensure Visual Studio is connected to your Azure DevOps instance
2. Open Test Explorer and right-click on a test
3. Choose Associate to test case
4. Enter the Test Case id from Azure DevOps
5. Repeat for all tests

Additional documentation: https://docs.microsoft.com/en-us/azure/devops/test/associate-automated-test-with-test-case?view=azure-devops

Build Setup

The same setup can be used from Scenario #1: Executing tests inside an Azure DevOps build

Release Setup

The same setup can be used from Scenario #2: Executing tests inside an Azure DevOps release with 1 exception.

1. Add a Visual Studio Test (2.*) task
   1. Set Select tests using to 'Test run'
   2. Update Test files to include a pattern that will find your project's compiled assembly
   3. Leave Test Run as the default value of '$(test.RunId)'
   4. Check Test mix contains UI tests
   5. Set Select test platform using to 'Version'
   6. Set Test platform version to 'Installed by Tools Installer'

Running from Test Plans

1. Select a Test Plan, Test Suite, or 1 more more Test Cases
2. Select Run
3. Select Run for web application
4. This will open a window validating the related tests, if everything passes the tests will start running in the background
5. Check back after the tests complete to see the results

Additional documentation https://docs.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops

Troubleshooting

The browser being tested against needs to be installed on the build agent. By default Microsoft hosted build agents do have a number of browsers supported by EasyRepro installed by default. If there are issues running tests, verify the Capabilities of the agent to ensure the version of the browser installed is compatible with the Selenium Web Driver version supported by EasyRepro.

Additional Troubleshooting