|
| 1 | +# Automated UI Tests for a Teams application |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +In this section we will provide an overview on how one could implement UI tests for a custom Teams application. |
| 6 | +Generally, to ensure a great experience for your end users, you would like to implement automated test which covers your features both in a browser as well as on mobile platforms. |
| 7 | + |
| 8 | +General observations: |
| 9 | +- Testing in a web browser is easier than on a native app. |
| 10 | +- Testing a Teams app on a mobile device in an automated way is more challenging due to the fact that you are testing an app within an app: |
| 11 | + - There is no Android Application Package (APK) / iOS App Store Package (IPA) publicly available for Microsoft Teams app itself. |
| 12 | + - Mobile testing frameworks are designed with the assumption that you own the APK/IPA of the app under test. |
| 13 | + - Workarounds need to be found to first automate the installation of Teams. |
| 14 | +- Should you choose working with emulators, testing in a local Windows box is more stable than in a CI/CD. The latter involves a CI/CD agent and an emulator in a VM. |
| 15 | + |
| 16 | +The following are learnings from various engagements |
| 17 | + |
| 18 | +## Web based UI tests |
| 19 | + |
| 20 | +To implement web-based UI tests for your Teams application, you can follow the same approach as for testing any other web application with a UI. [UI testing](README.md) provides good guidance on that. The starting point for your test would be to launch a browser in an automated way (using Selenium, or similar frameworks) and go to [https://teams.microsoft.com/](https://teams.microsoft.com/). |
| 21 | + |
| 22 | +If you would like to test a Teams app which was not yet published into the Teams' store, or if you would like to test the DEV/QA version of your app: |
| 23 | +- Using for example [Teams Toolkit](https://github.com/OfficeDev/TeamsFx), package your app based on the [manifest.json](https://learn.microsoft.com/en-us/microsoftteams/platform/resources/schema/manifest-schema). See below an example on how to achieve that: |
| 24 | + |
| 25 | +```javascript |
| 26 | +npx teamsfx package --env dev --manifest-path ... |
| 27 | +``` |
| 28 | + |
| 29 | +Once the app is installed, implement [selectors](https://www.browserstack.com/guide/css-selectors-in-selenium) to [access your custom app](https://learn.microsoft.com/en-us/microsoftteams/platform/concepts/deploy-and-publish/apps-upload#access-your-app) and to perform various actions within the app. |
| 30 | + |
| 31 | +### Pipeline |
| 32 | + |
| 33 | +If you are using Selenium and Edge as the browser, consider leveraging the [selenium/standalone-edge](https://hub.docker.com/r/selenium/standalone-edge) Docker image which contains a standalone Selenium server with the Microsoft Edge browser installed. By default, it would run in headless mode, but by setting `START_XVFB` variable to `True`, you can control whether to start a virtual framebuffer server (Xvfb) that allows GUI applications to run without a display. Below is a code snippet which illustrated the usage of the image in a Gitlab pipeline: |
| 34 | + |
| 35 | +```yml |
| 36 | +... |
| 37 | +run-tests-dev: |
| 38 | + allow_failure: false |
| 39 | + image: ... |
| 40 | + environment: |
| 41 | + name: dev |
| 42 | + stage: tests |
| 43 | + services: |
| 44 | + - name: selenium/standalone-edge:latest |
| 45 | + alias: selenium |
| 46 | + variables: |
| 47 | + START_XVFB: "true" |
| 48 | + description: "Start Xvfb server" |
| 49 | +... |
| 50 | +``` |
| 51 | + |
| 52 | +When running test, you need to use the Selenium server URL for remote execution. With the definition from above, the URL is: http://selenium:4444/wd/hub. The code snipped below illustrates how one could initialize the Selenium driver to point to the remote Selenium server using JavaScript: |
| 53 | + |
| 54 | +```javascript |
| 55 | +var { Builder } = require("selenium-webdriver"); |
| 56 | +const edge = require("selenium-webdriver/edge"); |
| 57 | + |
| 58 | +var buildEdgeDriver = function () { |
| 59 | + let builder = new Builder().forBrowser("MicrosoftEdge"); |
| 60 | + builder = builder.usingServer("http://selenium:4444/wd/hub"); |
| 61 | + builder.setEdgeOptions(new edge.Options().addArguments("--inprivate")); |
| 62 | + return builder.build(); |
| 63 | +}; |
| 64 | +``` |
| 65 | + |
| 66 | +## Mobile based UI tests |
| 67 | +Testing your custom Teams application on mobile devices is a bit more difficult using the web-based approach as it requires usage of actual or simulated devices. Running such tests in a CI/CD pipeline can be more difficult and resource-intensive. |
| 68 | + |
| 69 | +One approach is to use real devices or cloud-based emulators from vendors such as [BrowserStack](https://www.browserstack.com/) which requires a license. Alternatively, you can use virtual devices hosted in Azure Virtual Machines. |
| 70 | + |
| 71 | + |
| 72 | +### Using Android Virtual Devices |
| 73 | + |
| 74 | +This approach enables the creation of Android UI tests using virtual devices. It comes with the advantage of not requiring paid licenses to certain vendors. However, due to the nature of emulators, compared to real devices, it may prove to be less stable. Always choose the solution that best fits your project requirements and resources. |
| 75 | + |
| 76 | +Overall setup: |
| 77 | +- [AVD - Android Virtual Devices](https://developer.android.com/studio/run/managing-avds) - which are virtual representation of physical Android devices. |
| 78 | +- [Appium](https://appium.io/) is an open-source project designed to facilitate UI automation of many app platforms, including mobile. |
| 79 | + - Appium is based on the [W3C WebDriver specification](https://w3c.github.io/webdriver/). |
| 80 | + > Note: If you look at these commands in the WebDriver specification, you will notice that they are not defined in terms of any particular programming language. They are not Java commands, or JavaScript commands, or Python commands. Instead, they form part of an HTTP API which can be accessed from within any programming language. |
| 81 | + - Appium implements a client-server architecture: |
| 82 | + - The server (consisting of Appium itself along with any drivers or plugins you are using for automation) is connected to the devices under test, and is actually responsible for making automation happen on those devices. [UiAutomator](https://github.com/appium/appium-uiautomator2-driver) driver is compatible with Android platform. |
| 83 | + - The [client](https://appium.io/docs/en/2.0/intro/clients/) is responsible for sending commands to the server over the network, and receiving responses from the server as a result. You can choose the language of your choice to write the commands. For example, for Javascript [WebDriverIO](https://webdriver.io/) can be used as client. |
| 84 | + > [Here](https://appium.io/docs/en/2.0/ecosystem/) you can read more about Appium ecosystem |
| 85 | + - The advantage of this architecture is that it opens the possibility of running the server in a VM, and the client in a pipeline, enabling the tests to be ran automatically on scheduled basis as part of CI/CD pipelines. |
| 86 | + |
| 87 | +### Running mobile test locally on a Windows machine |
| 88 | + |
| 89 | +This requires: |
| 90 | + |
| 91 | +- An emulator ([AVD - Android Virtual Devices](https://developer.android.com/studio/run/managing-avds)), which will represent the physical device. |
| 92 | +- [Appium server](https://appium.io/docs/en/2.1/), which will redirect the commands from the test to your virtual device. |
| 93 | + |
| 94 | +#### Creating an Android Virtual Device: |
| 95 | + |
| 96 | +1. Install Android Studio from [official link](https://developer.android.com/studio). |
| 97 | + > Note: At the time of writing the documentation, the latest version available was Android Studio Giraffe, 2022.3.1 Patch 2 for Window. |
| 98 | + 1. Set ANDROID_HOME environment variable to point to the installation path of Android SDK. i.e. ` C:Users\<user-name>\AppData\Local\Android\Sdk` |
| 99 | +1. Install Java Development Kit (JDK) from [official link](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html). For the most recent devices JDK 9 is required, otherwise JDK 8 is required. Make sure you get the JDK and not the JRE. |
| 100 | + 1. Set JAVA_HOME environment variable to the installation path, i.e. ` C:\Program Files\Java\jdk-11` |
| 101 | +1. Create an AVD (Android Virtual Device): |
| 102 | + 1. Open Android Studio. From the Android Studio welcome screen, select **More Action -> Virtual Device Manager**, as instructed [here](https://developer.android.com/studio/run/managing-avds#:~:text=1%20Open%20the%20AVD%20Manager%20by%20clicking%20Tools,%20settings%2C%20such%20as%20the%20skin.%20See%20More.) |
| 103 | + 1. Click **Create Device**. |
| 104 | + 1. Choose a device definition with **Play Store** enabled. This is important, otherwise Teams cannot be installed on the device. |
| 105 | + 1. Choose a System image from the **Recommended** tab which includes access to Google Play services. You may need to install it before selecting it. |
| 106 | + 1. Start the emulator by clicking on the Run button from the Device Manage screen. |
| 107 | + 1. Manually install Microsoft Teams from Google Playstore on the device. |
| 108 | + |
| 109 | +#### Setting up Appium |
| 110 | +- Install `appium`: |
| 111 | + 1. Download NodeJs, if it is not already installed on your machine: [Download | Node.js (nodejs.org)](https://nodejs.org/en/download) |
| 112 | + 1. Install Appium globally: [Install Appium - Appium Documentation](https://appium.io/docs/en/2.0/quickstart/install/) |
| 113 | + 1. Install the UiAutomator2 driver: [Install the UiAutomator2 Driver - Appium Documentation](https://appium.io/docs/en/2.0/quickstart/uiauto2-driver/). Go through the `Set up Android automation requirements` in the documentation, to make sure you have set up everything correctly. |
| 114 | + |
| 115 | + > [Here](https://appium.io/docs/en/2.0/intro/drivers/) you can read more about Appium Drivers. |
| 116 | + 1. Start appium server by running `appium` command in a command prompt. |
| 117 | +#### Useful commands: |
| 118 | + |
| 119 | +List emulators that you have previously created, without opening Android Studio: |
| 120 | + |
| 121 | +```cli |
| 122 | +emulator -list-avds |
| 123 | +``` |
| 124 | + |
| 125 | +### Setting up an Azure VM for running mobile tests in a pipeline |
| 126 | + |
| 127 | +### Configure the VM |
| 128 | + |
| 129 | +This approach essentially hosts a virtual device in a virtual machine. In order to be able to set up the emulator (Android Virtual Device) in an Azure VM, the VM should support [nested virtualization](https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/). |
| 130 | + |
| 131 | +Azure VM configuration which, at the time of writing the documentation, worked successfully with AVD and appium: |
| 132 | +- Operating system: Windows (Windows-10 Pro) |
| 133 | +- VM generation: V1 |
| 134 | +- Size: Standard D4ds v5 16 GiB memory |
| 135 | + |
| 136 | +### Enable connection from outside to Appium server on the VM |
| 137 | + |
| 138 | +> Note: By default appium server runs on port 4723. The rest of the steps will assume that this is the port where your appium server runs. |
| 139 | +
|
| 140 | +In order to be able to reach appium server which runs on the VM from outside: |
| 141 | + |
| 142 | +1. Create an [Inbound Rule](https://learn.microsoft.com/en-us/windows/security/operating-system-security/network-security/windows-firewall/create-an-inbound-port-rule) for port 4723 from within the VM. |
| 143 | +1. Create an Inbound Security Rule in the NSG (Network Security Group) of the VM to be able to connect from that IP address to port 4723. |
| 144 | + - Find out the IP of the machine on which the tests will run on. |
| 145 | + - Replace the *Source IP Address* with the IP of your machine. |
| 146 | + |
| 147 | +### Installing Android Studio and create AVD inside the VM |
| 148 | + |
| 149 | +1. Follow the instructions from steps a) and b) in **How to run mobile end to end tests on a Windows machine** section above. |
| 150 | +1. When you launch the emulator, it may show a warning as below and will eventually crash: |
| 151 | + |
| 152 | +  |
| 153 | + |
| 154 | + Solution to fix it: |
| 155 | + 1. [Enable Windows Hypervisor Platform](https://devblogs.microsoft.com/visualstudio/hyper-v-android-emulator-support/) |
| 156 | + 1. [Enable Hyper-V](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v#enable-the-hyper-v-role-through-settings) (if not enabled by default) |
| 157 | +1. Restart the VM. |
| 158 | +1. Restart the AVD. |
| 159 | + |
| 160 | + |
| 161 | +### Inspecting the Teams app in Azure Virtual Device (AVD) |
| 162 | + |
| 163 | +Inspecting the app is very useful as you write new tests, as it allows you to find the ID of different elements displayed on the screen - similar to DevTools, which allows you to navigate through the DOM of a web page. |
| 164 | + |
| 165 | +[Appium Inspector](https://inspector.appiumpro.com/) is a very useful tool that allows you to inspect an app runing on an emulator. |
| 166 | + |
| 167 | +> Note: This section assumes that you have already performed the prerequisites from **How to run mobile end to end tests on a Windows machine** |
| 168 | +
|
| 169 | +### Inspect the Teams app: |
| 170 | + |
| 171 | +1. Run the appium server with [--alow-cors flag](https://appium.readthedocs.io/en/latest/en/writing-running-appium/server-args/) by running the following command in a terminal: |
| 172 | + ``` |
| 173 | + appium --allow-cors |
| 174 | + ``` |
| 175 | +2. Go to https://inspector.appiumpro.com and type in the following properties: |
| 176 | + ```json |
| 177 | + { |
| 178 | + "appium:deviceName": "your-emulator-name", |
| 179 | + "appium:appPackage": "com.microsoft.teams", |
| 180 | + "appium:appActivity": "com.microsoft.skype.teams.Launcher", |
| 181 | + "appium:automationName": "UiAutomator2", |
| 182 | + "platformName": "Android" |
| 183 | + } |
| 184 | + ``` |
| 185 | +
|
| 186 | + 1. "appium:deviceName" - is the name of your emulator. In **Useful commands** sections from above, you can see how to get the name of your AVD. |
| 187 | + 1. "appium:appPackage" - is the name of the package, should be kept to "**com.microsoft.teams**". |
| 188 | + 1. "appium:appActivity"- is the name of the activity in the app that you want to launch, should be kept to "**com.microsoft.skype.teams.Launcher**" |
| 189 | + 1. "appium:automationName" - is the name of the driver you are using, in this case, "**UiAutomator2**" |
| 190 | +
|
| 191 | + If the appium server runs on your local machine at the default portal, then Remote Host and Remote Port can be kept to the default values. |
| 192 | +
|
| 193 | + The configuration should look similar to the printscren below: |
| 194 | +  |
| 195 | +3. Press on **Start Session**. |
| 196 | + - In the browser, you should see a similar view as below: |
| 197 | +  |
| 198 | + - You can do any action on the emulator, and if you press on the "Refresh" button in the browser, the left hand side of the Appium Inspector will reflect your app. In the **App Source** you will be able to see the IDs of the elements, so you can write relevant selectors in your tests. |
| 199 | +
|
| 200 | +#### Connecting to Appium server |
| 201 | +
|
| 202 | +Below is how this can be accomplished with JavaScript. A similar approach can be followed for other languages. |
| 203 | +Assuming you are using [webdriverio](https://webdriver.io/) as the client, you would need to initialize the remote connection as follows: |
| 204 | +
|
| 205 | +```javascript |
| 206 | + const opts = { |
| 207 | + port: 4723, |
| 208 | + hostname: "your-hostname", |
| 209 | + capabilities: { |
| 210 | + platformName: "android", |
| 211 | + "appium:deviceName": "the-name-of-the-virtual-device", |
| 212 | + "appium:appPackage": "com.microsoft.teams", |
| 213 | + "appium:appActivity": "com.microsoft.skype.teams.Launcher", |
| 214 | + "appium:automationName": "the-name-of-the-driver", |
| 215 | + }, |
| 216 | + }; |
| 217 | +
|
| 218 | + // Create a new WebDriverIO instance with the Appium server URL and capabilities |
| 219 | + await wdio.remote(opts); |
| 220 | +``` |
| 221 | + |
| 222 | +1. "port": the port on which the Appium server runs on. By default, it is 4723. |
| 223 | +1. "hostname": the IP of the machine where the Appium sever runs on. If it is running locally, that is 127.0.0.1. If it runs in an Azure VM, it would be the public IP address of the VM. Note: ensure you have followed the steps from **2. Enable connection from outside to Appium server on the VM**. |
| 224 | +1. "platformName": Appium can be used to connect to different platforms (Windows, iOS, Android). In our case, it would be "android". |
| 225 | +1. "appium:deviceName": the name of the Android Virtual Device. See **Useful commands** on how to find the name of the device. |
| 226 | +1. "appium:appPackage": the name of the app's package that you would like to launch. Teams' package name is "com.microsoft.teams". |
| 227 | +1. "appium:appActivity": the activity within Teams that you would like to launch on the device. In our case, we would like just to launch the app. The activity name for launching Teams is called "com.microsoft.skype.teams.Launcher". |
| 228 | +1. "appium:automationName": the name of the driver you are using. Note: Appium can communicate to different platforms. This is achieved by installing a dedicated driver, designed for each platform. In our case, it would be [UiAutomator2](https://github.com/appium/appium-uiautomator2-driver) or [Espresso](https://github.com/appium/appium-espresso-driver), since they are both designed for Android platform. |
| 229 | + |
| 230 | +### Using BrowserStack |
| 231 | + |
| 232 | +- BrowserStack does not support out of the box the installation of Teams from the App Store or Play Store. However, there is a workaround, described in [their documentation](https://www.browserstack.com/support/faq/app-automate/app/can-i-install-an-app-from-the-app-store-or-play-store). Therefore, if you choose to go this way, you would first need to implement a step that installs Teams on the cloud-based device, by implementing the workaround described above. |
| 233 | +- You may encounter issues with Google login, as it requires a newly created Google account, in order to log in to the store. To overcome this, make sure to disable 2FA from Google, further described in [Troubleshooting Google login issues](https://www.browserstack.com/docs/app-automate/appium/advanced-features/setup-google-account#nodejs). |
| 234 | + |
| 235 | + |
| 236 | + |
| 237 | + |
0 commit comments