-
Notifications
You must be signed in to change notification settings - Fork 20
ADR System test framework
The objective of this document is to conduct a thorough evaluation of various system test frameworks while taking into account our specific use cases. We aim to gain insight into the advantages and disadvantages of the frameworks under consideration, thereby facilitating a fair comparison.
- Sourabh Mehta
- Daniel Brondani
- Jonatan Antony
- Samuel Pelegrinello Caipers
- Evgueni Driouk
- Thorsten de Buhr
The end-to-end (E2E) tests developed using a decided framework aim to validate the fully integrated applications, assessing the interactions among different utilities within the entire system. The overarching goal is to simulate user interactions and ensure that the desired build output is achieved through comprehensive system testing.
- Access to system env
- Flexibility to trigger EXEs, capture return code, stdout, stderr
- Detailed logs & reporting
- Generates HTML report
| Sr. No | key points | GoogleTest | PyTest | RobotTest |
|---|---|---|---|---|
| 1 | Language Support | C++ | python | keyword/macro based (backed by python) |
| 2 | Debug | yes | yes | debug with prints or logs |
| 3 | Dependencies | external dep (need to maintain as a submodule in repo) | external dep need to be installed | external dep need to be installed |
| 4 | HTML Report | Generates XML, 3rd party lib is needed to generate HTML | HTML reports with pytest-html plugin | Inbuilt HTML report generation |
| 5 | Ease of use | yes | yes | yes (Needs learning) |
| 6 | Flexibility and Customization | yes | yes | yes |
| 7 | Integration with CI/CD | yes | yes | yes |
| 8 | Feedback Mechanisms & diagnosis | easy to diagnose | easy to diagnose | use debug logs for diagnosis |
-
cbuildgen has toolchain-related tests (AC6, GCC, IAR, CLang). Are we running all toolchain tests on cmsis-toolbox?
The answer is Yes. Also, all the specific test examples needed to be converted into csolution projects. -
cbuildgen has test related to cpackget. Should these tests also be migrated?
No, Due to historical reasons, these tests were added in cbuildgen. The tool-specific test should already be implemented close to cpackget code
It was determined that the robot test would be the optimal choice. This option requires no language-specific learning and offers a more user-friendly reporting mechanism. Additionally, the robot library boasts extensive support and numerous prebuilt keywords.