feat(amazonq): Adding e2e tests #8494
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Changes This is the initial commit for UI Tests using the framework vscode-extension-tester. It instantiates the VSCode instance which opens Amazon Q and goes through some basic flows to login. This is done using the [VSCode-Extension-Tester Framework](https://github.com/redhat-developer/vscode-extension-tester/wiki/). (Note: It is missing full authentication and stops working once Amazon Q needs to open the browser). **To run this test** 1. First compile the test files `npm run testCompile` 2. Then run the test setup `node_modules/.bin/extest setup-tests` 2. Then run the script (NOTE: this js file should be generated when you compile the test files) `node_modules/.bin/extest run-tests aws-toolkit-vscode/packages/amazonq/dist/test/e2e/amazonq/VET.test.js` For information why each dependency / webpack change is needed, please see [here](https://quip-amazon.com/lCuBAOGibHzm/UI-Tests-PR-Explanation). Demo Video: https://github.com/user-attachments/assets/f1c6e59f-d4e3-4ae0-a164-1da389ec1339 --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
Merge master into feature/ui-e2e-tests
…at util (#7812) ## Problem We have yet to implement the multiple chat test and the view history chat test. The clearChat abstraction can also be confusing to some individuals since the /clear command is different. ## Solution I have made the clearChat abstraction clearer in name so people do not get confused with the /clear command. Additionally, I have implemented the multipleChatTest and the viewHistory chat test. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
…st suite (#7881) ## Changes I researched how we can make a very robust button clicking logic using Selenium best practices. I added this new general function into generalUtils.ts and replaced the MCP test suite with this logic so it no longer flakes. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
## Change I implemented the Rules Option Test and all the abstractions needed to write that test. Since there is only 1 test in the test suite, this will be the complete test suite unless the Rules feature gets expanded or the team decides to write more tests for it. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: tgodara-aws <tgodara@amazon.com> Co-authored-by: andrewyuq <89420755+andrewyuq@users.noreply.github.com> Co-authored-by: Jayakrishna P <parameja@amazon.com> Co-authored-by: Laxman Reddy <141967714+laileni-aws@users.noreply.github.com> Co-authored-by: Ralph Flora <floralph@amazon.com> Co-authored-by: Suraj Reddy <surajrdy@amazon.com> Co-authored-by: Roger Zhang <ruojiazh@amazon.com> Co-authored-by: Reed Hamilton <49345456+rhamilt@users.noreply.github.com> Co-authored-by: Roger Zhang <roger.zhang.cs@gmail.com> Co-authored-by: Jacob Chung <chungjac@amazon.com> Co-authored-by: aws-asolidu <asolidu@amazon.com> Co-authored-by: Newton Der <newton.der@gmail.com> Co-authored-by: Newton Der <dernewtz@amazon.com> Co-authored-by: Tyrone Smith <tsmithsz@amazon.com> Co-authored-by: Aidan Ton <atona@amazon.com> Co-authored-by: abhraina-aws <abhraina@amazon.com> Co-authored-by: atontb <104926752+atonaamz@users.noreply.github.com> Co-authored-by: Nitish <149117626+singhAws@users.noreply.github.com> Co-authored-by: Nitish Kumar Singh <nkrsingh@amazon.com> Co-authored-by: Blake Lazarine <blakelaz@amazon.com> Co-authored-by: mkovelam <mkovelam@amazon.com> Co-authored-by: BlakeLazarine <blake.l.lazarine@gmail.com> Co-authored-by: Will Lo <96078566+Will-ShaoHua@users.noreply.github.com> Co-authored-by: Na Yue <yueny@amazon.com> Co-authored-by: Lei Gao <97199248+leigaol@users.noreply.github.com> Co-authored-by: tsmithsz <84354541+tsmithsz@users.noreply.github.com> Co-authored-by: Reed Hamilton <reedham@amazon.com> Co-authored-by: Boyu <wangby56@gmail.com> Co-authored-by: Dung Dong <dungdong@amazon.com> Co-authored-by: aws-toolkit-automation <43144436+aws-toolkit-automation@users.noreply.github.com> Co-authored-by: Bryce Ito <bryceito@amazon.com>
…2E Tests and Inline Keybind Shortcut Test (#7840) ## Problem Currently we use a `sleep(8000)` function which causes us to wait an arbitrary 8 seconds for Amazon Q to implement the fibonacci sequence. This can cause flakiness and is generally not a good practice. We also have not fully updated our inline test and our writeToTextEditor function does not work well. We also have not implemented an inline test which uses keybinds to start an inline chat suggestion. ## Solution I have implemented a function which constantly checks for a "stable" state in which we check for any new number of lines generated by Amazon Q. If this stable state is repeated more than 2 times, meaning the number of lines has stayed the same for 3 seconds (1 second wait in between), we will confirm that AmazonQ has finished generating its response. We have a maximum generate time of 15 seconds, as a fibonacci sequence should not take more than that, or something is clearly wrong with the model and we will error out. We have implemented the inline tests and the writeToTextEditor function fix. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: laura-codess <121297288+laura-codess@users.noreply.github.com>
## Problem We have not implemented the shortcut tests for our UI E2E Test suite with VSCode-Extension-Tester. Additionally, we face a small problem in our pressShortcut abstraction that throws a type error when trying to reference a Key command like `Key.COMMAND` since it is a IKey String. ## Solution We have implemented the shortcut tests: - Optimize Code Shortcut - Generate Test Shortcut - Keybind Check - Explain Code Shortcut We have also fixed the pressShortcut abstraction with a small edit in the logic of accepted parameters. NOTE: We have chosen to implement the Inline Shortcut within the Inline Tests even though it could be classified under Shorcuts. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
… utils and Helpers (#7882) ## Problem Our tests are currently experiencing flakiness due to our usage of try-catch blocks which allow errors to pass through our tests. One of our more egregious examples of this is our `quickActions.test.ts` file which is unable to find the proper CSS element of our quickAction command overlay, but still allows our test to pass as this error is non-critical due to our try-catch block. This leaves an inherent problem in our tests making them flaky while still containing UI errors. Additionally, many of our helper functions return Booleans, and this logic of booleans is used in some instances like our `chat.test.ts`. ## Solution We will remove the majority of try-catch blocks that are used in our helpers to have more stricter errors in our tests. Our tests should fail if an error occurs while finding a UI element. Additionally, we update our `chat.test.ts` so that it can properly stop using the Boolean returned from the chat helper function. We have updated: - generalUtils - cleanupUtils - chat.test.ts - quickActionsHelper - switchModelHelper --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
…7829) ## Problem We have yet to implement the initial P0 Pin Context Tests. The pinContextHelper does not mesh well with sub menu items when clicking through different prompts to use. ## Solution I have implemented the File Context and Prompt Context Test. The Pin Context test with the @workspace was previously implemented. I have implemented the helpers: getSubMenuItems and clickSubMenuItem in the pinContextHelper. All tests still run together. Test Suite: - View Context Options - Reusable Prompt (New) - Add File as context (New) - Add Workspace as Context --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
## Change The test resources folder was accidentally committed to our feature branch, adding around 300k+ unnecessary lines, this commit removes them. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
…ext ability to run in full test suite (#7928) ## Problem We were faced with the problem that when all our tests and helpers were merged together, and then run as a full suite, we saw errors with the pinContext.test.ts file. Namely, it was targeting a background overlay that had identical css components to the pinContext overlay. This is problematic since our entire suite relies on these css components, and even test-ids were not differentiated. Additionally, our pinContextHelpers used sleep functions in the range of 100-3000 ms which is naturally arbitrary values. ## Solution We implement the XPath css selectors that are used within Selenium to identify the pinContext Buttons by text, not css classes or ids. This drastically decreases the length and complexity of our helpers while also eliminating the need for sleep functions other than a sleep(0) function for subMenuItems that is needed for the DOM to be loaded. Below is our success of the tests: <img width="313" height="77" alt="Screenshot 2025-08-19 at 1 49 34 PM" src="https://github.com/user-attachments/assets/dd378c02-67c1-4f73-a6e7-cd709e028c80" /> --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
## Problem Currently our test suite outputs into the terminal, but this is not stored anywhere. ## Solution For a more robust suite, we will begin logging our test runs by the exact date stored in the logs folder within our e2e_new folder. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
## Problem There were some CSS changes to the quick actions menu which caused the old test suite to break. ## Solution This change fixes the CSS to match the new quick actions menu, and rewrites the tests to utilize the current Quick Actions commands --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Boyu <wangby56@gmail.com>
## Change Added logic in the auth flow to make the auth autonomous LOCALLY only if you are already mid-way authenticated. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
fix(amazonq): fix for merge conflicts for e2e tests branch
Merge master into feature/ui-e2e-tests
|
⏳ I'm reviewing this pull request for security vulnerabilities and code quality issues. I'll provide an update when I'm done |
|
✅ I finished the code review, and didn't find any security or code quality issues. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Solution
feature/xbranches will not be squash-merged at release time.