Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QA Team - Keeping organized reports of relevant Rocket-E2E runs #143

Open
MathieuLamiot opened this issue Sep 17, 2024 · 0 comments
Open
Labels
documentation Improvements or additions to documentation

Comments

@MathieuLamiot
Copy link
Contributor

Context

https://wp-media.slack.com/archives/CUKB44GNN/p1726589916414399

We often have failing tests when running Rocket-E2E. While one axis of improvement is to solve those errors, an other one is to keep track of previous Rocket-E2E run reports so that we can check if the errors we face now already occured before (and hence can be discarded).
Currently, we have to re-run the tests in old configurations (old plugin versions, old Rocket-E2E versions) to validate or not if an error is accepted, which cost a lot of time.
Also, having records will help identify if we make progress or not when it comes to reducing the number of errors we face with Rocket-E2E, and will allow us to identify areas of improvements.

Expected outcome
We would need a shared folder or repository to store and access test reports from Rocket-E2E runs. I don't have specific requirements about the setup, but here are some inputs:

  • Rocket-E2E is able to output a test report I believe ; we discussed this with @jeawhanlee and @wp-media/qa-team at some point I think.
  • We don't need to store all the reports from all the runs we do, but we would need to keep the representative ones (typically when a next version of the plugin and/or the rocket-E2E is released or tagged)?
  • We would need a precise naming convention to easily get the version of the plugin, the version of Rocket-E2E (based on Rocket-E2E repo tags ; if they are not used yet, that's a good place to start), and maybe the website tested against?
  • Those details should be properly documented for anyone to easily navigate the reports and add theirs if needed.
@MathieuLamiot MathieuLamiot added the documentation Improvements or additions to documentation label Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

1 participant