Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snapshot test for invoked integrationv2 tests #5097

Open
goatgoose opened this issue Feb 6, 2025 · 1 comment
Open

Add snapshot test for invoked integrationv2 tests #5097

goatgoose opened this issue Feb 6, 2025 · 1 comment
Assignees
Labels
priority/medium Rank 3 size/medium type/ci CodeBuild or CI related

Comments

@goatgoose
Copy link
Contributor

goatgoose commented Feb 6, 2025

Problem:

While working on #5096, I found it difficult to determine if a change to the invalid_test_parameters() logic in the integrationv2 framework would result in changes to the tests that are run. It would be nice to have assurance that changes don't result in unintentionally reducing test coverage.

Solution:

We should consider snapshot testing the integrationv2 tests that are invoked for each libcrypto by committing the list of invoked tests into the repo and comparing this list to a new list generated in each PR. If these lists differ, the PR should fail until the committed list is updated. This will ensure that any changes to test coverage are acknowledged by the author and observable to reviewers.

@goatgoose goatgoose changed the title Snapshot test for invoked integrationv2 tests Add snapshot test for invoked integrationv2 tests Feb 6, 2025
@jmayclin
Copy link
Contributor

Request: report at two levels

  • individual test_*.py that was run
  • test instances that were run.

@dougch dougch self-assigned this Mar 3, 2025
@dougch dougch added the type/ci CodeBuild or CI related label Mar 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/medium Rank 3 size/medium type/ci CodeBuild or CI related
Projects
None yet
Development

No branches or pull requests

3 participants