[Scout] Test config manifests #245135
Open
[Scout] Test config manifests #245135
+9,183
−76
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dolaru
added
release_note:skip
elastic-vault-github-plugin-prod
bot
requested review from
a team
as code owners
elastic-vault-github-plugin-prod
bot
requested review from
a team
as code owners
Because Scout is based on Playwright, detailed information about tests is only available while Playwright is running. This means that any time we need test-level information from a test config file we have to start a Playwright process which can be quite slow and resource-intensive if you want to do this for a larger number of tests configurations. These changes are trying to address that by introducing a system where this runtime-only information is stored in manifest files, stored in the proximity of test config files, automatically kept up to date by CI. ### Usage Locally, run the new kbn-scout CLI command (via `node scripts/scout update-test-config-manifests`) to regenerate manifests. - Running without `--includingUpToDate` will only regenerate manifests that appear outdated (component directory SHA mismatch) - Use `--includingUpToDate` to force regenerate all manifests In Buildkite, `node scripts/scout update-test-config-manifests` runs as a quick check and any changes will be auto-commited. ### Notable changes - Add Scout reporting package with a registry of test configs and components - Add a Playwright reporter that writes manifest files that capture test information for static analysis - Add a Scout CLI command `update-test-config-manifests` to create/update manifests - Add a versatile Playwright CLI wrapper for Scout tooling to call Playwright programmatically - Add Buildkite quick check to run manifest updates and commit any changes as `[Scout] Automated config manifest updates` - Add new path constants/globs/regex for discovering Scout configs - Changed how test IDs are calculated for Scout: combine test file path and test title when hashing - Remove project name and test file path from Scout test titles
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Public APIs missing comments
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
History
cc @dolaru |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because Scout is based on Playwright, detailed information about tests is only available while Playwright is running. This means that any time we need test-level information from a test config file we have to start a Playwright process which can be quite slow and resource-intensive if you want to do this for a larger number of test configs.
These changes are trying to address that by introducing a system where this runtime-only information is stored in manifest files, in close proximity of test config files, automatically kept up to date by CI.
Usage
Locally, run the new kbn-scout CLI command (via
node scripts/scout update-test-config-manifests) to regenerate manifests.--includingUpToDatewill only regenerate manifests that appear outdated (component directory SHA mismatch)--includingUpToDateto force regenerate all manifestsIn Buildkite,
node scripts/scout update-test-config-manifestsruns as a quick check and any changes will be auto-commited.Notable changes
update-test-config-manifeststo create/update manifests[Scout] Automated config manifest updates