-
Notifications
You must be signed in to change notification settings - Fork 156
Add recipe specification #3884
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add recipe specification #3884
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for kicking this off! Added a couple comments.
A recipe is a YAML file that captures a complete, static snapshot of a tmt | ||
run after all dynamic evaluation has been resolved. It includes preprocessed | ||
information about plans, tests, results, and run configuration. All environment | ||
variables are evaluated, and git references are replaced with fixed commit hashes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps it would be good to also mention here the use case which was the main motivation for creating the concept of recipes: Processing the recipe and filtering tests in order to submit only a subset of the discovered tests.
# Run configuration | ||
run: | ||
root: "/path/to/fmf/root" | ||
remove: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to be able to reproduce the run we need the git repo details as well. It should be basically fmf id, just without the name, so url
of the repo, ref
in the form of a hash and path
to the metadata tree. The name
part should be covered by plan names under the plans
key below.
Also, I guess, we should store here the original environment
and context
with which the run was created? The complete environment
should be stored for each test in tests.yaml
, but the environment variables are also delivered to other places, such as scripts executed during the prepare
step.
results: | ||
- name: /test/name | ||
result: pass | ||
... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we explicitly mention that when submitting a new run based on a recipe the results
key is ignored and new results will be created based on the actual test execution? Or do we expect some integration of the past results as well?
tests: | ||
- name: /test/name | ||
summary: Test summary | ||
... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to be able to execute tests the expanded test metadata will not be enough. Although we do not expect any adjust
rules to be evaluated or any other dynamic steps, we will have to fetch the git repositories again so that the test code itself is available. Perhaps a short mention of this should be added here or in the story description?
This PR introduces a draft of a recipe specification. The recipe can be used by tmt to reproduce a run with the exact same configuration, without requiring additional input. The recipe file contains preprocessed data about tests, results, and run configuration to eliminate the need for dynamic evaluation.
Some notes:
Resolves: #3854
Related: TT-273
Pull Request Checklist