Skip to content

Add recipe specification #3884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add recipe specification #3884

wants to merge 1 commit into from

Conversation

therazix
Copy link
Collaborator

This PR introduces a draft of a recipe specification. The recipe can be used by tmt to reproduce a run with the exact same configuration, without requiring additional input. The recipe file contains preprocessed data about tests, results, and run configuration to eliminate the need for dynamic evaluation.

Some notes:

  • Recipe contains tests with all adjustments already applied. Since context variables and environment settings have been evaluated and propagated to individual tests before recipe generation, do we need to preserve the original environment configurations from different levels (command-line, plan)?
  • How should test requirements be handled? Tests may require additional files. Where should we save them? A similar issue is with Ansible playbooks, but these could potentially be stored directly inside the recipe.

Resolves: #3854
Related: TT-273

Pull Request Checklist

  • implement the feature
  • write the documentation
  • extend the test coverage
  • update the specification
  • adjust plugin docstring
  • modify the json schema
  • mention the version
  • include a release note

@therazix therazix added this to the 1.53 milestone Jul 16, 2025
@therazix therazix added the specification Metadata specification (core, tests, plans, stories) label Jul 16, 2025
@github-project-automation github-project-automation bot moved this to backlog in planning Jul 16, 2025
@therazix therazix moved this from backlog to implement in planning Jul 16, 2025
Copy link
Collaborator

@psss psss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for kicking this off! Added a couple comments.

A recipe is a YAML file that captures a complete, static snapshot of a tmt
run after all dynamic evaluation has been resolved. It includes preprocessed
information about plans, tests, results, and run configuration. All environment
variables are evaluated, and git references are replaced with fixed commit hashes.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it would be good to also mention here the use case which was the main motivation for creating the concept of recipes: Processing the recipe and filtering tests in order to submit only a subset of the discovered tests.

# Run configuration
run:
root: "/path/to/fmf/root"
remove: false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to be able to reproduce the run we need the git repo details as well. It should be basically fmf id, just without the name, so url of the repo, ref in the form of a hash and path to the metadata tree. The name part should be covered by plan names under the plans key below.

Also, I guess, we should store here the original environment and context with which the run was created? The complete environment should be stored for each test in tests.yaml, but the environment variables are also delivered to other places, such as scripts executed during the prepare step.

results:
- name: /test/name
result: pass
...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we explicitly mention that when submitting a new run based on a recipe the results key is ignored and new results will be created based on the actual test execution? Or do we expect some integration of the past results as well?

tests:
- name: /test/name
summary: Test summary
...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to be able to execute tests the expanded test metadata will not be enough. Although we do not expect any adjust rules to be evaluated or any other dynamic steps, we will have to fetch the git repositories again so that the test code itself is available. Perhaps a short mention of this should be added here or in the story description?

@psss psss modified the milestones: 1.53, 1.54 Jul 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
specification Metadata specification (core, tests, plans, stories)
Projects
Status: implement
Development

Successfully merging this pull request may close these issues.

Create the tmt recipe specification
2 participants