Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auto generated tests #463

Open
Garados007 opened this issue Jan 8, 2024 · 0 comments
Open

Add auto generated tests #463

Garados007 opened this issue Jan 8, 2024 · 0 comments

Comments

@Garados007
Copy link
Member

It is quite difficult to run auto generated tests using igortest. A current approach with the existing igortest could be:

  1. Perform the first test run. (must be inside an IM)
    1. Each test case can generate new code (as a string) and save it as a new file in a specified path.
    2. Run compilation tests using the generated code.
  2. Perform the second test run.
    1. Load all code from the specified path.
    2. Search for test cases there and execute them.
  3. Cleanup all auto generated code.

This results in two test runs and multiple coverage and test result files. You also need to change your CI code to combine all results. It is also difficult to associate the generated code with the generator function.

A simpler approach could be to create new assertion/helper functions that:

  • Take the new code as a string and save them at a temporary path known to igortest.
  • Trigger a compile check (like our compilation assertions). The generated code can also intentionally fail!
  • Only if the generated code compiles and is intended to be compiled:
    • Search for all new test cases in the generated code and inject them after the current test case in our test schedule. The test case names of the injected ones are all prefixed with the current test case name. They all share the same test suite!
    • Execute the new test cases normally.
  • Cleanup all generated code (if desired they can be kept for debugging purpose)
  • Continue with the usual test schedule

Auto generated code can be quite useful if you need lots of combinations that is difficult to test with usual code and needs to be written one by one. It is also quite useful if you want lots of compilation checks with small snippets of code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant