Add tests for TFLite models #5
Comments
|
We can also test across different |
This was referenced Sep 6, 2024
ScottTodd
added a commit
that referenced
this issue
Sep 19, 2024
Progress on #6. A sample test report HTML file is available here: https://scotttodd.github.io/iree-test-suites/onnx_models/report_2024_09_17.html These new tests * Download models from https://github.com/onnx/models * Extract metadata from the models to determine which functions to call with random data * Run the models through [ONNX Runtime](https://onnxruntime.ai/) as a reference implementation * Import the models using `iree-import-onnx` (until we have a better API: iree-org/iree#18289) * Compile the models using `iree-compile` (currently just for `llvm-cpu` but this could be parameterized later) * Run the models using `iree-run-module`, checking outputs using `--expected_output` and the reference data Tests are written in Python using a set of pytest helper functions. As the tests run, they can log details about what commands they are running. When run locally, the `artifacts/` directory will contain all the relevant files. More can be done in follow-up PRs to improve the ergonomics there (like generating flagfiles). Each test case can use XFAIL like `@pytest.mark.xfail(raises=IreeRunException)`. As we test across multiple backends or want to configure the test suite from another repo (e.g. [iree-org/iree](https://github.com/iree-org/iree)), we can explore more expressive marks. Note that unlike the ONNX _operator_ tests, these tests use `onnxruntime` and `iree-import-onnx` at test time. The operator tests handle that as an infrequently ran offline step. We could do something similar here, but the test inputs and outputs can be rather large for real models and that gets into Git LFS or cloud storage territory. If this test authoring model works well enough, we can do something similar for other ML frameworks like TFLite (#5).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See nod-ai/SHARK-TestSuite#291
Search around for upstream test suites (corpus of .tflite files)
Could also test TOSA operators maybe using https://git.mlplatform.org/tosa/conformance_tests.git/ (see https://www.mlplatform.org/tosa/software.html)
The text was updated successfully, but these errors were encountered: