Explore Optimizing and Running Tests in Parallel for Faster CI in litdata #612
Description
Title: Explore Running Tests in Parallel for Faster CI in litdata
Description:
We should investigate the possibility of running the test suite in parallel to speed up CI and local test execution. As the test base grows, serial execution becomes a bottleneck, especially when contributing frequently or iterating on PRs.
🧪 An initial test was conducted in #608.
Tasks:
- Review the current test setup and determine if there are any global state dependencies or side effects.
- Look into test cases optimizations
- Explore using
[pytest-xdist](https://pypi.org/project/pytest-xdist/)or an alternative for parallelization. - Identify and resolve any tests that may fail due to parallel execution.
- Benchmark before/after test runtimes.
Goal:
Improve test speed while maintaining test accuracy and reproducibility.