Fix parquet cache

[philgzl]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #544.

This fix removes two lines of code in StreamingDataset.__init__ which were executed only for Hugging Face datasets:

• The first line was causing the cache_dir option to be ignored.
• The second line was causing the _try_create_cache_dir call in subsample_streaming_dataset to be skipped, and input_dir.path = index_path to be used as the cache dir.

The Parquet files are now cached in ~/.lightning/chunks/ by default, like the LitData-optimized chunks. They used to be cached in ~/.cache/litdata-cache-index-pq/ next to the index file. The index file is now still cached in ~/.cache/litdata-cache-index-pq/. The index file is also cached in ~/.lightning/chunks/ or the user-provided cache dir.

Added a test.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79%. Comparing base (cb3fc81) to head (707ccb8).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #560   +/-   ##
===================================
  Coverage    79%    79%           
===================================
  Files        40     40           
  Lines      6101   6099    -2     
===================================
  Hits       4820   4820           
+ Misses     1281   1279    -2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bhimrazy]

LGTM, Thanks @philgzl 🎉

[tchaton]

Nice !