CI error: `All chunks should've been deleted` keeps coming back

[deependujha]

🐛 Bug

https://github.com/Lightning-AI/litdata/actions/runs/12377277308/job/34546400027

This CI issue keeps coming back, even though it was fixed before.

To Reproduce

Steps to reproduce the behavior...

Code sample

# Ideally attach a minimal code sample to reproduce the decried issue.
# Minimal means having the shortest code but still preserving the bug.

Expected behavior

Additional context

Environment detail

• PyTorch Version (e.g., 1.0):
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bhimrazy]

Functions falling under this:

• tests/streaming/test_dataset.py::test_subsample_streaming_dataset_with_token_loader
• tests/streaming/test_dataset.py::test_dataset_for_text_tokens_distributed_num_workers_end_to_end