Local cache dir not fully clearing in DDP multi-node training.

[JackUrb]

🐛 Bug

At the moment it seems that a significant portion of data stored in the cache (~40-60%) ends up not removed over the course of training, and then remains in the cache dir upon completion of training. I suspect this

To Reproduce

Currently training a few models over 8 nodes with a large source dataset (only 1 epoch), and the cache dir size accumulates indefinitely. Over the process, it seems that the lock counts for these files are much greater than 0, so I wonder if it has to do with force_download-related behavior?

Expected behavior

The cache dir does not expand significantly above its target size, and files aren't being locked as many times.

Additional context

Environment detail

• PyTorch Version (e.g., 1.0):
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

[tchaton]

Hey @JackUrb. Could you try to disable force download to see if it helps ?

[JackUrb]

Unfortunately if we disable force download, we end up stalling out in multiple places, and the run has a high chance of failing outright early in changing.

Perhaps the (band-aid) solution when running into a force download is to run the download without increasing the lock count, as supposedly if you arrive in a situation where you would be calling the force, it's because you expected it to be present, meaning you expect to have already incremented the lock?

[tchaton]

Hey @JackUrb. Good idea.

[bhimrazy]

Hi @JackUrb, we recently added a few fixes related to cache clearance. It's not released yet, but available on the main branch—feel free to give it a try and let us know how it goes.

Thanks!

[JackUrb]

Hi @bhimrazy - launching a new run on litdata main this week, will report back.

[bhimrazy]

Hi @bhimrazy - launching a new run on litdata main this week, will report back.

Thank you @JackUrb

[JackUrb]

@bhimrazy Unfortunately no dice - we end up with an overfull cache directory containing all of the dat-0-x.bin from with x between 0 to 40, and then around 1 in 10 of the next 100,000 shards.

The first 40 end up having a .cnt of ~22+/-1, and almost all of the rest have 1 with an occasional 2. I suspect the greater in the first ones to be caused by using the same dataset for val as for train. Two separate runs used the same cache, so the 1's and occasional 2 makes sense to me as some race condition occurring roughly 1 in 20 shards.

[bhimrazy]

Thank you, @JackUrb, for the detailed follow-up.

If possible, could you also share a bit more about your training setup? Specifically:

• Number of nodes
• Number of CPU cores and devices per node
• Number of workers allocated per process
• Cache size allocated per node (in GB)
• Total dataset size (in GB/TB)
• Remaining cache size at the end of epoch (in GB)
• Any other details you think might help with debugging

This info would be really helpful as we try to replicate and test the issue on smaller-scale setups.
Really appreciate your time and insights!