Failed final renaming when downloading to a network volume

[gillesdegottex]

Hi,

When calling s3.download_file(.) with destination being on a network volume, the following can appear:

    s3.download_file(
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/inject.py", line 192, in download_file
    return transfer.download_file(
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/transfer.py", line 405, in download_file
    future.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 264, in result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 135, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 158, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/download.py", line 671, in _main
    osutil.rename_file(fileobj.name, final_filename)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/utils.py", line 291, in rename_file
    rename_file(current_filename, new_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/network-volume/file.pth.9050F218' -> '/network-volume/file.pth'

My guess is: It is a race condition where the network volume did not expose yet the final file name, whereas the download seems to have finished from the perspective of s3transfer.

Maybe add a few retries on checking the existence of the final file before raising the exception, as long as it is below the connection timeout?

This is of course not systematic and actually quite rare, though problematic in a production environment.

This happens on runpod.io, though I believe this can easily happen on any network mounted volume. Thus, the creation of this issue on this repo.