-
Notifications
You must be signed in to change notification settings - Fork 146
Description
S3 transfer has a set of existing exceptions that it will retry when reading from the GetObject
response stream:
s3transfer/s3transfer/utils.py
Lines 44 to 50 in 9a71d52
S3_RETRYABLE_DOWNLOAD_ERRORS = ( | |
socket.timeout, | |
SOCKET_ERROR, | |
ReadTimeoutError, | |
IncompleteReadError, | |
ResponseStreamingError, | |
) |
I would like the ability to configure s3transfer to also retry checksum errors, botocore.exceptions.FlexibleChecksumError
. With the assumption that a failed checksum validation on download is due to a transient issue (i.e network blip, bad hardware somewhere, etc.), it seems reasonable to retry these scenarios. Presumably S3 would have never allowed an object to be stored with an invalid checksum, otherwise you'd never be able to download the object.
It makes sense that is functionality is not possible in botocore, but I think for a high level library like this, there's a case to be made for this being added to the default set of retryable exceptions (when would you not want this behavior?), but if that's too risky for a default change, it would be great if there was some config option where I could add additional exceptions like that that s3transfer could retry.
Upload is a less interesting, but still useful case where checksum errors should be retried (this seems reasonable to add to botocore but I can understand why it's not done there). This case is also harder to implement because it's an unmodeled exception in this scenario (e.g. botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation: The CRC32 you specified did not match the calculated checksum.
), but under the same assumption of a transient issue during upload, I would like these to also be retried (what would be a scenario where you wouldn't want to do this?).