Skip to content

Add support for retrying checksum errors on upload/download #333

@jamesls

Description

@jamesls

S3 transfer has a set of existing exceptions that it will retry when reading from the GetObject response stream:

S3_RETRYABLE_DOWNLOAD_ERRORS = (
socket.timeout,
SOCKET_ERROR,
ReadTimeoutError,
IncompleteReadError,
ResponseStreamingError,
)

I would like the ability to configure s3transfer to also retry checksum errors, botocore.exceptions.FlexibleChecksumError. With the assumption that a failed checksum validation on download is due to a transient issue (i.e network blip, bad hardware somewhere, etc.), it seems reasonable to retry these scenarios. Presumably S3 would have never allowed an object to be stored with an invalid checksum, otherwise you'd never be able to download the object.

It makes sense that is functionality is not possible in botocore, but I think for a high level library like this, there's a case to be made for this being added to the default set of retryable exceptions (when would you not want this behavior?), but if that's too risky for a default change, it would be great if there was some config option where I could add additional exceptions like that that s3transfer could retry.

Upload is a less interesting, but still useful case where checksum errors should be retried (this seems reasonable to add to botocore but I can understand why it's not done there). This case is also harder to implement because it's an unmodeled exception in this scenario (e.g. botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation: The CRC32 you specified did not match the calculated checksum.), but under the same assumption of a transient issue during upload, I would like these to also be retried (what would be a scenario where you wouldn't want to do this?).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions