Using SotoS3FileTransfer copy from occasional HTTP disconnects?

[vade]

Hello

We are using the latest swift SPM version of SOTO / SotoS3 and are having an occasional HTTP disconnect when using the sync from local folder to destination S3.

The error:

Got Error: uploadFailed(HTTPClientError.remoteConnectionClosed, SotoS3FileTransfer.S3FileTransferManager.UploadOperation(transfers: [(from: SotoS3FileTransfer.S3FileTransferManager.FileDescriptor(name: "/var/folders/4k/1wv3k9bj65q4_64fnw814n480000gn/T/job-55d334b4-05c2-4c70-bdee-48432fc32502/2025-03-18:20-14-04.jpg", modificationDate: 2025-03-18 20:14:06 +0000, size: 52143), to: s3://ozu-development-application-asset-bucket/users/0194be2c-47dc-7c93-b27b-48797cb81215/jobs/55d334b4-05c2-4c70-bdee-48432fc325022025-03-18:20-14-04.jpg)]))
HTTP connection closed on the s3 side

The code:

    private func uploadExportToS3() async throws
    {
        guard
            let destS3Folder = S3Folder(url: "s3://\(self.jobData.destS3Bucket)/\(self.jobData.destS3Key)"),
            let parentS3Folder = destS3Folder.parent()
        else
        {
            throw RenderError.invalidS3Destination
        }
        
        
        try await self.s3FileTransfer.sync(from: self.tempJobFolder().path(percentEncoded: false),
                                           to:parentS3Folder, delete: true)
    }

This error occurs VERY seldomly, but weve yet to find a cause.

My understanding is SOTO has retry 'built in' - do I need to be cautious and roll my own retry by catching and re-trying here?

Thanks for SOTO! Its made a lot of complicated stuff way less compliated!

[adam-fowler]

Are any of the files large enough such that s3 file transfer would choose to use multipart upload?

[adam-fowler]

Also have you tried using the resume functions to retry an upload.

[vade]

Thanks for the super prompt reply!

• files are small typically 2-10 Mb although we might have some outliers. Around what sizes does the Soto tooling switch to multipart?

• I have not tried any manual resume or re-upload attempts to date. Is that recommended given that Soto has retry faculties (as I understand it)?

Thanks !

[adam-fowler]

Ok it looks like you will occasionally use multipart to upload files. These currently use streaming to upload the files (so progress can be reported). You cannot retry a streamed file. If you don't care about progress and are ready to take the hit of storing complete files in memory then you can up the configuration value multipartThreshold to something that is larger than your largest file.

Otherwise if you want to continue as is S3FileTransferManager has helper functions that will resume an upload that failed. You can extract the UploadOperation from the error. This will include the files still to upload and then pass that into S3FileTransferManager.resume. I stupidly got the function parameter name wrong, so the parameter is called download.

[vade]

Thanks @adam-fowler

Is there any sample code that documents a correctly formulated retry strategy?

If not, no worries, ill give it a go :)

[adam-fowler]

Sorry no sample code. You could look at the tests eg testCancelledDownload