replace minio client with aws sdk#547
Conversation
…ature/replace-minio-client-with-aws-sdk
|
Thanks, we're just doing some additional testing and should be able to merge this soon. It does seem like Minio's |
Er, no, that's not quite right. fPutObject does support multi-part upload (was it working previously?) https://min.io/docs/minio/linux/developers/javascript/API.html#fPutObject so we probably do want to switch to the corresponding AWS S3 API to ensure that we don't break larger uploads. The WACZ files can get large (and there's no sizeLimit by default) |
|
Thanks for starting this - continuing this work in #943 with multi-part upload support. |
…upload (#943) Extends work in #547 adds Upload via @aws-sdk/lib-storage library: - Replaces minio client with official aws s3 client - Uses @aws-sdk/lib-storage for multi-part upload support Testing: - This should address issues from #479 and webrecorder/browsertrix#2925 ideally - Tested with all the major S3 implementations: VersityGW, RustFS, SeaweedFS, Garage as well as Minio --------- Co-authored-by: Mattia <m@ttia.it> Co-authored-by: Mattia <mattia@guella.it> Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
The Minio client has issues connecting to different providers (e.g. Cloudflare R2, see minio/minio-js#619 (comment)). Moving to the AWS SDK solves that problem.
This should also support files up to 5GB so it helps with #479 but doesn't solve it completely, as for bigger ones (up to 5TB) the multipart upload is probably needed.
However it's a first step toward solving the issue of big files and it's already a big improvement over the current Minio client as it allows to use S3 compatible providers different from AWS and Minio.