Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support native S3 conditional writes #6682

Merged

Commits on Nov 5, 2024

  1. Support native S3 conditional writes

    Add support for `PutMode::Create` and `copy_if_not_exists` on native AWS
    S3, which uses the underlying conditional write primitive that Amazon
    launched earlier this year [0].
    
    The conditional write primitive is simpler than what's available in
    other S3-like products (e.g., R2), so new modes for
    `s3_copy_if_not_exists` and `s3_conditional_put` are added to select the
    native S3-specific behavior.
    
    To maintain strict backwards compatibility (e.g. with older versions of
    LocalStack), the new behavior is not on by default. It must be
    explicitly requested by the end user.
    
    The implementation for `PutMode::Create` is straightforward. The
    implementation of `copy_if_not_exists` is a bit more involved, as it
    requires managing a multipart upload that uses the UploadPartCopy
    operation, which was not previously supported by this crate's S3 client.
    
    To ensure test coverage, the object store workflow now runs the AWS
    integration tests with conditional put both disabled and enabled.
    
    Fix apache#6285.
    
    [0]: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
    benesch committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    3206e8c View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Address review feedback

    benesch committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    9cf76bc View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Fix clippy failure

    benesch committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    a974e76 View commit details
    Browse the repository at this point in the history
  2. Upgrade localstack in GitHub Actions

    To a version that supports conditional writes.
    benesch committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    8bbf641 View commit details
    Browse the repository at this point in the history