S3: Add LegacyMd5Plugin to S3 client builder #12264

ebyhr · 2025-02-14T04:48:33Z

The recent AWS SDK bump introduced strong integrity checksums, and broke compatibility with many S3-compatible object storages (pre-2025 Minio, Vast, Dell EC etc).

In Trino project, we received the error report (Missing required header for this request: Content-Md5) from several users and had to disable the check temporarily. We recommend disabling it in Iceberg as well. I faced this issue when I tried upgrading Iceberg library to 1.8.0 in Trino.

Relates to trinodb/trino#24954 & trinodb/trino#24713

amogh-jahagirdar · 2025-02-14T05:51:05Z

aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java

+  // TODO Remove me once all of the S3-compatible storage support strong integrity checks
+  @SuppressWarnings("deprecation")
+  static AwsRequestOverrideConfiguration disableStrongIntegrityChecksums() {
+    return AwsRequestOverrideConfiguration.builder().signer(AwsS3V4Signer.create()).build();


I'm not sure this is the right way to disable it? Feels like this is just a side effect of setting a new signer but I don't think we want that.

aws/aws-sdk-java-v2#5848 (comment)

https://github.com/aws/aws-sdk-java-v2/blob/468d10445765c26deeaa451d34c446b6f0616f6b/services/s3/src/it/java/software/amazon/awssdk/services/s3/checksum/HttpChecksumIntegrationTest.java#L428

Oh I see, is the issue in compatibility is that older versions of Minio/3rd party object storage solutions expect Content-MD5 and in the new SDK we are not sending that and so the service rejects the request? Still feels like there should be a different way to force setting MD5

Setting WHEN_REQUIRED for checksum calculation/validation doesn't resolve as far as I tested. Calculating MD5 for PutObjectRequest looks feasible, but I'm not sure how to do it for DeleteObjectsRequest.

@wendigo Do you happen to know any other approaches?

We have the same issue on the PyIceberg: apache/iceberg-python#1546 How about making this configurable?

@ebyhr no. The only workaround that I have found is to force previous signature and it works against Dell ECS, old Minio versions and Ozone. I haven't tested Vast since I don't have access to it

We added support in Ceph. There is a lot of places this can pop up in PutObject, UploadPart, CompleteMultipartUpload, AWSv2 vs AWSv4 semantics, etc.

We noticed the breakage when watsonx.data rev'd their version of the java sdk

amogh-jahagirdar · 2025-02-14T05:54:33Z

aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java

@@ -149,4 +151,10 @@ static void configurePermission(
      Function<ObjectCannedACL, S3Request.Builder> aclSetter) {
    aclSetter.apply(s3FileIOProperties.acl());
  }
+
+  // TODO Remove me once all of the S3-compatible storage support strong integrity checks


I understand the intent but this doesn't feel like a practical way of going about this just because there's many out there and "all" just seems like a moving goalpost.

nastra · 2025-02-19T10:03:52Z

aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java

@@ -29,15 +29,18 @@
 import software.amazon.awssdk.services.s3.S3ClientBuilder;

 public class MinioUtil {
+  public static final String LATEST_TAG = "latest";
+  // This version doesn't support strong integrity checks
+  static final String LEGACY_TAG = "RELEASE.2024-12-18T13-15-44Z";


can we try and roll back all of the changes in the PR and then only update the TAG to RELEASE.2025-01-20T14-49-07Z (see also minio/minio#20845 (comment)). I've tried it locally and that fixed the issue described in #12237 when running those tests locally with Docker Desktop

@nastra it does, but it won't still work with some other S3-compatible storages like Vast, Dell ECS, so upgrading Minio to compatible version doesn't solve the issue

@nastra We intentionally specified pre-2025 here. Using a newer tag hides the actual problem.

nastra · 2025-02-19T16:53:27Z

I think we should revert the AWS SDK version (#12339) for 1.8.1 and then properly fix it for 1.9.0. @ebyhr does that make sense?

wendigo · 2025-02-19T17:01:08Z

@nastra that will work :)

RussellSpitzer · 2025-02-20T21:13:18Z

@ebyhr How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

nk1506 · 2025-02-21T11:41:46Z

How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

Hi @RussellSpitzer,
@jbonofre had a similar vision, which led to the creation of Project Iceland with the goal of providing integration test coverage for various catalogs. Initially, the plan was to cover all catalog types, but given the recent focus on REST and Polaris, we are currently prioritizing REST-based catalogs.

mmgaggle · 2025-02-25T00:15:50Z

How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

Hi @RussellSpitzer, @jbonofre had a similar vision, which led to the creation of Project Iceland with the goal of providing integration test coverage for various catalogs. Initially, the plan was to cover all catalog types, but given the recent focus on REST and Polaris, we are currently prioritizing REST-based catalogs.

This exists, but it's mostly python / boto based.

https://github.com/ceph/s3-tests

Various vendors create test runners to validate S3 compat including Snowflake, Splunk (uses an older branch of s3-tests), Terradata, etc.

Hadoop common has one as well:

https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/test

If there were an Iceberg one, it would be something we'd validate our Ceph releases against.

steveloughran · 2025-03-13T12:48:24Z

@mmgaggle I'm actually setting up the s3a tests to actually test through iceberg and parquet, so we can validate features and performance optimisations through our code.

Initially, apache/hadoop#7316 has gone in for the bulk delete API of #10233; (please can someone review/merge this!)... it will then act as a regression test of the s3a connector, as well as being easy test local iceberg/parquet builds against arbitrary stores through our test harness. That test harness uses the hadoop IOStatistics API to make assertions about the actual number of remote S3 calls made -this lets you identify regressions in the actually amount of network IO which takes place. Everyone cares about this.

Even with this, you should have a test harness which

can be targeted at production S3 stores
contains a good set of operations, both low level FileIO and higher level API calls
has many of those tests abstracted up to work with all FileIO implementation.
provides really good diagnostics on test failures.

If someone starts that, I'd be happy to help. What i'm not going to is say "here are the tests you need". I did try to do that with spark and the spark-hadoop-cloud module, but there was no interest in full integration tests. I'd only do it for iceberg as part of a collaborative work with others.

ajantha-bhat · 2025-03-17T13:21:33Z

@ebyhr: Thanks for pinging on the mailing list. I have added the 1.9.0 milestone for this. If we can get it merged in a week. We can include this.

So, the only blocker is we want to have integration tests? I think running tests at Trino and sharing report is should be enough for this release, as the fix seems to impacting users. But agree that we need a strong framework in Iceberg or as a separate project to validate these before every release.

aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java

steveloughran · 2025-03-17T18:08:16Z

if you are really in a hurry to ship, use a 2.29.x SDK version.

ajantha-bhat · 2025-03-26T07:48:55Z

I have opened a PR to revert the version in master for 1.9.0 release as this PR didn't made good progress within required time.
#12649

We can still continue work on this. Thanks for all the efforts.

steveloughran · 2025-04-01T10:26:04Z

FWIW I'm trying to convince the AWS SDK team that third party stores do matter -so the SDK should add an option to be compatible -without downstream libraries having to implement workarounds. Let's see what happens.

The more applications which say "no, will stay at 2.29.52 for now", the more motivation the SDK team should have for this

stubz151 · 2025-04-29T15:31:48Z

aws/aws-sdk-java-v2#6055 I see that the SDK team has merged a fix for this. @ebyhr are you going to update this PR? If not I'm happy to own the sdk upgrade.

ebyhr · 2025-04-29T22:32:30Z

@stubz151 Yse, I'm going to update this PR :)

Newer SDKs break compatibility with third-party S3-compatible services. See: apache/iceberg#12264

Newer versions break compatibility with S3-compatible services. See: - apache/iceberg#12264 - Graylog2/graylog-plugin-enterprise#10504

aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java

github-actions · 2025-06-05T00:17:36Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java

stevenzwu

LGTM

nastra · 2025-07-03T09:15:15Z

aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java

@@ -82,11 +83,15 @@ public class AwsClientProperties implements Serializable {
  /** Controls whether vended credentials should be refreshed or not. Defaults to true. */
  public static final String REFRESH_CREDENTIALS_ENABLED = "client.refresh-credentials-enabled";

+  /** Controls whether legacy md5 plugin should be added or not. Defaults to false. */


we might want to improve the docs here and mention what this flag does exactly as it won't be obvious to most people what this flag actually does. I would probably add some description that says the SDK's behavior changed in version X and therefore affects S3-compliant storage implementations since that version and such

we might also want to include the wording from https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/LegacyMd5Plugin.html here to make it clear for users

nastra · 2025-07-03T09:18:32Z

aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java

@@ -106,6 +106,7 @@ static class DefaultAwsClientFactory implements AwsClientFactory {
    @Override
    public S3Client s3() {
      return S3Client.builder()
+          .applyMutation(awsClientProperties::applyLegacyMd5Plugin)


I believe we also need to update DefaultS3FileIOAwsClientFactory

we do have also other implementations of AwsClientFactory, so we should check those places as well whether this needs to be applied there

nastra · 2025-07-04T06:07:46Z

aws/src/main/java/org/apache/iceberg/aws/s3/DefaultS3FileIOAwsClientFactory.java

@@ -46,6 +46,7 @@ public void initialize(Map<String, String> properties) {
  public S3Client s3() {
    return S3Client.builder()
        .applyMutation(awsClientProperties::applyClientRegionConfiguration)
+        .applyMutation(awsClientProperties::applyLegacyMd5Plugin)


this should most likely be applied to the s3Async() case below as well

github-actions bot added the AWS label Feb 14, 2025

ebyhr force-pushed the ebi/s3-integrity-check branch from e629e83 to 83185e7 Compare February 14, 2025 05:02

amogh-jahagirdar reviewed Feb 14, 2025

View reviewed changes

This was referenced Feb 17, 2025

Update Iceberg version to 1.8.0 trinodb/trino#24713

Closed

CORE: return false when view exists endpoint isn't supported #12259

Closed

nastra reviewed Feb 19, 2025

View reviewed changes

nastra mentioned this pull request Feb 19, 2025

TestS3FileIO fails locally (on OSX with Docker Desktop) due to missing Content-MD5 header during delete #12237

Open

3 tasks

zeroshade mentioned this pull request Feb 24, 2025

Support for list types? apache/iceberg-go#309

Closed

mderoy mentioned this pull request Feb 27, 2025

S3FileIO does not raise exception when failing to drop/purge data #12420

Open

3 tasks

ajantha-bhat added this to the Iceberg 1.9.0 milestone Mar 17, 2025

jackye1995 reviewed Mar 17, 2025

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java Outdated Show resolved Hide resolved

ajantha-bhat mentioned this pull request Mar 26, 2025

Build: Revert AWS SDK from 2.30.31 to 2.29.52 #12649

Merged

nastra modified the milestones: Iceberg 1.9.0, Iceberg 1.10.0 Mar 26, 2025

bernd added a commit to Graylog2/graylog2-server that referenced this pull request Apr 30, 2025

Revert AWS SDK to 2.29.52

fb17afa

Newer SDKs break compatibility with third-party S3-compatible services. See: apache/iceberg#12264

bernd added a commit to Graylog2/graylog2-server that referenced this pull request Apr 30, 2025

Revert AWS SDK to 2.29.52

0d8fb88

Newer SDKs break compatibility with third-party S3-compatible services. See: apache/iceberg#12264

bernd mentioned this pull request Apr 30, 2025

Revert AWS SDK to 2.29.52 Graylog2/graylog2-server#22449

Merged

bernd added a commit to Graylog2/graylog2-server that referenced this pull request Apr 30, 2025

Restrict AWS SDK updates to version 2.29.x (#22451)

abc1213

Newer versions break compatibility with S3-compatible services. See: - apache/iceberg#12264 - Graylog2/graylog-plugin-enterprise#10504

ebyhr force-pushed the ebi/s3-integrity-check branch from 83185e7 to 5062417 Compare April 30, 2025 12:48

github-actions bot added the KAFKACONNECT label Apr 30, 2025

ebyhr force-pushed the ebi/s3-integrity-check branch 2 times, most recently from 9555b8c to 54cf4e5 Compare April 30, 2025 12:52

ebyhr changed the title ~~S3: Disable strong integrity checksums~~ Build, S3: Update awssdk-bom to 2.31.32 + Add LegacyMd5Plugin to S3 client builder Apr 30, 2025

stubz151 reviewed Apr 30, 2025

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/s3-integrity-check branch from 54cf4e5 to 1eb0925 Compare May 5, 2025 06:23

ebyhr changed the title ~~Build, S3: Update awssdk-bom to 2.31.32 + Add LegacyMd5Plugin to S3 client builder~~ S3: Add LegacyMd5Plugin to S3 client builder May 5, 2025

github-actions bot removed the KAFKACONNECT label May 5, 2025

github-actions bot added the stale label Jun 5, 2025

nastra added not-stale and removed stale labels Jun 5, 2025

quantranhong1999 mentioned this pull request Jun 10, 2025

JAMES-4137 Upgrade S3 SDK to 2.30.x onwards apache/james-project#2739

Merged

zeitounator mentioned this pull request Jun 24, 2025

Cannot backup on s3_compatible with version 0.24.1 (XAmzContentSHA256Mismatch) thelastpickle/cassandra-medusa#873

Open

stubz151 approved these changes Jun 24, 2025

View reviewed changes

ebyhr force-pushed the ebi/s3-integrity-check branch from 1eb0925 to 0740a68 Compare July 2, 2025 20:39

stevenzwu reviewed Jul 2, 2025

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java Outdated Show resolved Hide resolved

stevenzwu reviewed Jul 2, 2025

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java Outdated Show resolved Hide resolved

stevenzwu approved these changes Jul 2, 2025

View reviewed changes

nastra reviewed Jul 3, 2025

View reviewed changes

ebyhr added 2 commits July 4, 2025 10:32

S3: Add LegacyMd5Plugin to S3 client builder

8cf9c0a

fixup! S3: Add LegacyMd5Plugin to S3 client builder

5c9bcfe

ebyhr force-pushed the ebi/s3-integrity-check branch from f9dfaf7 to 5c9bcfe Compare July 4, 2025 01:37

nastra reviewed Jul 4, 2025

View reviewed changes

fixup! S3: Add LegacyMd5Plugin to S3 client builder

1fc8cbc

nastra approved these changes Jul 4, 2025

View reviewed changes

nastra merged commit fb0af7e into apache:main Jul 4, 2025
42 checks passed

S3: Add LegacyMd5Plugin to S3 client builder #12264

S3: Add LegacyMd5Plugin to S3 client builder #12264

Uh oh!

Conversation

ebyhr commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebyhr Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmgaggle Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebyhr Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra commented Feb 19, 2025

Uh oh!

wendigo commented Feb 19, 2025

Uh oh!

RussellSpitzer commented Feb 20, 2025

Uh oh!

nk1506 commented Feb 21, 2025

Uh oh!

mmgaggle commented Feb 25, 2025

Uh oh!

steveloughran commented Mar 13, 2025

Uh oh!

ajantha-bhat commented Mar 17, 2025

Uh oh!

Uh oh!

steveloughran commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajantha-bhat commented Mar 26, 2025

Uh oh!

steveloughran commented Apr 1, 2025

Uh oh!

stubz151 commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebyhr commented Apr 29, 2025

Uh oh!

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

stevenzwu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebyhr commented Feb 14, 2025 •

edited

Loading

ebyhr Feb 14, 2025 •

edited

Loading

mmgaggle Feb 25, 2025 •

edited

Loading

amogh-jahagirdar Feb 14, 2025 •

edited

Loading

ebyhr Feb 19, 2025 •

edited

Loading

steveloughran commented Mar 17, 2025 •

edited

Loading

stubz151 commented Apr 29, 2025 •

edited

Loading