S3: Disable strong integrity checksums #12264

ebyhr · 2025-02-14T04:48:33Z

The recent AWS SDK bump introduced strong integrity checksums, and broke compatibility with many S3-compatible object storages (pre-2025 Minio, Vast, Dell EC etc).

In Trino project, we received the error report (Missing required header for this request: Content-Md5) from several users and had to disable the check temporarily. We recommend disabling it in Iceberg as well. I faced this issue when I tried upgrading Iceberg library to 1.8.0 in Trino.

Relates to trinodb/trino#24954 & trinodb/trino#24713

amogh-jahagirdar · 2025-02-14T05:51:05Z

aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java

+  // TODO Remove me once all of the S3-compatible storage support strong integrity checks
+  @SuppressWarnings("deprecation")
+  static AwsRequestOverrideConfiguration disableStrongIntegrityChecksums() {
+    return AwsRequestOverrideConfiguration.builder().signer(AwsS3V4Signer.create()).build();


I'm not sure this is the right way to disable it? Feels like this is just a side effect of setting a new signer but I don't think we want that.

aws/aws-sdk-java-v2#5848 (comment)

https://github.com/aws/aws-sdk-java-v2/blob/468d10445765c26deeaa451d34c446b6f0616f6b/services/s3/src/it/java/software/amazon/awssdk/services/s3/checksum/HttpChecksumIntegrationTest.java#L428

Oh I see, is the issue in compatibility is that older versions of Minio/3rd party object storage solutions expect Content-MD5 and in the new SDK we are not sending that and so the service rejects the request? Still feels like there should be a different way to force setting MD5

Setting WHEN_REQUIRED for checksum calculation/validation doesn't resolve as far as I tested. Calculating MD5 for PutObjectRequest looks feasible, but I'm not sure how to do it for DeleteObjectsRequest.

@wendigo Do you happen to know any other approaches?

We have the same issue on the PyIceberg: apache/iceberg-python#1546 How about making this configurable?

@ebyhr no. The only workaround that I have found is to force previous signature and it works against Dell ECS, old Minio versions and Ozone. I haven't tested Vast since I don't have access to it

We added support in Ceph. There is a lot of places this can pop up in PutObject, UploadPart, CompleteMultipartUpload, AWSv2 vs AWSv4 semantics, etc.

We noticed the breakage when watsonx.data rev'd their version of the java sdk

amogh-jahagirdar · 2025-02-14T05:54:33Z

aws/src/main/java/org/apache/iceberg/aws/s3/S3RequestUtil.java

@@ -149,4 +151,10 @@ static void configurePermission(
      Function<ObjectCannedACL, S3Request.Builder> aclSetter) {
    aclSetter.apply(s3FileIOProperties.acl());
  }
+
+  // TODO Remove me once all of the S3-compatible storage support strong integrity checks


I understand the intent but this doesn't feel like a practical way of going about this just because there's many out there and "all" just seems like a moving goalpost.

nastra · 2025-02-19T10:03:52Z

aws/src/test/java/org/apache/iceberg/aws/s3/MinioUtil.java

@@ -29,15 +29,18 @@
 import software.amazon.awssdk.services.s3.S3ClientBuilder;

 public class MinioUtil {
+  public static final String LATEST_TAG = "latest";
+  // This version doesn't support strong integrity checks
+  static final String LEGACY_TAG = "RELEASE.2024-12-18T13-15-44Z";


can we try and roll back all of the changes in the PR and then only update the TAG to RELEASE.2025-01-20T14-49-07Z (see also minio/minio#20845 (comment)). I've tried it locally and that fixed the issue described in #12237 when running those tests locally with Docker Desktop

@nastra it does, but it won't still work with some other S3-compatible storages like Vast, Dell ECS, so upgrading Minio to compatible version doesn't solve the issue

@nastra We intentionally specified pre-2025 here. Using a newer tag hides the actual problem.

nastra · 2025-02-19T16:53:27Z

I think we should revert the AWS SDK version (#12339) for 1.8.1 and then properly fix it for 1.9.0. @ebyhr does that make sense?

wendigo · 2025-02-19T17:01:08Z

@nastra that will work :)

RussellSpitzer · 2025-02-20T21:13:18Z

@ebyhr How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

nk1506 · 2025-02-21T11:41:46Z

How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

Hi @RussellSpitzer,
@jbonofre had a similar vision, which led to the creation of Project Iceland with the goal of providing integration test coverage for various catalogs. Initially, the plan was to cover all catalog types, but given the recent focus on REST and Polaris, we are currently prioritizing REST-based catalogs.

mmgaggle · 2025-02-25T00:15:50Z

How hard would it be for us to get some integration tests with one of these systems into the Iceberg project? Seems like we should have some coverage for these other S3-Compat systems. I'd also be ok with a separate project that we just use as a canary before release.

Hi @RussellSpitzer, @jbonofre had a similar vision, which led to the creation of Project Iceland with the goal of providing integration test coverage for various catalogs. Initially, the plan was to cover all catalog types, but given the recent focus on REST and Polaris, we are currently prioritizing REST-based catalogs.

This exists, but it's mostly python / boto based.

https://github.com/ceph/s3-tests

Various vendors create test runners to validate S3 compat including Snowflake, Splunk (uses an older branch of s3-tests), Terradata, etc.

Hadoop common has one as well:

https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-aws/src/test

If there were an Iceberg one, it would be something we'd validate our Ceph releases against.

steveloughran · 2025-03-13T12:48:24Z

@mmgaggle I'm actually setting up the s3a tests to actually test through iceberg and parquet, so we can validate features and performance optimisations through our code.

Initially, apache/hadoop#7316 has gone in for the bulk delete API of #10233; (please can someone review/merge this!)... it will then act as a regression test of the s3a connector, as well as being easy test local iceberg/parquet builds against arbitrary stores through our test harness. That test harness uses the hadoop IOStatistics API to make assertions about the actual number of remote S3 calls made -this lets you identify regressions in the actually amount of network IO which takes place. Everyone cares about this.

Even with this, you should have a test harness which

can be targeted at production S3 stores
contains a good set of operations, both low level FileIO and higher level API calls
has many of those tests abstracted up to work with all FileIO implementation.
provides really good diagnostics on test failures.

If someone starts that, I'd be happy to help. What i'm not going to is say "here are the tests you need". I did try to do that with spark and the spark-hadoop-cloud module, but there was no interest in full integration tests. I'd only do it for iceberg as part of a collaborative work with others.

ajantha-bhat · 2025-03-17T13:21:33Z

@ebyhr: Thanks for pinging on the mailing list. I have added the 1.9.0 milestone for this. If we can get it merged in a week. We can include this.

So, the only blocker is we want to have integration tests? I think running tests at Trino and sharing report is should be enough for this release, as the fix seems to impacting users. But agree that we need a strong framework in Iceberg or as a separate project to validate these before every release.

jackye1995 · 2025-03-17T17:12:55Z

aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java

@@ -275,6 +275,7 @@ private List<String> deleteBatch(String bucket, Collection<String> keysToDelete)
        DeleteObjectsRequest.builder()
            .bucket(bucket)
            .delete(Delete.builder().objects(objectIds).build())
+            .overrideConfiguration(S3RequestUtil.disableStrongIntegrityChecksums())


Sorry late for this, just saw the devlist. Have we discussed if we could introduce a config for this (can default to false), so people can still turn this check on if they want to?

We already have s3.checksum-enabled in S3FileIOProperties. Should we reuse it, or would it be better to introduce a new one?

I think they're slightly different checksums; there's always been md5 in places (bulk delete?) ; AWS S3 and this SDK have moved to a stronger checksum algorithm. When that is disabled you still need to restore the MD5 signing where retired.

It may be best to move to an enum here, e.g. "none, classic, strong" to cope with any future changes.

steveloughran · 2025-03-17T18:08:16Z

if you are really in a hurry to ship, use a 2.29.x SDK version.

github-actions bot added the AWS label Feb 14, 2025

S3: Disable strong integrity checksums

83185e7

ebyhr force-pushed the ebi/s3-integrity-check branch from e629e83 to 83185e7 Compare February 14, 2025 05:02

amogh-jahagirdar reviewed Feb 14, 2025

View reviewed changes

This was referenced Feb 17, 2025

Update Iceberg version to 1.8.0 trinodb/trino#24713

Closed

CORE: return false when view exists endpoint isn't supported #12259

Closed

nastra reviewed Feb 19, 2025

View reviewed changes

nastra mentioned this pull request Feb 19, 2025

TestS3FileIO fails locally (on OSX with Docker Desktop) due to missing Content-MD5 header during delete #12237

Open

3 tasks

zeroshade mentioned this pull request Feb 24, 2025

Support for list types? apache/iceberg-go#309

Closed

mderoy mentioned this pull request Feb 27, 2025

S3FileIO does not raise exception when failing to drop/purge data #12420

Open

3 tasks

ajantha-bhat added this to the Iceberg 1.9.0 milestone Mar 17, 2025

jackye1995 reviewed Mar 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3: Disable strong integrity checksums #12264

S3: Disable strong integrity checksums #12264

ebyhr commented Feb 14, 2025 •

edited

Loading

amogh-jahagirdar Feb 14, 2025

amogh-jahagirdar Feb 14, 2025

ebyhr Feb 14, 2025 •

edited

Loading

Fokko Feb 14, 2025

wendigo Feb 16, 2025

mmgaggle Feb 25, 2025 •

edited

Loading

amogh-jahagirdar Feb 14, 2025 •

edited

Loading

nastra Feb 19, 2025

wendigo Feb 19, 2025

ebyhr Feb 19, 2025 •

edited

Loading

nastra commented Feb 19, 2025

wendigo commented Feb 19, 2025

RussellSpitzer commented Feb 20, 2025

nk1506 commented Feb 21, 2025

mmgaggle commented Feb 25, 2025

steveloughran commented Mar 13, 2025

ajantha-bhat commented Mar 17, 2025

jackye1995 Mar 17, 2025

ebyhr Mar 21, 2025

steveloughran Mar 21, 2025

steveloughran commented Mar 17, 2025 •

edited

Loading

S3: Disable strong integrity checksums #12264

Are you sure you want to change the base?

S3: Disable strong integrity checksums #12264

Conversation

ebyhr commented Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmgaggle Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

amogh-jahagirdar Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

nastra commented Feb 19, 2025

wendigo commented Feb 19, 2025

RussellSpitzer commented Feb 20, 2025

nk1506 commented Feb 21, 2025

mmgaggle commented Feb 25, 2025

steveloughran commented Mar 13, 2025

ajantha-bhat commented Mar 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveloughran commented Mar 17, 2025 • edited Loading

ebyhr commented Feb 14, 2025 •

edited

Loading

ebyhr Feb 14, 2025 •

edited

Loading

mmgaggle Feb 25, 2025 •

edited

Loading

amogh-jahagirdar Feb 14, 2025 •

edited

Loading

ebyhr Feb 19, 2025 •

edited

Loading

steveloughran commented Mar 17, 2025 •

edited

Loading