S3FileIO does not raise exception when failing to drop/purge data

[mderoy]

Apache Iceberg version

1.8.0 (latest release)

Query engine

None

Please describe the bug 🐞

Due to a failure using the latest AWS sdk the deleteObjects api began failing. This is because the AWS SDK 2.30.0 added changes to the default integrity protection (see aws/aws-sdk-java-v2#5802), which negatively affects S3-compliant object storage solutions.

This means that the dropTable api's purge option stopped working.

This caused thousands of objects to build up on our minio s3 filesystem without our immediate knowledge, as the exception was not being raised to the application. Expected behavior would be that failing to purge files should raise some kind of error when calling the catalog.dropTable API

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[nastra]

@mderoy we just released Iceberg 1.8.1 where the AWS SDK version has been downgraded to 2.29.52. Can you please check if that fixes your issue?

[mderoy]

@nastra I have already verified that downgrading to 2.29.52 does in fact fix the issue dropping the objects from S3....however what I've filed this bug for is the lack of any exception or error being raised if such a condition occurs when dropping the table (either due to this bug or any other issue).

[nastra]

@mderoy I've updated the description, since the issue is caused by the AWS SDK making changes to the default integrity protection (see aws/aws-sdk-java-v2#5802), which negatively affects S3-compliant object storage solutions. It isn't caused by the issues you had linked earlier (#12237, #12264).

[steveloughran]

AWS SDK team do not test the SDK with third party stores, it is left to downstream projects. S3a project does has a qualifying an SDK update doc, but I'm revising it to require long-haul connections and at least one third party store (which must be named). Google GCS is the obvious target, as it is easy to setup.

One part of the new doc, along with the "don't make this last minute" will be "pick a release two weeks old, so you can see what bugs have surfaced". Maybe it should become "four weeks old"

[steveloughran]

looking a bulk delete with the v2 sdk, as well as the 2.30.x regression, there are two other issues

• 500 failures of individual files are not reported; if an s3 shard behind the load balancer fails, that is not picked up and retried
• 503 failures of individual files are not escalated to a 503 reponse to the entire message

In both cases, the HTTP request returns 200, and the SDK itself doesn't do retries/throttling. It's left to the caller.