Delta protocol version is too new for Athena DDL engine" Error When Querying Delta Tables

[marcoaanogueira]

Environment

Delta-rs version: 0.20.0

Environment:

• Cloud provider: AWS

---

Bug

What happened:
When trying to query Delta Lake tables via Athena, I received the following error:
Delta protocol version is too new for Athena DDL engine.
This error occurred even though the metadata shows a Delta protocol version that, according to the documentation, should be compatible with Athena.

What you expected to happen:
Athena should be able to query the Delta Lake tables without encountering a protocol version incompatibility issue.

How to reproduce it:
Create a Delta Lake table in Athena using the following command:

CREATE EXTERNAL TABLE
<schema>.<table>
LOCATION '<delta table path>'
TBLPROPERTIES ('table_type' = 'DELTA')

More details:

[rtyler]

How was the table created? There may be some features used that pushed the protocol version too high for Athena SQL to use

[marcoaanogueira]

Sorry for closing it; I thought I should close it since the instruction in another issue was to open a support ticket with AWS.

I created the table using the write_deltalake function:

write_deltalake(
        table_or_uri=s3_destination,
        data=pa_source_delta_casted,
        mode="append",
        schema_mode="merge",
        storage_options=STORAGE_OPTIONS,
    )

[marcoaanogueira]

The version appears to be correct and compatible with what is stated in the AWS documentation.

[rtyler]

I would not be surprised if this table was turned into minReaderVersion: 3, minWriterVersion: 7 due to a timestamp without a timezone being in the data set used to create it. Thus it would be related to the tests I added recently in #2935

Which documentation are you referring to? Athena is kind of a name used for Athena Spark and Athena SQL and those have different query engines and Delta support 😱