-
Notifications
You must be signed in to change notification settings - Fork 566
Open
Description
What happened?
Description
Merge operations fail with case sensitivity errors when a table has camelCase column names and nullable=False.
Error
DeltaError: Schema error: No field named submittedat. Column names are case sensitive.
You can use double quotes to refer to the "submittedat" column or set the
datafusion.sql_parser.enable_ident_normalization configuration. Did you mean 'submittedAt'?.
Very bizarrely, it seems like this error only hits for the combination of nullable=False and camelCase columns.
Changing nullable=False to nullable=True makes it work. The bug only triggers when:
- Column name has mixed case (e.g., submittedAt)
- Column has nullable=False
Expected behavior
It "Just Works", as it did in 1.3.0.
Operating System
macOS
Binding
Python
Bindings Version
1.3.2
Steps to reproduce
Run this code with/without the camelCase column having nullable=False.
from deltalake import DeltaTable, Field, Schema
from deltalake.schema import PrimitiveType
import pyarrow as pa
import tempfile
path = tempfile.mkdtemp()
schema = Schema([
Field("submittedAt", PrimitiveType("long"), nullable=False),
Field("id", PrimitiveType("string"), nullable=True),
])
DeltaTable.create(table_uri=path, schema=schema, mode="ignore")
table = DeltaTable(path)
data = pa.Table.from_pylist([{"submittedAt": 123, "id": "test"}])
(
table.merge(data, "source.id = target.id", source_alias="source", target_alias="target")
.when_matched_update_all()
.when_not_matched_insert_all()
.execute()
)Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
No status