Skip to content

[Bug]: Merge fails on camelCase columns with nullable=False #4082

@FrankPortman

Description

@FrankPortman

What happened?

Description

Merge operations fail with case sensitivity errors when a table has camelCase column names and nullable=False.

Error

DeltaError: Schema error: No field named submittedat. Column names are case sensitive.
You can use double quotes to refer to the "submittedat" column or set the
datafusion.sql_parser.enable_ident_normalization configuration. Did you mean 'submittedAt'?.

Very bizarrely, it seems like this error only hits for the combination of nullable=False and camelCase columns.

Changing nullable=False to nullable=True makes it work. The bug only triggers when:

  1. Column name has mixed case (e.g., submittedAt)
  2. Column has nullable=False

Expected behavior

It "Just Works", as it did in 1.3.0.

Operating System

macOS

Binding

Python

Bindings Version

1.3.2

Steps to reproduce

Run this code with/without the camelCase column having nullable=False.

  from deltalake import DeltaTable, Field, Schema
  from deltalake.schema import PrimitiveType
  import pyarrow as pa
  import tempfile

  path = tempfile.mkdtemp()

  schema = Schema([
      Field("submittedAt", PrimitiveType("long"), nullable=False),
      Field("id", PrimitiveType("string"), nullable=True),
  ])

  DeltaTable.create(table_uri=path, schema=schema, mode="ignore")
  table = DeltaTable(path)
  data = pa.Table.from_pylist([{"submittedAt": 123, "id": "test"}])

  (
      table.merge(data, "source.id = target.id", source_alias="source", target_alias="target")
      .when_matched_update_all()
      .when_not_matched_insert_all()
      .execute()
  )

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions