[Server-Side Planning] Column names containing period

[murali-db]

Summary

Implement proper handling of column names containing dots (literal column names vs. nested fields) for Iceberg server-side planning. This PR adds backtick escaping for literal dotted columns in projections while correctly using raw field names for filters (where Iceberg's binding logic handles disambiguation automatically).

Background

Column names can contain dots in various scenarios:

• Unity Catalog tables with dots in column names
• Flattened nested schemas
• Data from systems that allow dots in column names (e.g., address.city as a single column name)

It's important to distinguish between:

• Literal dotted column names: address.city is a single top-level column
• Nested field references: address.intCol refers to field intCol within struct address

Key Insights

After thorough investigation of both Spark and Iceberg behavior:

1. Projections: Need backtick escaping to distinguish in JSON

   • Example: ["address.city", "address.intCol"]
   • Escaped literal columns vs. unescaped nested field paths

2. Filters: Don't need escaping - Iceberg's schema-aware binding handles it

   • Iceberg Expression API uses raw field names: Expressions.equal("address.city", value)
   • Binder.bind(schema, expression) resolves ambiguity using schema structure
   • Binding checks for literal field match FIRST, then tries nested parsing
   • JSON sent over HTTP also uses raw field names (no escaping)

Implementation

Projection Column Escaping

File: ServerSidePlannedTable.scala (only file changed)

Added logic to escape literal dotted columns in projections:

• escapeProjectedColumns(): Processes required schema fields
• escapeColumnNameIfNeeded(): Recursively handles nested structures
• Logic: If a column name contains dots and exists as a top-level field in schema, wrap it in backticks
• Nested field access (e.g., parent.child) is handled recursively without escaping the parent

Filter Handling

File: SparkToIcebergExpressionConverter.scala (no changes needed)

Filters correctly use raw field names (no escaping):

• All filter operators use plain column names as-is
• Iceberg's Binder.bind() resolves literal vs. nested using schema
• Server-side binding prioritizes literal field names over nested parsing
• No client-side escaping needed

Test Coverage

Comprehensive Test Suite ✅

1. TestSchemas.scala: Defines schema with BOTH literal dotted columns AND nested structs

   • Literal: address.city, a.b.c, location.state, etc.
   • Nested: address struct with intCol, metadata struct with stringCol

2. SparkToIcebergExpressionConverterSuite.scala:

   • 17 test cases covering all operators with dotted columns
   • Verifies raw field names are used (no backticks in Expression objects)

3. IcebergRESTCatalogPlanningClientSuite.scala:

   • Tests filter/projection sent to REST server
   • Verifies Binder.bind() correctly resolves dotted names
   • Tests both literal and nested dotted columns

4. ServerSidePlannedTableSuite.scala:

   • End-to-end tests with filter and projection pushdown
   • 12 tests covering full query execution

Testing

✅ All unit tests pass:

• 22 iceberg tests (SparkToIcebergExpressionConverterSuite, IcebergRESTCatalogPlanningClientSuite)
• 12 spark tests (ServerSidePlannedTableSuite)
• Total: 34 tests passing
• Scalastyle checks: 0 errors, 0 warnings
• Tested with Java 17

Files Changed

Single file modified:

• ServerSidePlannedTable.scala: Added projection escaping logic (+51 lines)

Total: 1 file changed, 51 insertions(+), 1 deletion(-)