Add support for collated strings in OpConverter

[stefankandic]

Summary

Since its 4.0 release, Spark now supports parametrizing StringType with different collations which define how string data is compared. This PR adds backwards-compatible support for passing collation information in Delta Sharing predicate expressions, with version-specific implementations for Spark 3.5 (Scala 2.12) and Spark 4.0 (Scala 2.13).

Changes

Core Implementation

1. Added ExprContext case class to hold expression metadata, specifically collationIdentifier for string comparisons
2. Extended comparison operations to accept optional exprCtx parameter:
   - EqualOp, LessThanOp, LessThanOrEqualOp, GreaterThanOp, GreaterThanOrEqualOp
   - Also applies to In expressions which are converted to EqualOp chains
3. Created version-specific CollationExtractor implementations:
   - Scala 2.13 (Spark 4.0): Extracts collation information from Spark'sStringType and populates collationIdentifier in format: provider.collationName.icuVersion (e.g., icu.UNICODE_CI.75.1, spark.UTF8_LCASE.75.1)
   - Scala 2.12 (Spark 3.5): Does not create collationIdentifier and instead defaults to UTF8_BINARY comparisons as collations are just a writer feature and delta.
4. Updated OpConverter to:
   - Call CollationExtractor.extractCollationIdentifier() to extract collation information

Backwards Compatibility

• The exprCtx parameter is optional (Option[ExprContext] = None), ensuring existing code continues to work
• The valueType field remains as plain "string" (not "string collate "), maintaining compatibility with older clients
• Collation information is stored separately in ExprContext, allowing non-collation-aware servers to ignore it
• Default UTF8_BINARY collations (non-collated strings) work on both Spark 3.5 and 4.0

Validation

Added safety checks to prevent invalid comparisons:

• Throws IllegalArgumentException when comparing strings with different collations

Protocol Documentation

Updated PROTOCOL.md to document the new exprCtx field and collationIdentifier format with examples.

[linzhou-db]

Is this ready for review?

[stefankandic]

Is this ready for review?

Yes it should be. Although it is still not clear how we can have this change pass the 2.12 checks given that it uses new changes added in Spark 4.0.

[linzhou-db]

Is this ready for review?

Yes it should be. Although it is still not clear how we can have this change pass the 2.12 checks given that it uses new changes added in Spark 4.0.

cc @littlegrasscao

[charlenelyu-db]

This looks good to me and thanks for the nice test coverage! Can you also update the PROTOCOL.md to reflect the new field?

[charlenelyu-db]

nit: can revert

[stefankandic]

We can't actually. Without this change this line was only matching the case object of StringType class, but we want it to match it as well as each instance we create for collated types eg. StringType("UTF8_LCASE")

[chakankardb]

Consider adding this as a comment.

[stefankandic]

Done!

[littlegrasscao]

Is this ready for review?

Yes it should be. Although it is still not clear how we can have this change pass the 2.12 checks given that it uses new changes added in Spark 4.0.

cc @littlegrasscao

If the change only applies to 2.13 not in 2.12, you would need to make 2 copies of the files. 1 in 2.13 folder and apply the new change, 1 in 2.12 folder which still has the old code.

Check out examples like: client/src/main/scala-2.13/org/apache/spark/sql/DeltaSharingScanUtils.scala

[chakankardb]

Can we guard this change via some config/param?

[stefankandic]

We could but since OpConverter does not have a context of SqlConfig we would have to change a whole lot of code and public APIs to pipe this through to here. So I would say that this is not worth doing for this specific change.

[chakankardb]

Consider adding this as a comment.

[charlenelyu-db]

LGTM. Thanks for the change!