Coral-Spark: Migrate 'CAST(ROW: RECORD_TYPE)' and 'extract_union' transformations from RelNode to SqlNode layer

[ljfgem]

Similar to #351, this PR migrates CAST(ROW: RECORD_TYPE) and extract_union transformations from RelNode layer in IRRelToSparkRelTransformer to SqlNode layer.

In order to avoid creating a PR that is too big, I will open several more PRs to finish migrating all the transformations in IRRelToSparkRelTransformer to SqlNode layer and remove IRRelToSparkRelTransformer.

Tests:

1. Existing unit tests, which already cover the changes
2. Regression test
3. Querying several complex production views on gateway, no regression

[aastha25]

thanks @ljfgem for the PR!
Left minor comments around documentation.

[aastha25]

nit: transforms Coral IR function ..... to Spark compatible operator named_struct....

[ljfgem]

Not sure if it's suitable to call it Coral IR function, I think it's a function in CoralSqlNode converted from Coral IR?

[aastha25]

in the path:
source SQL -> sourceSqlNode1 -> CoralSqlNode1 -> CoralRelNode -> CoralSqlNode2 -> TargetLangSqlNode -> target SQL

if an operator is present in CoralSqlNode1, CoralRelNode, CoralSqlNode2 and we transform this operator when we go to LanguageSqlNode, such operators can be called Coral IR functions.

[aastha25]

as per the current status of our code base -
hive SQL -> HiveSqlNode (same as CoralSqlNode1) -> CoralRelNode -> CoralSqlNode2 -> SparkSqlNode -> Spark SQL
I think this operator qualifies as Coral Operator

[ljfgem]

Thanks for clarification, do you mean an operator is a Coral IR function if it appears in CoralSqlNode1 / CoralRelNode / CoralSqlNode2 or CoralSqlNode1 & CoralRelNode & CoralSqlNode2?
Not sure because it doesn't appear in CoralSqlNode1.

[aastha25]

for a corresponding input hive SQL, what is the operator we use in the SqlNode representation? how do we represent the named_struct on LHS?

but generally yes, an operator present in CoralSqlNode1 would be present in CoralRelNode and then generated back in CoralSqlNode2.

[ljfgem]

It's still represented as named_struct in CoralSqlNode1, and Calcite converts it to CAST(ROW: RECORD_TYPE) in RelNode.

[aastha25]

I see. In that case, could you investigate if it's possible to add a lightweight override in CoralRelNodeToCoralSqlNode to such that the coralSqlNode2 generated uses the same operator named_struct as coralSqlNode1?

[ljfgem]

Opened #357 for tracking.

[ljfgem]

@aastha25 Thanks for your review! Modified these 2 comments based on your suggestions. PTAL.