Skip to content

Conversation

@KevinGe00
Copy link
Contributor

@KevinGe00 KevinGe00 commented Oct 30, 2024

What changes are proposed in this pull request, and why are they necessary?

RelNode type derivation on the Coral IR -> Spark SQL path was introduced in #507 in ExtractUnionFunctionTransformer here. The type derivation requires calling Calcite's validate function which was incorrectly dropping the LATERAL keyword, this was previously only fixed for ... LATERAL (SELECT ..... case but has since been fixed for .. LATERAL (UNNEST ... here.

This PR bumps the Calcite version to include the fix and adds a new unit to ensure LATERALs are not dropped for views with the Coral IR -> Spark SQL type derivation (through ExtractUnionFunctionTransformer).

How was this patch tested?

  • ./gradlew clean build
  • new unit test passing when using newest Calcite with fix
  • no regressions when testing on production views
    • since certain views that were already failing due to this issue, their translation wouldn't be tested in the regression pipeline (since the pipeline is designed is test regressions, we take only the views with valid translations on the coral master branch and run translations on that subset of views only with the feature coral branch), ran adhoc regression tests with those failing views and they now succeed with these changes.

@KevinGe00 KevinGe00 marked this pull request as ready for review November 12, 2024 21:03
@KevinGe00 KevinGe00 changed the title [WIP DO NOT MERGE] Bump Calcite to preserve LATERALs in LATERAL UNNEST calls Bump Calcite to preserve LATERALs in LATERAL UNNEST calls Nov 12, 2024
@KevinGe00 KevinGe00 merged commit 15fc504 into linkedin:master Nov 14, 2024
1 check passed
simbadzina added a commit to simbadzina/coral that referenced this pull request Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants