Description
Is your feature request related to a problem?
Yes. We need to incorporate the newly developed PPL-Calcite library into the Spark runtime so that Spark can execute PPL queries using the Calcite engine. Without this integration, Spark continues to rely on the legacy PPL engine, which creates maintenance overhead and limits future enhancements.
What solution would you like?
Integrate the PPL-Calcite library into Spark by:
- Creating a dedicated feature branch for the integration [TBD].
- Moving legacy PPL-related classes in Spark to prevent class name conflicts with the library.
- Adding the PPL-Calcite library to Spark via SBT dependency.
- Ensure it builds and runs correctly within the Spark environment.
What alternatives have you considered?
N/A
Do you have any additional context?
This task is part of a broader initiative to enable unified query execution via Calcite across OpenSearch and Spark. The PPL-Calcite library must first be published and made compatible with Spark’s runtime. The goal of this issue is to enable internal use of the library without introducing build-time or runtime conflicts.