[FEATURE] Enable PPL-Calcite logical plan integration with Spark

[dai-chen]

Is your feature request related to a problem?

Yes. Once the PPL-Calcite library is available in Spark, there is currently no integration to route PPL queries to the new Calcite engine. There is development effort required to get the Calcite engine working within Spark, including integration with Spark’s parser and schema system.

What solution would you like?

Enable logical plan integration between Spark and the PPL-Calcite engine by:

• Implementing a SparkSchema adapter to map Spark’s catalog to Calcite’s schema system.
• Adding a query entry point (e.g., reuse the current legacy PPL parser) to intercept PPL queries and route them to the Calcite engine.
• Introducing a Spark SQL configuration flag (e.g., spark.sql.ppl.calcite.enabled) to toggle this integration.
• Caching schema metadata for both Calcite and SparkSQL planners to improve performance [TBD]

What alternatives have you considered?

• Alternative design are discussed in the related meta issue.

Do you have any additional context?

This task is part of the broader initiative to unify PPL support across OpenSearch and Spark using the new Calcite engine. This issue focuses on enabling the parsing and planning stages in Spark to process PPL queries through Calcite, rather than relying on the legacy PPL engine.