[Data] Added a SQL API

[soffer-anyscale]

Why are these changes needed?

This PR introduces a comprehensive SQL API for Ray Datasets, enabling users to execute standard SQL queries on distributed data using Ray's parallel processing capabilities. This addresses a significant user need for familiar SQL syntax when working with large-scale distributed datasets.

Key benefits:

• Familiar Interface: Allows data analysts and engineers to use standard SQL syntax with Ray Datasets
• Distributed Execution: Leverages Ray's distributed computing for scalable SQL query processing
• Seamless Integration: SQL results return as Ray Datasets, maintaining full compatibility with existing Ray Data workflows
• Production Ready: Includes comprehensive error handling, configuration options, and optimization features

Core Features:

• Standard SQL operations: SELECT, WHERE, JOIN, GROUP BY, ORDER BY, LIMIT
• Aggregate functions: COUNT, SUM, AVG, MIN, MAX, STD
• String functions: UPPER, LOWER
• All join types: INNER, LEFT, RIGHT, FULL OUTER
• Automatic table registration and schema inference
• Query optimization using SQLGlot
• Integration with Ray's DataContext for configuration

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

---

Note

Introduce an experimental Ray Data SQL API with execution engine, SQLGlot/DataFusion optimization, configuration, comprehensive docs, tests, and dependency updates.

• Data/SQL (experimental):
  • Add new SQL engine under python/ray/data/experimental/sql/* (parser/compiler via SQLGlot, execution for SELECT/WHERE/JOIN/GROUP BY/ORDER BY/LIMIT, aggregates, validators, registry/schema, exceptions).
  • Integrate optional DataFusion optimizer (engines/datafusion/*) with hint-driven execution.
  • Provide user-facing API in experimental/sql_api.py (sql, register, list_tables, clear_tables) and config via DataContext (sql_* and sql_use_datafusion).
• Docs:
  • Add SQL docs: doc/source/data/sql*.rst, API reference doc/source/data/api/sql.rst; update navigation and landing pages.
• Tests/Build:
  • Add SQL tests (test_sql_api.py, test_sql_datafusion.py, test_sql_optimizers.py) and BUILD targets.
• Deps:
  • Add sqlglot to requirements.

^{Written by Cursor Bugbot for commit 7b68d1f. This will update automatically on new commits. Configure here.}

[gemini-code-assist]

Summary of Changes

Hello @soffer-anyscale, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request delivers a significant new feature to Ray Data: a native SQL interface. It enables users to interact with distributed Ray Datasets using familiar SQL syntax, translating SQL queries into efficient Ray Data operations. The implementation follows Ray Data's lazy evaluation patterns and integrates deeply with the existing API, providing a powerful and scalable tool for data manipulation.

Highlights

• New SQL API for Ray Datasets: This PR introduces a comprehensive SQL API, ray.data.sql, allowing users to execute standard SQL queries directly on Ray Datasets. This includes support for SELECT, WHERE, JOIN, GROUP BY, ORDER BY, LIMIT, and various aggregate and string functions.
• Distributed Execution & Integration: The SQL API leverages Ray's distributed computing capabilities for scalable query processing. Results are returned as Ray Datasets, ensuring seamless integration with existing Ray Data workflows and allowing for method chaining with other Ray Data operations.
• Architecture & Components: A new ray.data.sql package is added, comprising several key components: a SQLParser (using SQLGlot), ASTOptimizer and LogicalPlanner for query optimization and planning, a QueryExecutor that applies Ray Data operations, and a DatasetRegistry for managing registered tables and their schemas. An ExpressionCompiler converts SQL expressions into Python callables.
• Automatic Table Registration & Schema Inference: The API supports automatic table registration, where Ray Datasets in the caller's scope can be recognized by their variable names. It also includes automatic schema inference from registered datasets.
• Testing and Examples: Comprehensive unit tests (test_sql_api.py) and a dedicated testing.py module with TestRunner and ExampleRunner are included to validate the engine's functionality and demonstrate its usage with various SQL scenarios.
• Dependency Update: The sqlglot library (version 27.4.1) is added as a new dependency to python/requirements/ml/data-test-requirements.txt to power the SQL parsing and optimization.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is an excellent and comprehensive pull request that introduces a powerful SQL API for Ray Datasets. The architecture is well-thought-out, with a clear separation of concerns for parsing, optimization, and execution. The integration with sqlglot is a solid choice.

I've found a few issues that should be addressed before merging. Most notably, there is a critical bug in the LIMIT clause implementation that needs to be fixed. I've also identified a high-severity issue in the validation logic for aggregate functions and several medium-severity issues related to API design, maintainability, and robustness. My detailed comments and suggestions should help in polishing this feature for its release.