feat: add QuixDatalakeSink for writing Kafka data to blob storage in …

[ovv]

…Avro format

• Introduce QuixDatalakeSink and QuixDatalakeSinkConfig for persisting raw Kafka topic data to blob storage (S3, ADLS, GCS, etc.) using fsspec.
• Store data in Avro format with Hive-style partitioning and maintain Parquet metadata indexes for efficient querying.
• Add integration with Quix Data Catalog for cache refresh.
• Update pyproject.toml with new dependencies for datalake support.
• Add comprehensive tests for sink functionality, including partitioning, headers, multiple flushes, and empty flushes.

[Copilot]

Pull Request Overview

This PR introduces a new sink, QuixDatalakeSink, for writing raw Kafka topic data in Avro format to blob storage with Hive-style partitioning and Parquet metadata, along with integration for refreshing a Quix Data Catalog cache. It also adds supporting tests and updates project dependencies.

• Add QuixDatalakeSink and QuixDatalakeSinkConfig, implement Avro file writing, Parquet metadata indexing, and optional Data Catalog notification
• Provide a dummy fsspec filesystem and comprehensive tests covering partitioning, headers, multiple records, and multiple flushes
• Update pyproject.toml to include new datalake dependencies and import ignores

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
quixstreams/sinks/core/datalake/sink.py	Implements QuixDatalakeSink, configuration, Avro/Parquet I/O, and Data Catalog notification
tests/test_quixstreams/test_sinks/test_core/test_datalake_sink.py	Adds end-to-end tests for basic writes, partition handling, headers, multi-record, and flush logic
quixstreams/sinks/core/datalake/init.py	Exports QuixDatalakeSink and its config in the package
quixstreams/sinks/init.py	Registers new sink and config in the main sinks module
pyproject.toml	Adds datalake extra requirements and mypy ignores for fsspec

Comments suppressed due to low confidence (3)

quixstreams/sinks/core/datalake/sink.py:139

• [nitpick] The list comprehension unpacks header tuples using k, which shadows the outer variable k for the record key; use distinct names (e.g., header_key, header_value) to avoid confusion.

                "headers": [{"key": k, "value": v} for k, v in headers],

quixstreams/sinks/core/datalake/sink.py:238

• There are no tests covering the Data Catalog notification logic; consider adding tests for scenarios when datacatalog_api_url is set and when the notification succeeds or fails.

    def _notify_datacatalog(self, topic: str) -> None:

quixstreams/sinks/core/datalake/sink.py:104

• [nitpick] Private methods like _get_protocol lack docstrings; consider adding brief docstrings to explain their purpose and parameters.

    def _get_protocol(self, url: str) -> str: