[Feature] Apache Flink Catalog Integration

[carlesarnal]

Parent Epic

Part of #6993 - Data Platform Integration

Description

Implement Apache Flink Catalog integration, enabling Flink SQL jobs to resolve table schemas directly from Apicurio Registry.

Background

Apache Flink is the de facto standard for stateful stream processing. Flink SQL users need schema resolution for:

• Table definitions in Flink SQL
• State schema evolution
• Consistency between Kafka sources and Flink processing

Requirements

Flink Catalog Provider

1. Catalog Implementation

   • Implement Flink's Catalog interface
   • Map Apicurio groups to Flink databases
   • Map Apicurio artifacts to Flink tables

2. Schema Resolution

   • Resolve Avro schemas to Flink row types
   • Resolve JSON Schema to Flink row types
   • Handle nested structures and arrays

3. Configuration

   • Catalog factory for SQL CLI configuration
   • Connection settings (URL, auth)
   • Caching configuration

State Schema Evolution

1. State Schema Management

   • Register Flink state schemas
   • Track state schema versions
   • Compatibility checking for state evolution

2. Savepoint Compatibility

   • Validate schema changes against savepoint state
   • Detect incompatible state evolution

Integration Points

• Flink SQL DDL integration
• Kafka connector schema resolution
• Table API schema lookup

Acceptance Criteria

• Flink Catalog connects to Apicurio Registry
• Flink SQL can CREATE TABLE using registry schemas
• State schemas are versioned in registry
• Documentation covers Flink integration setup

Technical Considerations

• Flink Catalog SPI requirements
• Type mapping between Avro/JSON and Flink types
• Handle Flink's watermark and timestamp columns

References

• Flink Catalogs
• Flink Schema Evolution
• Flink Avro Format