Implement python serdes

[carlesarnal]

Summary

• Implemented Python serializers/deserializers (SerDes) for Apicurio Registry supporting Avro, JSON Schema, and Protobuf
• Added Kafka integration wrappers compatible with the kafka-python library
• Created comprehensive test suite with 86 passing tests
• Added full documentation in the Apicurio Registry docs

Root Cause

Apicurio Registry lacked Python client support for serialization/deserialization, limiting adoption in Python-based Kafka applications. This implementation provides parity with the existing Java
SerDes functionality.

Changes

Core SerDes Implementation (python-serdes/apicurio_registry_serdes/)

• config/serde_config.py, config/id_option.py: Configuration classes supporting registry URL, authentication, caching, retry logic, and Kafka-style config keys
• schema_parser.py: Abstract base class for schema parsers with parse_schema(), serialize_schema(), and get_schema_from_data() methods
• schema_resolver.py: Schema resolver handling registry lookups by ID (content/global), artifact coordinates (group/artifact/version), auto-registration, and TTL-based caching
• serde_base.py: Base serializer/deserializer classes with wire format handling (magic byte + 4-byte schema ID + payload)
• avro.py: Avro SerDes using fastavro with schema evolution support (reader/writer schemas)
• jsonschema.py: JSON Schema SerDes using jsonschema library with optional validation
• protobuf.py: Protobuf SerDes supporting both typed and dynamic deserialization via DynamicProtobufDeserializer

Kafka Integration (python-serdes/apicurio_registry_serdes/kafka/)

• avro.py: KafkaAvroSerializer, KafkaAvroDeserializer - kafka-python compatible wrappers
• jsonschema.py: KafkaJsonSchemaSerializer, KafkaJsonSchemaDeserializer
• protobuf.py: KafkaProtobufSerializer, KafkaProtobufDeserializer
• init.py: Lazy imports to handle optional dependencies gracefully

Package Configuration

• pyproject.toml: Poetry-based project with optional extras ([kafka], [protobuf], [jsonschema], [all])
• init.py: Public API exports with lazy loading for optional serializers

Tests (python-serdes/tests/)

• test_config.py: Configuration and validation tests
• test_serde_base.py: Base class and wire format tests
• test_avro.py: Avro serialization/deserialization tests
• test_jsonschema.py: JSON Schema tests including validation
• test_kafka.py: Kafka wrapper compatibility tests

Documentation

• docs/modules/ROOT/pages/getting-started/assembly-using-python-client-serdes.adoc:
  • Installation instructions with optional dependencies
  • Configuration guide with authentication options
  • Usage examples for Avro, JSON Schema, and Protobuf
  • Kafka producer/consumer integration examples
  • Known limitations section (schema references)
  • Complete configuration reference

Key Features

• Wire format compatibility: Uses same format as Java SerDes (magic byte + schema ID) for interoperability
• Schema ID options: Supports both content ID and global ID for schema identification
• Auto-registration: Optionally register schemas on first use
• Caching: TTL-based schema caching to reduce registry calls
• Authentication: Basic auth and bearer token support
• kafka-python integration: Callable interface compatible with value_serializer/value_deserializer

Known Limitations

• Schema references (Avro references, JSON Schema $ref, Protobuf imports) require registry-side dereferencing via dereference_schema=True

Test plan

• Run unit tests: cd python-serdes && pytest (86 tests passing)
• Integration test with running Apicurio Registry instance
• Test interoperability with Java SerDes (produce in Python, consume in Java and vice versa)
• Verify documentation renders correctly