KafkaConnect Sink: Dynamic routing should respect iceberg.table.namespace #14530
Description
Feature Request / Improvement
Hey,
my use case:
debezium cdc into kafka + iceberg kafka connect sink
iceberg kafka connect config:
topics.regex: '^dev.(.*)' --> fetch every topic starting with dev so only one kafka iceberg sink connector which is great
idea:
sink into iceberg.table.namespace + topic name therefore $1 from the matcher of the regex. but currently that's not possible, this would be great if this could be handled.
improvement:
the dynamic feature is somehow great to sink data into the landing zone. However, if we use:
iceberg.tables.dynamic-enabled: "true"
iceberg.tables.route-field: "source_table" --> filed in the record called source_table
it does not respect the iceberg.table.namespace therefore not the schema e.g. "landing" and tries to write without any schema. So only the route-field as table name is being used instead namespace.route-field.
Therefore we need to create a field in the record with the schema in order to route but that's not a good idea to be honest. This feature should be somehow rewritten at least with iceberg.table.namespace being respected or even better with a regex pattern to fetch from the incoming topic name. the record itself should be independent from the target.
I do not see any possibility to do a 1:1 dynamic sink with the kafka connect iceberg sink at the current feature set.
Query engine
None
Willingness to contribute
- I can contribute this improvement/feature independently
- I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- I cannot contribute this improvement/feature at this time