S3 Sink: org.apache.avro.SchemaParseException: Illegal character in

[sbuliarca]

Hi!

We're doing a backup of all topics using Parquet and 'store.envelope'=true to store also the headers. Here are the details:

Stream-reactor-version: 8.1.30
Kafka connect version: based on Docker image confluentinc/cp-kafka-connect:7.8.0
Backup config:

{
"connector.class": "io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector",
"tasks.max": "10",
"connect.s3.aws.auth.mode": "Default",
"connect.s3.compression.codec": "zstd",
"topics.regex": ".*",
"key.converter.schemas.enable": "false",
"connect.s3.kcql": "insert into msk-backup:msk-backup-parquet select * from * PARTITIONBY _topic, _partition STOREAS PARQUET PROPERTIES ('store.envelope'=true,'flush.count'=1000,'flush.interval'=300, 'flush.size'=50000000, 'partition.include.keys'=false);",
"name": "msk-s3-backup-sink-all-parquet",
"value.converter.schemas.enable": "false",
"connect.s3.compression.level": "9",
"connect.s3.aws.region": "eu-west-1",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter"
}

The Problem:
When encountering a message with the following headers:

{"traceparent":"00-fe5798fdc39c42a21010f92fea8a85fc-dbb78d479a0b3099-01","reason-for-dlq":"test from actionTestDlq","original-topic":"beb-dlq-test"}

if fails with the error:

org.apache.avro.SchemaParseException: Illegal character in: original-topic

Checked also on latest version 10.0.0 and the same issue happens

[stheppi]

Hi @sbuliarca ,

it is a feature request we are aware of. When storing the Kafka message, the parquet record needs to have a schema. The incoming headers name don't adhere to the Parquet schema field name requirements. The solution requires the code to rename the headers as it persists them. An alternative for now is to use an SMT (Single Message Transform) to change the header name before the Kafka message is sent, by the Connect framework, to the sink to store it.

[sbuliarca]

Hey @stheppi !

Thank you for the suggestion!

I just want to point out that this issue needs to be considered on restore too. As we want to reconstruct fully the original message. So if we replace - with _ on backup, on the headers, we need to do the reverse on restore. This works only if we don't use the _ char in other headers. My point is that this solution boils down to "don't use this character in the headers".

So, currently the workaround for us is to not use "-" in the headers. Maybe this could be documented as a limitation in the Sink.