Apicurio migration ReflectionAvroSerializer

[smeana]

Hello

I am testing up a possible migration from schema registry to Apicurio.
I cannot find the equivalent to io.confluent.kafka.streams.serdes.avro.ReflectionAvroSerializer
In case it is not implemented yet, is any straight forward way using Quarkus to serialize Avro without generating Avro friendly classes from an Avro schema?

Regards

[famarting]

Hi, we have that feature.

Check here https://www.apicur.io/registry/docs/apicurio-registry/2.1.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-avro_registry and look for apicurio.registry.avro-datum-provider

[famarting]

thank you @smeana for using Apicurio Registry. I think your last issue about Unknown magic byte issue could be fixed by adding the property apicurio.registry.as-confluent=true to both your producers and consumers.

Apicurio uses 8 bytes to store the identifier in the kafka message body, while Confluent serdes use 4 bytes. We have the property apicurio.registry.as-confluent in our serdes to align that behavior.

[smeana]

Many thanks @famartinrh
with the following config we still have the magic byte issue:

    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.8.0.1:9092");
    props.put(ProducerConfig.CLIENT_ID_CONFIG, "test-producer");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, AvroKafkaSerializer.class.getName());
    props.putIfAbsent(ProducerConfig.ACKS_CONFIG, "all");
    props.putIfAbsent(SerdeConfig.REGISTRY_URL, "http://10.8.0.1:8080/apis/registry/v2");
    props.putIfAbsent(SerdeConfig.AUTO_REGISTER_ARTIFACT, Boolean.TRUE);
    props.putIfAbsent(SerdeConfig.ENABLE_CONFLUENT_ID_HANDLER, Boolean.TRUE);
    props.putIfAbsent(AvroKafkaSerdeConfig.AVRO_DATUM_PROVIDER, ReflectAvroDatumProvider.class.getName());

The Consumer is pointing to http://10.8.0.1:8080/apis/ccompat/v6 and using confluent avro deserializer

[famarting]

hmm that configuration should work I think... @smeana is it possible now you are getting that error from old messages when you started testing Apicurio? that could explain it... otherwise we would need to look more in depth into it...

[smeana]

Hello @famartinrh we tested it on new topics. I am trying to figure out what can be the problem.
regards

[EricWittmann]

Could you perhaps use e.g. kafkacat to validate that the magic byte is correct on the messages in the topic? With the above config, the first byte of the message body should be 0x0 and then the next 4 bytes should be the schema's globalId.