Skip to content

Rethink KafkaProducer/KafkaProducerConnection #572

@bplommer

Description

@bplommer

I’ve been thinking about how we can simplify the most common use cases.

KafkaProducerConnection really represents an untyped Kafka producer - it easily could, and possibly should, have a method produce[P, K: Deserializer, V: Deserializer](producerRecords: ProducerRecords[P, K, V]). KafkaProducer is really nothing more than a partial application of that general produce method.

I suggest that in 3.0 we rename KafkaProducerConnection to KafkaProducer and KafkaProducer to SerializingKafkaProducer.

I also suggest we add KafkaTopicProducer, which has both serializers and a topic name provided on instantiation and takes just key-value pairs rather than ProducerRecords as arguments for its produce method. This would simplify calling code in, I think, the great majority of use cases.

Further, there could be a variant that automatically derives keys from values (typically the key would be the same as some kind of id field from the value), so that calling code only needs to provide the value and not the key.

The main drawback I see is that the resulting number of variants could cause confusion, but I think careful API design would mitigate that.

Any thoughts? @vlovgr @LMnet

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions