Can we add at-most-once semantic to producer?

[laststem]

Basically, StreamProducer provide at-least-once semantic.

but, we need a at-most-once semantic.

and we can't use deduplication because we don't have a proper publishingId.

[acogoluegnes]

Under what circumstances did you observe duplicate messages?

[laststem]

In docs, RabbitMQ Stream provides publisher confirms to avoid losing messages: once the broker has persisted a message it sends a confirmation for this message. But this can lead to duplicate messages: imagine the connection closes because of a network glitch after the message has been persisted but before the confirmation reaches the producer. Once reconnected, the producer will retry to send the same message, as it never received the confirmation. So the message will be persisted twice.

This question is a curiosity that arose after reading the document.

It was difficult to determine whether duplicate messages were observed, as network errors are difficult to reproduce on localhost.

Also, it seems like the above theory might actually happen in this code.

[michaelklishin]

Message Deduplication with RabbitMQ Streams has a section called Publishing with Deduplication.

Duplicates due to retries in case of failure is a fact of life in distributed systems and cannot be avoided for every failure scenario, in particular with N clients connected to M nodes.

[michaelklishin]

but, we need a at-most-once semantic

and we need specific details, not single line descriptions.

[michaelklishin]

A 4-5 sentence-long description above is already an improvement over this one liner.

Message Deduplication with RabbitMQ Streams has a section called Publishing with Deduplication.

[acogoluegnes]

@laststem It is possible to have at-most-once semantics by ignoring publish confirms (easy) and deactivating retry attempts after a connection failure (retry is mandatory currently). It would be possible to introduce an option to have such a "fire-and-forget" behavior (publish confirms at the protocol level would still arrive, but the client would ignore them). Still, it implies at-most-once semantics, so some messages may never reach the broker.

[laststem]

deactivating retry attempts after a connection failure

Does this mean https://github.com/rabbitmq/rabbitmq-stream-java-client/blob/main/src/main/java/com/rabbitmq/stream/impl/StreamProducer.java#L505 ?

Well, that's what I wanted to say.

Could I contribute something that would allow the client to opt-in to this re-publishing? If not, could you add this feature?

[acogoluegnes]

Yes, this would mean clearing the unconfirmedMessages map after the producer recovery, instead of trying to republish the messages it contains.

Could I contribute something that would allow the client to opt-in to this re-publishing?

Sure, contributions are welcome.