Skip to content

Conversation

Awethon
Copy link
Contributor

@Awethon Awethon commented May 12, 2025

This was introduced due to the default behavior of Kafka prior to version 3.0. However, it now causes more harm than good by preventing fail-safe behavior during rebalancing, as NotLeaderOrFollowerException is not retried.

Since Kafka 3.0, the default is enable.idempotence=true, which ensures safe behavior according to both the SoftwareMill blog (mentioned in producers.md) and the Kafka docs.

Allowing retries while setting enable.idempotence to false and max.in.flight.requests.per.connection to greater than 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.

@Awethon
Copy link
Contributor Author

Awethon commented May 12, 2025

After reading more kafka code, I think this override is even more harmful than I previously thought as it disables idempotence and makes kafka producer less reliable:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L580-L593

@Awethon
Copy link
Contributor Author

Awethon commented May 21, 2025

@aartigao can we get this merged please?

@Awethon
Copy link
Contributor Author

Awethon commented May 26, 2025

@vlovgr maybe you can take a look if possible? 🙏 Can't find any other active maintainer.

@vlovgr
Copy link
Contributor

vlovgr commented May 26, 2025

This looks great @Awethon, thanks!

@vlovgr vlovgr merged commit fc1068c into fd4s:series/3.x May 26, 2025
8 checks passed
@sideeffffect
Copy link

Thank you so much @vlovgr can you also make a new release with this fix?

@vlovgr
Copy link
Contributor

vlovgr commented May 26, 2025

@sideeffffect I've released v3.8.0 with these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants