Introducing new KafkaRoller

[tinaselenge]

POC implementation

[fvaleri]

Just a first pass, as I need more time to digest this. I think it would be useful to illustrate the new behavior with a couple of examples of the form: with this roller configuration and cluster state, these are the node groups and their restart order. Wdyt?

[tinaselenge]

@fvaleri Thank you for the feedback. I have added an example of rolling update. Please let me know what you think.

[see-quick]

Nice proposal. Thanks for it 👍 .

---

STs POV:

I think we would need to also design multiple tests to cover all states, which KafkaRoller v2. We have a few tests but for sure that's not 100% coverage. So, we should maybe have a meeting to talk about this...

Side note about performance:

What would be appropriate performance metrics for us to consider when designing performance tests? Are there any critical ones? For sure I can image that we would see significant improvement on RollingUpdates of multiple nodes when we use batching mechanism...

[fvaleri]

@tinaselenge thanks for the example, it really helps.

I left some comments, let me know if something is not clear or you want to discuss further.

[fvaleri]

In addition to rebalance, we have the same race condition with replication factor change (the new integration between CC and TO), maybe you can mention this.

The roller should be able to call the CC's user_tasks endpoint, and check if there is any pending task. In that case, the roller has two options: wait for all tasks completion, or continue as today with the potential issue you describe here. You can't really stop the tasks because the current batch will still be completed, and the operators will try to submit a new task in the next reconciliation loop.

I think that we should let the user decide which policy to apply through a configuration. By default the roller would wait for all CC tasks to complete, logging a warning. If the user set or switch to "force" policy, then the roller would behave like today. Wdyt?

[tinaselenge]

Should this be perhaps included/discussed in a separate proposal or issue? The idea was to mention that there is a race condition we could fix with the new roller in the future, which is not easy to fix with the old roller. How we fix it and other similar problems should be a separate discussion I think.

[fvaleri]

This should have a dedicated proposal IMO, but let's start by logging an issue.

[stanislavkozlovski]

Would calling the ListReassigningPartitions API be enough to know this?

[katheris]

Overall this looks good to me, but I had a few questions and wording suggestions. I definitely think this will be useful since I've experienced first hand how tricky it is to debug the existing code.

[fvaleri]

Thanks for answering all my questions. Good job.

[tinaselenge]

Thanks for answering all my questions. Good job.

Thank you @fvaleri , I really appreciate you reviewing the proposal thoroughly.

[tombentley]

In general Slack is not really ideal for keeping details of problems in the long term. Better to create an issue, which can be discovered more easily by anyone who faces a similar problem.

[tinaselenge]

Hi @tombentley @scholzj @ppatierno, do you have any further comments on this proposal?

[tinaselenge]

Converting this PR to Draft, as I'm currently rewriting the proposal to make it more focused on the proposed solutions, rather than implementation details. However, I'm not changing the core of the proposal. I'm hoping that the upcoming update will make it easier to review.

[rotem-human]

Any updates on this?

[tinaselenge]

Hi @strimzi/maintainers. I have made some updates to the proposal recently, to hopefully make it easier to review. The proposal should be now focusing on the design only and all the implementation details are in the linked POC PR. You can build and run the POC locally, to see if how it works.

Before I invest more time and effort into this, I would like to get a clear decision on the next steps for this proposal since there have been some mixed opinions.

• Do you see this proposal being accepted and implemented now or in the future? Or should we close this proposal for now?
• Is there a maintainer/s who would be willing to support this proposal through the implementation if it does get accepted?

I also have a question to ask the community users and other contributors:
If we were to go ahead with this proposal, is there community users and contributors who are willing to help with the implementation part or able to test this feature in their environments while it's still behind a feature gate. I think testing it in their environment and providing feedback would be the most important contribution for this feature. This also may weigh into the maintainer's decision as well. So please let us know if you are interested in or need this feature for your use case.

[katheris]

• Do you see this proposal being accepted and implemented now or in the future? Or should we close this proposal for now?
• Is there a maintainer/s who would be willing to support this proposal through the implementation if it does get accepted?

My stance on this hasn't changed. I believe this is a useful feature, both in terms of the benefits to users around batching the rolling updates for large clusters, and in terms of improving the KafkaRoller code to make it easier to understand. I don't personally have the time to work on the implementation of this, but am willing to put myself forward as a maintainer who will prioritise reviewing the changes and understanding the code for future maintenance once it's in Strimzi.

I'm also interested to hear what other maintainers and users think about the usefulness of this feature, so let's also add it to the Strimzi community call agenda late today (17th April).

[ppatierno]

After three weeks from the community call where we discussed this, there weren't any additional reviews or comments here (apart from the Kate's one, but before the community call).
I had a quick look at the proposal but mostly the first part describing known issues affecting the current Kafka roller.
My current feeling is:

• despite some known issues, I can't see more and more community users complaining about them. Are they really issues or are we overthinking here?
• the additional features it can bring (rolling brokers in batches, slowing down rolling update) are still not pushed more from the community. So even in this case, how much are they important/requested?

The two above will be the big benefits we are getting by using a new Kafka Roller but without a real need, I have the concerns about testing and have it working to win against the advantages we can take.
From my pov, we should have community users showing interest in it. For example, the referenced issue #8547 was started by @yyang48 and it would be interesting to know if there is still interest and willing to help with it.

Said that, from the community call it seems that we have got:

• @tinaselenge to work on the implementation
• @katheris as the core maintainer willing to help and review the changes
• @im-konge helping with testing it

[im-konge]

Thanks a lot for the proposal, I went through it and it LGTM (I left some minor comments). I would like to go through the PoC and possibly try it to see how it works. In terms of testing, we should have a look on writing more STs now, as IIRC there are just few for the old roller and it would be good to have them improved. Also we discussed with @see-quick that it would be maybe beneficial to look at the performance tests in the future (but it's not something urgent right now).

If the current code is too complex and the new KafkaRoller would make the code clearer + other code changes or new features would be much simpler to implement, it's good to have it there FMPOV.

Also, you mentioned various things that will be implemented -> from my understanding you expect and want to have them all in place with the initial implementation, right? Or you want to do 1:1 KafkaRoller in comparison with the old one and then add the new features?

[im-konge]

Suggested change

	| Pod is Running but lacking Ready stats | Broker state == 2 | RECOVERING
	| Pod is Running but lacking Ready status | Broker state == 2 | RECOVERING

[im-konge]

Because Strimzi doesn't support Zookeeper anymore, are these two lines needed? Just asking out of curiosity, as I know that the proposal was written in the moment when we supported both.

[im-konge]

We should change it to 0.47/0.48 or something. It's just a reminder.