Support deploying AutoMQ with the Strimzi operator

[josephlim75]

I understand that running AutoMQ in Kubernetes requires a Business/Enterprise license and is not available for the Community edition. My question is, if we use the existing container image or rebuild our own AutoMQ image and manage the container deployment ourselves in a Kubernetes cluster, is this permitted under the Community version?

[daniel-y]

Hi @josephlim75, some of our customers are already using K8s or Strimzi to deploy the AutoMQ Community Edition. You're welcome to do the same. If you need any support from our team, we're happy to help and can set up a WeChat group or Slack channel for you.

[stlava]

This might be a bit specific but how are people resolving the version conflicts between the automq dist tar and strimzi when packaging automq via the strimzi build? I'm seeing version conflicts with opentelemetry, jackson, and guava.

Edit: I ended up updating the strimzi deps to match automq. Haven't run the strimzi integration tests but I was able to produce a kafka-base docker image successfully.

[mnuttall]

Hi @daniel-y in this commit three days ago you write that Kubernetes deployment is 'coming soon' - is there a branch or issue number that we can see that work being done under, please?

[stlava]

Just a side note I was able to get automq built and deployed via strimzi. Here's my branch which updates the strimzi deps to match automq. I also left some instructions on how to build the docker image.

stlava/strimzi-kafka-operator@2d4086c#commitcomment-156305711

[daniel-y]

@mnuttall I guess we should create an issue to track K8s developments. Our current idea is to adapt AutoMQ to Bitnami or Strimzi directly, so the AutoMQ community won't need to maintain a chart or operator.

[daniel-y]

Just a side note I was able to get automq built and deployed via strimzi. Here's my branch which updates the strimzi deps to match automq. I also left some instructions on how to build the docker image.

stlava/strimzi-kafka-operator@2d4086c#commitcomment-156305711

Sounds great. So, we just need to resolve the dependency issues, and then Strimzi can manage the AutoMQ cluster, right? If so, is the next step to merge the code into Strimzi?

[stlava]

So there are two cases that make this potentially hard to resolve:

1. automq shares common dep with strimzi and is ahead
2. automq shares common dep with kafka and is ahead

Examples of these:

1. automq has a newer version than kafka eg build(dependencies): update Jackson to 2.17.1 #1963
2. automq has a different version than strimzi eg 781b212#diff-9a3ae364f6836158ed7f4b32cb24c8252c8a4a7dc0cfb0b85dd9aa20bdd06c01R99-R100

When strimzi packages things up it checks to see if there are any overlapping classes between different jars. They do have a mechanism to allow this overlap but I'm not sure how safe that is to do with something like jackson. The end result is both jars are included and jvm loader would decide which gets loaded.

That all being said, I think the purpose of an automq operator is to take advantage of automq's ability to scale quickly to keep up with load where as strimzi is really going to be for steady state management where you're not dynamically scaling the cluster.

[daniel-y]

That all being said, I think the purpose of an automq operator is to take advantage of automq's ability to scale quickly to keep up with load where as strimzi is really going to be for steady state management where you're not dynamically scaling the cluster.

So you think using Strimzi to manage the AutoMQ cluster isn't a good idea, right? @stlava

[stlava]

So you think using Strimzi to manage the AutoMQ cluster isn't a good idea, right?

I think the answer is a bit nuanced. Strimzi keeps Kafka stable and running healthy either during intended changes (upgrades, manual scaling) or unintended (hardware failure, k8s upgrades, k8s disruptions, etc). A lot of the complexity in management of these are through the lens that Kafka nodes are stateful and the cluster needs to be babied.

Most of that probably doesn't completely go away with AutoMQ because you still need some graceful orchestration. However, Strimzi hasn't built up a muscle to dynamically scale Kafka because it has not been a thing one does. Is auto scaling important to everyone/anyone, maybe? However, I personally wouldn't over index on that when Strimzi handles most of what one needs operationally and all the other stuff like ACLs, topics, users, authn/authz, cruise control, etc in a kubernetes native way.

My recommendation would be to use Strimzi as the base and then layer or extend it to handle AutoMQ's strengths/abilities for those who need that functionality.

[daniel-y]

@stlava Thanks for your professional advice, I have changed the title of this issue and let's dedicate it to discuss AutoMQ and Strimzi, Also, I have created another issue to track our progress to provide a Bitnami-compatible image for AutoMQ: #2541

[1sonofqiu]

Hi all, We recently completed the adaptation for Bitnami and are excited to announce the release of AutoMQ v1.5.0, which fully integrates with Bitnami Helm Charts. Here are the relevant demo and documentation to help you deploy it on Kubernetes.
Feel free to give it a try and let us know your feedback!

[1sonofqiu]

Sorry for the wait! Following AutoMQ's support for Bitnami deployment, we've added Strimzi feature support in version 1.6.0-rc0. You can now quickly deploy with the Strimzi Kafka Operator via automq-labs demo – we look forward to your feedback!