Skip to content

Commit 80f778e

Browse files
aswinayyolathAswin A
aswinayyolath
authored and
Aswin A
committed
This proposal describes design details of stretch cluster
Signed-off-by: Aswin A <[email protected]>
1 parent f81b6cd commit 80f778e

File tree

1 file changed

+155
-0
lines changed

1 file changed

+155
-0
lines changed

‎083-stretch-cluster.md

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Stretch Kafka cluster
2+
3+
The Strimzi Kafka operator currently manages Kafka clusters within a single Kubernetes environment. This proposal aims to extend support to stretch Kafka clusters, where brokers and controllers are distributed across multiple Kubernetes clusters.
4+
5+
## Current situation
6+
7+
At present, the availability of Strimzi-managed Kafka clusters is directly tied to the availability of the underlying Kubernetes cluster. If a Kubernetes cluster experiences an outage, the entire Kafka cluster becomes unavailable, disrupting all connected Kafka clients.
8+
9+
## Motivation
10+
11+
By distributing Kafka nodes across multiple clusters, a stretch Kafka cluster can tolerate outages of individual Kubernetes clusters and will continue to serve clients seamlessly even if one of the clusters goes down.
12+
13+
## Proposal
14+
15+
This proposal seeks to enhance the Strimzi Kafka operator to support stretch Kafka clusters, distributing brokers and controllers across multiple Kubernetes clusters. The intent is to focus on high-availability of the data plane. The proposal outlines high-level topology and design concepts for such deployments, with a plan to incrementally include finer design and implementation details for various aspects.
16+
17+
### Prerequisites
18+
19+
- **Multiple Kubernetes Clusters**: Stretch Kafka clusters will require multiple Kubernetes clusters. Ideally, an odd number of clusters (at least three) is needed to maintain quorum in the event of a cluster outage.
20+
21+
- **Low Latency**: Kafka clusters should be deployed in environments that allow low-latency communication between Kafka brokers and controllers. Stretch Kafka clusters should be deployed in environments such as data centers or availability zones within a single region, and not across distant regions where high latency could impair performance.
22+
23+
- **KRaft**: As Kafka and Strimzi transition towards KRaft-based clusters, this proposal focuses exclusively on enabling stretch deployments for KRaft-based Kafka clusters. While Zookeeper-based deployments are still supported, they are outside the scope of this proposal.
24+
25+
### Design
26+
27+
The cluster operator will be deployed in all Kubernetes clusters and will manage Kafka brokers/controllers running on that cluster. One Kubernetes cluster will act as the control point for defining custom resources (Kafka, KafkaNodePool) required for stretch Kafka cluster. The KafkaNodePool custom resource will be extended to include information about a Kubernetes cluster where the pool should be deployed. The cluster operator will create necessary resources (StrimziPodSets, services etc.) on the target clusters specified within the KafkaNodePool resource.
28+
29+
This approach will allow users to specify/manage the definition of stretch Kafka cluster in a single location. The operators will then create necessary resources in target Kubernetes clusters, which can then be reconciled/managed by operators on those clusters.
30+
31+
### Reconciling Kafka and KafkaNodePool resources
32+
<img width="1321" alt="image" src="https://media.github.ibm.com/user/473362/files/0a1a4e7f-8227-47da-86b1-4b27a83fdb9d">
33+
34+
### Reconciling StrimziPodSet resources
35+
<img width="1317" alt="image" src="https://media.github.ibm.com/user/473362/files/f4524847-3c9c-4a8c-a5fd-4a76b6ec0bf4">
36+
37+
#### KafkaNodePool changes
38+
A new optional field (`target`) will be introduced in the KafkaNodePool resource specification, to allow users to specify the details of the Kubernetes cluster where the node pool should be deployed. This section will include the target cluster's URL (Kubernetes cluster where resources for this node pool will be created) and the secret containing the kubeconfig data for that cluster.
39+
40+
An example of the KafkaNodePool resource with the new fields might look like:
41+
42+
```yaml
43+
apiVersion: kafka.strimzi.io/v1beta2
44+
kind: KafkaNodePool
45+
metadata:
46+
name: controller
47+
labels:
48+
strimzi.io/cluster: my-cluster
49+
spec:
50+
replicas: 3
51+
target:
52+
clusterUrl: <K8S Cluster URL>
53+
secret: <SecretName>
54+
listenerConfig:
55+
- configuration:
56+
bootstrap:
57+
alternativeNames:
58+
- bootstrap-ext.<kubernetes infra host>
59+
- broker-100.<kubernetes infra host>
60+
- broker-101.<kubernetes infra host>
61+
host: bootstrap-ext.<kubernetes infra host>
62+
brokers:
63+
- broker: 100
64+
host: broker-100.<kubernetes infra host>
65+
- broker: 101
66+
host: broker-101.<kubernetes infra host>
67+
class: nginx
68+
name: connectext
69+
roles:
70+
- controller
71+
storage:
72+
.........
73+
```
74+
75+
#### Kafka changes
76+
77+
```yaml
78+
apiVersion: kafka.strimzi.io/v1beta2
79+
kind: Kafka
80+
metadata:
81+
name: my-cluster
82+
annotations:
83+
strimzi.io/node-pools: enabled
84+
strimzi.io/kraft: enabled
85+
strimzi.io/stretch-mode: enabled
86+
spec:
87+
kafka:
88+
version: 3.7.0
89+
metadataVersion: 3.7-IV4
90+
listeners:
91+
- port: 9093
92+
tls: true
93+
name: connectext
94+
type: ingress
95+
```
96+
97+
A new annotation (`stretch-mode: enabled`) will be introduced in Kafka custom resource to indicate when it is representing a stretch Kafka cluster. This approach is similar to how Strimzi currently enables features like KafkaNodePool (KNP) and KRaft mode.
98+
99+
In a stretch Kafka cluster, we'll need bootstrap and broker services to be present on each Kubernetes cluster and be accessible from other clusters. The Kafka reconciler will identify all target clusters from KafkaNodePool resources and create these services in target Kubernetes clusters. This will ensure that even if the central cluster experiences an outage, external clients can still connect to the stretch cluster and continue their operations without interruption.
100+
101+
#### Cross-cluster communication
102+
Kafka controllers/brokers are distributed across multiple Kubernetes environments and will need to communicate with each other. Currently, the Strimzi Kafka operator defines Kafka listeners for internal communication (controlplane and replication) between brokers/controllers (Kubernetes services using ports 9090 and 9091). The user is not able to influence how these services are set up and exposed outside the cluster. We would remove this limitation and allow users to define how these internal listeners are configured in the Kafka resource, just like they do for Kafka client listeners.
103+
104+
Users will also be able to override listener configurations in each KafkaNodePool resource, if the listeners need to be exposed in different ways (ingress host names, Ingress annotations etc.) for each Kubernetes cluster. This will be similar to how KafnaNodePools are used to override other configuration like storage etc. To override a listener, KafkaNodePool will define configuration with same listner name as in KAfka resource.
105+
106+
#### Resource cleanup on remote Kubernetes clusters
107+
As some of the Kubernetes resources will be created on a remote cluster, we will not be able to use standard Kubernetes approaches for deleting resources based on owner references. The operator will need to delete remote resources explicitly when the owning resource is deleted.
108+
109+
- The exact mechanism that will be used for such cleanup in various scenarios is not detailed out yet and will be added here before the proposal is complete.
110+
111+
#### Network policies
112+
In a stretch Kafka cluster, some Network policies will be relaxed to allow communication from other Kubernetes clusters that are specified as targets in various KafkaNodePool resources. This will allow brokers/controllers on separate Kubernetes clusters to communicate effectively.
113+
114+
#### Secrets
115+
We need to create Kubernetes Secrets in the central cluster that will store the credentials required for creating resources on the target clusters. These secrets will be referenced in the KafkaNodePool custom resource.
116+
117+
#### Entity operator
118+
We would recommend that all KafkaTopic and KakfaUser resources are managed form the cluster that holds Kafka and KafkaNodePool resources, and that should be the cluster where the entity operator should be enabled. This will allow all resource management/configuration form a central place. The entity operator should not be impacted by changes in this proposal.
119+
120+
## Additional considerations
121+
122+
Once the general approach is agreed, this proposal will be updated to include any impact on other important aspects like:
123+
- rolling updates
124+
- scaling brokers/controllers
125+
- certificate rotation
126+
- cruise control
127+
- observability
128+
129+
## Affected/not affected projects
130+
131+
This proposal only impacts strimzi-kafka-operator project.
132+
133+
## Rejected alternatives
134+
135+
- Use network technologies like skupper, submariner etc to allow internal services to be visible on other clusters
136+
- introduces additional dependencies and complexity to the Strimzi project
137+
138+
<img width="1057" alt="image" src="https://media.github.ibm.com/user/473362/files/9a9720f9-9ccf-4d4f-a56c-9a35917af711">
139+
140+
141+
An alternative approach considered was setting up a stretch Kafka cluster with synchronized `KafkaStretchCluster` and `Kafka` custom resources (CRs). The idea was to introduce a new CR called `KafkaStretchCluster`, which would contain details of all the clusters involved in the stretch Kafka deployment. The spec would include information such as cluster names, secrets for connecting to each Kubernetes cluster, and a list of node pools across the entire stretch cluster.
142+
143+
The Kafka CR could be created in any of the Kubernetes clusters, and it would be propagated to the remaining clusters through coordinated actions by the Cluster Operator. Similarly, changes to the Kafka CR could be made in any Kubernetes cluster, and once detected by the Cluster Operator, the changes would be propagated to the CRs in the other clusters. The `KafkaNodePool` resources would be deployed to individual Kubernetes clusters, requiring users to apply the KafkaNodePool CR in each cluster separately.
144+
145+
### Pros
146+
147+
- Users can modify CRs even if one of the clusters in the setup is unreachable, as there is no central cluster where all CRs must be created.
148+
- This model offers greater resilience and higher fault tolerance.
149+
150+
151+
### Cons
152+
153+
- This approach is challenging to implement, as the Cluster Operator must be coordinated to propagate CR changes across clusters.
154+
- The secrets required to connect to all the Kubernetes clusters would need to be available to all clusters, raising potential security concerns.
155+
- Since Kafka CRs are spread across clusters, it becomes difficult to identify the overall cluster state from the Kafka status, as each CR is reconciled by its own Cluster Operator.

0 commit comments

Comments
 (0)