Worker communication through REST interface

[lmenezes]

Hi there,

I'm running Strimzi 0.44.0 with Kafka 3.7.1 on Kubernetes v1.28 and I found that when using the REST interface to create/delete connectors requests will at times fail (when the non-leader worker tries to forward the request to the leader worker, as described here).

The error looks like this:

{
    "error_code": 500,
    "message": "IO Error trying to forward REST request: java.net.UnknownHostException: kafka-connect-elasticsearch-sink-cluster-connect-d768bfb959wcgl: Name or service not known"
}

My understanding is that the advertised host is the pod's name, which won't work for cross pod communication (and doesn't match the behavior described in the issue above)

I have tried to find ways to override this but with no luck.

Is there a way to solve this issue? Perhaps not overriding the advertised host, but at least having it use the ip address rather than pod name?

[katheris]

Hi, can you share some more information about your setup, from your description I'm struggling to follow what is failing. For example what does your KafkaConnect CR look like, how are you accessing the REST API (e.g. from inside or outside the cluster, through a service, or through some ingress you created?).

I'm not sure what kafka-connect-elasticsearch-sink-cluster-connect-d768bfb959wcgl is referring to because the Kafka connect pod names should be e.g. "my-connect-cluster-connect-0", "my-connect-cluster-connect-1" for a Kafka Connect cluster called "my-connect-cluster"

[lmenezes]

Hi @katheris

thanks for getting back so quickly :)

1. Kafka Connect CR (some details omitted for brevity):

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
spec:
  bootstrapServers: kafka-server:9092
  config:
    config.providers: directory
    config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
    config.storage.replication.factor: -1
    config.storage.topic: kafka-connect-elasticsearch-sink.internal.connect-cluster-configs
    group.id: kafka-connect-elasticsearch-sink-group
    offset.storage.replication.factor: -1
    offset.storage.topic: kafka-connect-elasticsearch-sink.internal.connect-cluster-offsets
    status.storage.replication.factor: -1
    status.storage.topic: kafka-connect-elasticsearch-sink.internal.connect-cluster-status
  image: xxxxxxx.dkr.ecr.us-east-1.amazonaws.com/kafka-connect-elasticsearch-sink:commit-5085ca5
  replicas: 2
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: application
                operator: In
                values:
                - services-graviton
      metadata:
        annotations:
          sidecar.istio.io/inject: "true"

2. We are accessing the REST API both from within the cluster as well as from outside (with the same result: requests failing if not handled by the leader, as it cannot resolve the host when trying to forward the request to the leader)
3. kafka-connect-elasticsearch-sink -> This is a custom imaged base off quay.io/strimzi/kafka:0.44.0-kafka-3.7.1. It basically includes our custom elasticsearch connector plugin and nothing else
4. Each pod runs also an istio sidecar

Now I'm curious to understand why the pod names are not matching what you describe.

For context, We started having this issue after upgrading strimzi from 0.38 to 0.44, I'm assuming because of the changes made to docker-images/kafka-based/kafka/scripts/kafka_connect_run.sh in this commit:

- # Prepare hostname depending on whether we use StrimziPodSets (Stable Pod Identities) or Deployments
- # For StrimziPodSets we use the Pod DNS name assigned through the headless service
- # For Deployments we use the Pod IP address
- if [ "$STRIMZI_STABLE_IDENTITIES_ENABLED" = "true" ]; then
-   ADVERTISED_HOSTNAME=$(hostname -f | cut -d "." -f1-4)
-   export ADVERTISED_HOSTNAME
- else
-   ADVERTISED_HOSTNAME=$(hostname -I | awk '{ print $1 }')
-   export ADVERTISED_HOSTNAME
- fi
+ # Prepare hostname - for StrimziPodSets we use the Pod DNS name assigned through the headless service
+ ADVERTISED_HOSTNAME=$(hostname -f | cut -d "." -f1-4)
+ export ADVERTISED_HOSTNAME

[scholzj]

I think there are several possible issues.

• Strimzi does not support or integrate with Istio. If it works for you with it, it is purely a coincidence.
• Make sure your container image is build from the same Strimzi and Kafka version you are using.
• If you have multiple Connect clusters, make sure they are each configured properly as an independent Connect cluster (there is a special chapter for it in the docs). Having multiple Connect's with the same configs causes this kind of issues. (Of course, given the lack of provided logs and detailed information, hard to say for sure.)
• Check for general networking issues.

[lmenezes]

Hey @scholzj

thanks so much for your reply.

Make sure your container image is build from the same Strimzi and Kafka version you are using.

After your comment, I double checked the version we are running and you were absolutely right, there was a mismatch between the strimzi version we have on kubernetes and the base image I was using (0.35.1 x 0.44.0).

We will be upgrading the operator to 0.44 next week, and hopefully this makes all things work smoothly (we have no issues with istio running, but I will make sure to remove it if we run into issues as a first step).

I will share the results here once complete, but thank you very much for your support ❤️

[lmenezes]

Hey folks!

Just want to follow up here that after aligning the versions, everything worked flawlessly. Even with Istio.

Thanks again for helping out ❤️