Kafka Connect keystore password was incorrect

[Yedidyah27]

Hey!!
I have been having a reoccurring issue with my kafka-connect cluster. I am using the Strimzi kafka and kafkaconnect operator to run a cluster on kubernetes.
I have been getting random "keystore password was incorrect" failures in my kafka-connect cluster.
The only possibility that I can think of is that I updated the config of my kafka cluster before which caused its certificates to be updated and leaves the kafka-connects certificates not up to date. Is this possible?
Restarting the kafka-connect cluster always resolves this issue but I cant find the underlying cause.
Has anybody experienced something similar?

Here is my Kafka and KafkaConnect YAML:

kind: KafkaConnect
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  annotations:
    meta.helm.sh/release-name: kafka-connect-connectors
    meta.helm.sh/release-namespace: namespace1
  creationTimestamp: '2021-03-03T17:35:01Z'
  generation: 51
  labels:
    app.kubernetes.io/managed-by: Helm
  name: kafka-connect-connectors
  namespace: namespace1
spec:
  bootstrapServers: 'kafka-kafka-bootstrap:9093'
  config:
    config.storage.replication.factor: 3
    config.storage.topic: connect-cluster-configs
    group.id: connect-cluster
    key.converter.schemas.enable: false
    offset.storage.replication.factor: 3
    offset.storage.topic: connect-cluster-offsets
    status.storage.replication.factor: 3
    status.storage.topic: connect-cluster-status
    value.converter.schemas.enable: false
  replicas: 3
  resources:
    limits:
      cpu: '2'
      memory: 2048Mi
    requests:
      cpu: '1'
      memory: 1024Mi
  template:
    connectContainer:
      env:
        - name: CONNECT_VALUE_CONVERTER
          value: org.apache.kafka.connect.storage.StringConverter
  tls:
    trustedCertificates:
      - certificate: ca.crt
        secretName: kafka-cluster-ca-cert
  version: 2.5.0

kind: Kafka
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  creationTimestamp: '2021-03-03T15:41:29Z'
  generation: 2
  name: kafka
  namespace: namespace1
  resourceVersion: '135227391'
  selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/namespace/kafkas/kafka
  uid: 1e638f8a-12e0-436d-919e-531af17609e7
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      auto.create.topics.enable: false
      inter.broker.protocol.version: 2.5.0
      log.message.format.version: '2.5'
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    jmxOptions:
      authentication:
        type: password
    kafkaQueries:
      - attributes:
          - Count
        outputs:
          - standardOut
        targetMBean: 'kafka.server:type=BrokerTopicMetrics,name=*'
    listeners:
      external:
        tls: false
        type: nodeport
      plain: {}
      tls: {}
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: '2'
        memory: 4096Mi
      requests:
        cpu: '1'
        memory: 1024Mi
    storage:
      type: jbod
      volumes:
        - class: csi-sc-ssd3
          deleteClaim: false
          id: 0
          size: 5Gi
          type: persistent-claim
    version: 2.5.0
  kafkaExporter:
    resources:
      limits:
        cpu: '1'
        memory: 2048Mi
      requests:
        cpu: '0.5'
        memory: 500Mi
  observedGeneration: 5
  zookeeper:
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    metrics:
      lowercaseOutputName: true
      rules:
        - name: zookeeper_$2
          pattern: org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+)><>(\w+)
          type: GAUGE
        - labels:
            replicaId: $2
          name: zookeeper_$3
          pattern: >-
            org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+),
            name1=replica.(\d+)><>(\w+)
          type: GAUGE
        - labels:
            memberType: $3
            replicaId: $2
          name: zookeeper_$4
          pattern: >-
            org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+),
            name1=replica.(\d+), name2=(\w+)><>(Packets\w+)
          type: COUNTER
        - labels:
            memberType: $3
            replicaId: $2
          name: zookeeper_$4
          pattern: >-
            org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+),
            name1=replica.(\d+), name2=(\w+)><>(\w+)
          type: GAUGE
        - labels:
            memberType: $3
            replicaId: $2
          name: zookeeper_$4_$5
          pattern: >-
            org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+),
            name1=replica.(\d+), name2=(\w+), name3=(\w+)><>(\w+)
          type: GAUGE
        - name: zookeeper_$2
          pattern: org.apache.ZooKeeperService<name0=StandaloneServer_port(\d+)><>(\w+)
          type: GAUGE
        - name: zookeeper_$2
          pattern: >-
            org.apache.ZooKeeperService<name0=StandaloneServer_port(\d+),
            name1=InMemoryDataTree><>(\w+)
          type: GAUGE
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: '2'
        memory: 2048Mi
      requests:
        cpu: '0.5'
        memory: 500Mi
    storage:
      class: csi-sc-ssd3
      deleteClaim: true
      size: 2Gi
      type: persistent-claim
status:
  conditions:
    - lastTransitionTime: '2021-04-08T14:56:40+0000'
      status: 'True'
      type: Ready   
  listeners:
    - addresses:
        - host: kafka-kafka-bootstrap.namespace1.svc
          port: 9092
      bootstrapServers: 'kafka-kafka-bootstrap.namespace1.svc:9092'
      type: plain
    - addresses:
        - host: kafka-kafka-bootstrap.namespace1.svc
          port: 9093
      bootstrapServers: 'kafka-kafka-bootstrap.namespace1.svc:9093'
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
          ******************************************************************
          -----END CERTIFICATE-----
      type: tls
    - addresses:
        - host: 23.989.199.51
          port: 40812
        - host: 23.989.199.55
          port: 40812
        - host: 23.989.199.59
          port: 40812
      bootstrapServers: '23.989.199.51:40812,23.989.199.55:40812,23.989.199.59:40812'
      type: external
  observedGeneration: 2

[scholzj]

So, can you share the actual logs? Also, what version of Strimzi are you using?

[Yedidyah27]

Our kafka brokers are using the 0.20.1-kafka-2.5.0 image. The kafka connect image is from latest-kafka-2.6.0

[scholzj]

If you se Strimzi 0.21.1, you should not use brokers with 0.20.1-kafka-2.5.0 images which are from Strimzi 0.20.1. Similarly, the latest-kafka-2.6.0 is basically changing with every build in master. So it is not completely clear what version of it and from what time do you have. So with Strimzi 0.21.1, you should use 0.21.1-kafka-2.6.0 as the base image for Connect.

[Yedidyah27]

I will align the versions. I think though that this wont solve the problem. I took a look and in version 0.22.1 these lines were added.

 STORE=/tmp/kafka/cluster.keystore.p12
    rm -f "$STORE"
    create_keystore "$STORE" "$CERTS_STORE_PASSWORD" "/opt/kafka/connect-certs/${KAFKA_CONNECT_TLS_AUTH_CERT}"
"/opt/kafka/connect-certs/${KAFKA_CONNECT_TLS_AUTH_KEY}" "${KAFKA_CONNECT_TLS_AUTH_CERT}"
    echo "Preparing keystore is complete"

In the older version we are using I dont see the removal of the old truststore and since CERTS_STORE_PASSWORD is updated we cant access the existing trustore after a crash.

[scholzj]

No, these are exactly the lines which will help you ;-). Because of the support for read-only file systems, when the container restarts (without the pod being deleted - for example as you described when you run out of memory) then the old files will remain. So the old file with the old password will be there, but the new password does not work for it. So deleting the keystore actually helps because it can be created with the new password.

[Yedidyah27]

I found your fix here
https://github.com/strimzi/strimzi-kafka-operator/pull/4162
We were also using the old repo so we weren't really using latest.
I will upgrade and change the repo and I think this will resolve
Thank you!