0.21 -clusterCaCert metadata not getting updated

[PravinMagdum]

Hello Team,
strimzi operator 0.21.1
kafka version 2.7.0

I have added clusterCaCert metadata in kafka manifest file.
If I describe kafka resource I can see it is present however when I describe ca certificates ,could not see this metadata updated.

What I have added -

after upgrading I can see it is correctly upgraded

When I describe cluster ca certs secrets I dont see metadata for them.

What could I be missing here? How can I troubleshoot this further?

[scholzj]

Can you share the full YAML of your Kafka CR? Ideally including the status section to see if it already reconciled etc. (i.e. do kubectl get kafka ... -o yaml)

[PravinMagdum]

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  annotations:
    meta.helm.sh/release-name: dec-kafka
    meta.helm.sh/release-namespace: kafka
  creationTimestamp: "2021-01-19T16:47:24Z"
  generation: 10
  labels:
    app.kubernetes.io/managed-by: Helm
  name: dec-kafka
  namespace: kafka
  resourceVersion: "41232503"
  selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/kafka/kafkas/dec-kafka
  uid: ac478228-fd5b-47e2-8a50-7b05be97fbc2
spec:
  clientsCa:
    generateCertificateAuthority: false
  clusterCa:
    generateCertificateAuthority: false
  cruiseControl: {}
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      log.message.format.version: "2.4"
      message.max.bytes: 20971520
      offsets.topic.replication.factor: 1
      replica.fetch.max.bytes: 20971520
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
      external:
        authentication:
          type: tls
        overrides:
          bootstrap:
            dnsAnnotations:
              external-dns.alpha.kubernetes.io/access: private
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap-.
              external-dns.alpha.kubernetes.io/ttl: "60"
        tls: true
        type: loadbalancer
      plain: {}
      tls: {}
    metrics:
      lowercaseOutputName: true
      rules:
      - labels:
          clientId: $3
          partition: $5
          topic: $4
        name: kafka_server_$1_$2
        pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
        type: GAUGE
      - labels:
          broker: $4:$5
          clientId: $3
        name: kafka_server_$1_$2
        pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+),
          brokerPort=(.+)><>Value
        type: GAUGE
      - name: kafka_$1_$2_$3_percent
        pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
        type: GAUGE
      - name: kafka_$1_$2_$3_percent
        pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
        type: GAUGE
      - labels:
          $4: $5
        name: kafka_$1_$2_$3_percent
        pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
        type: GAUGE
      - labels:
          $4: $5
          $6: $7
        name: kafka_$1_$2_$3_total
        pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
        type: COUNTER
      - labels:
          $4: $5
        name: kafka_$1_$2_$3_total
        pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
        type: COUNTER
      - name: kafka_$1_$2_$3_total
        pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
        type: COUNTER
      - labels:
          $4: $5
          $6: $7
        name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
        type: GAUGE
      - labels:
          $4: $5
        name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
        type: GAUGE
      - name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
        type: GAUGE
      - labels:
          $4: $5
          $6: $7
        name: kafka_$1_$2_$3_count
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
        type: COUNTER
      - labels:
          $4: $5
          $6: $7
          quantile: 0.$8
        name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
        type: GAUGE
      - labels:
          $4: $5
        name: kafka_$1_$2_$3_count
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
        type: COUNTER
      - labels:
          $4: $5
          quantile: 0.$6
        name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
        type: GAUGE
      - name: kafka_$1_$2_$3_count
        pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
        type: COUNTER
      - labels:
          quantile: 0.$4
        name: kafka_$1_$2_$3
        pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
        type: GAUGE
    replicas: 3
    storage:
      type: jbod
      volumes:
      - deleteClaim: false
        id: 0
        size: 100Gi
        type: persistent-claim
    template:
      clusterCaCert:
        metadata:
          annotations:
            test: test-annotations
          labels:
            name: pm1
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: true
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: true
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
    version: 2.6.0
  zookeeper:
    replicas: 3
    storage:
      deleteClaim: false
      size: 100Gi
      type: persistent-claim
status:
  conditions:
  - lastTransitionTime: "2021-01-28T12:21:33.230518Z"
    message: log.message.format.version does not match the Kafka cluster version,
      which suggests that an upgrade is incomplete.
    reason: KafkaLogMessageFormatVersion
    status: "True"
    type: Warning
  - lastTransitionTime: 2021-01-28T12:23:44+0000
    message: Pod dec-kafka-kafka-0 is currently not rollable
    reason: UnforceableProblem
    status: "True"
    type: NotReady
  - lastTransitionTime: "2021-01-28T12:21:31.606586Z"
    message: In API version kafka.strimzi.io/v1beta1 the object kafkaListeners at
      path spec.kafka.listeners.kafkaListeners has been deprecated. This object has
      been replaced with GenericKafkaListener.
    reason: DeprecatedObjects
    status: "True"
    type: Warning
  - lastTransitionTime: "2021-01-28T12:21:31.606586Z"
    message: In API version kafka.strimzi.io/v1beta1 the property metrics at path
      spec.kafka.metrics has been deprecated. This feature should now be configured
      at path spec.kafka.metricsConfig.
    reason: DeprecatedFields
    status: "True"
    type: Warning
  listeners:
  - addresses:
    - host: dec-kafka-kafka-bootstrap.kafka.svc
      port: 9092
    bootstrapServers: dec-kafka-kafka-bootstrap.kafka.svc:9092
    type: plain
  - addresses:
    - host: dec-kafka-kafka-bootstrap.kafka.svc
      port: 9093
    bootstrapServers: dec-kafka-kafka-bootstrap.kafka.svc:9093
    type: tls
  - addresses:
    - host: 000000000000-7a3d26cbf27392e5.elb.us-west-2.amazonaws.com
      port: 9094
    bootstrapServers: 0000000000-7a3d26cbf27392e5.elb.us-west-2.amazonaws.com:9094
    type: external
  observedGeneration: 10

here is full yaml output file

[scholzj]

Weird. It seems to work fine for me. Are you checking it on the right secret? It should be on dec-kafka-cluster-ca-cert.

[PravinMagdum]

yes .. I am checking on dec-kafka-cluster-ca-cert .
Any guideline how can check any logs etc ? tried to look into operators pod ,couldnt get anything around

[scholzj]

Oh ... wait ... you are using your own custom certificates:

  clientsCa:
    generateCertificateAuthority: false
  clusterCa:
    generateCertificateAuthority: false

In that case, we do not touch the secrets since they are yours. So if you want to set any labels or annotations on them, you should just be able to label / annotate them your self.

[PravinMagdum]

oh ..alright.. Really appreciate your response. thanks :)