Unable to use custom certs signed by a public CA for client authentication

[yashaswinoonela]

Here is the error which im facing when trying to list topics

/bin/kafka-topics.sh --bootstrap-server mydomain.com:9093 --list --command-config /root/client.properties

[2024-11-30 12:08:40,970] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (mydomain.com/10.0.220.3:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2024-11-30 12:08:40,973] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:146)
        at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)

Content of /root/client.properties

security.protocol=SSL
ssl.truststore.location=/root/truststore.jks
ssl.truststore.password=123@abc
ssl.keystore.location=/root/keystore.jks
ssl.keystore.password=123@abc
ssl.key.password=123@abc

My kafka deployment file

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  clientsCa:
    generateCertificateAuthority: false
  kafka:
    version: 3.8.0
    metadataVersion: 3.8-IV0
    authorization:
      type: simple
    rack:
      topologyKey: topology.kubernetes.io/zone
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            alternativeNames:
              - mydomain.com
            annotations:
              cloud.google.com/load-balancer-type: "Internal"
              networking.gke.io/internal-load-balancer-allow-global-access: "true"
              cloud.google.com/neg: '{"exposed_ports": {"9093":{"name": "kafka-ingress"}}}' 
              cloud.google.com/neg-status: '{ network_endpoint_groups : { "9093" : "kafka-ingress" } }'
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      log.cleanup.interval.mins: 5
      log.flush.interval.messages: 3000
      log.flush.interval.ms: 10000
      log.flush.scheduler.interval.ms: 2000
      log.retention.hours: 24
      log.segment.bytes: 1073741824
      message.max.bytes: 4194320
      num.io.threads: 8
      num.network.threads: 8
      num.partitions: 2
      num.replica.fetchers: 4
      queued.max.requests: 1000
      replica.fetch.max.bytes: 4194320
      socket.receive.buffer.bytes: 2097152
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 2097152
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    autoRebalance:
      - mode: add-brokers
        template:
          name: add-remove-brokers-rebalancing-template
      - mode: remove-brokers
        template:
          name: add-remove-brokers-rebalancing-template
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

Created my-kafka-clients-ca and my-kafka-clients-ca-cert

kubectl create secret generic my-kafka-clients-ca --from-file=ca.key=certs/mydomain.com.key -n kafka
kubectl create secret generic my-kafka-clients-ca-cert --from-file=ca.crt=certs/mydomain.com.crt -n kafka
kubectl label secret my-kafka-clients-ca strimzi.io/kind=Kafka strimzi.io/cluster="my-kafka" -n kafka
kubectl label secret my-kafka-clients-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster="my-kafka" -n kafka
kubectl annotate secret my-kafka-clients-ca strimzi.io/ca-key-generation="0" -n kafka
kubectl annotate secret my-kafka-clients-ca-cert strimzi.io/ca-cert-generation="0" -n kafka

Create kafka user

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: my-user
  labels:
    strimzi.io/cluster: my-kafka
spec:
  authentication:
    type: tls-external
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operation: All
      - resource:
          type: group
          name: "*"
          patternType: literal
        operation: All
      - resource:
          type: cluster
          name: "*"
          patternType: literal
        operation: All

Commands used to create truststore.jks and keystore.jks

keytool -keystore truststore.jks -alias CARoot -import -file certs/mydomain.com.crt
keytool -importkeystore -srckeystore certs/mydomain.com.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype jks

I don't have any issue while using strimzi operator signed clientCa, facing this error when trying to use custom cert for client authentication.

Can you help me to understanding which step am i missing here or what wrong am i doing?

[scholzj]

For a start, you should probably explain how do you expect the Public CA to work for this use-case. Public CAs are not suitable for client authentication. You also need to explain what exactly are you putting into the different Secrets and truststores/keystores as without that the commands have no meaning.

Also, keep in mind that the first error suggests some configuration or networking issue - so that would be even before the whole client authentication comes in. But the SSL errors are sometimes misleading, so who knows.

[yashaswinoonela]

Hi @scholzj,

Im trying to setup strimzi in production, where i want to use certs signed by an public CA for clients ( producers and consumers ) to authenticate using that signed certs.

This is what im trying to achieve. If my initial is not the right way to achieve this. can you please guide me with the right approach to get this working?

should i be using brokerCertChainAndKer instead of custom client CA?

[scholzj]

For client authentication, you have to use a custom clients CA. But again - this is not done using public trusted CAs.

[yashaswinoonela]

What I understand is, I cannot use an cert which is issued by GoDaddy or any other cert provider for a domain name (mydomain.com) and point this domain to tls bootstrap service. Either i have to use my own CA or use strimzi default certs for client authentication.

[scholzj]

So, now you seem to be mixing different things. If you use your own domain as a DNS names for the load balancers, you can use a certificate from a public CA for that. In such case brokerCertChainAndKey is the right place. You just have to make sure the certificate is either some wildcard certificate or has all the DNS names in its SANs - the bootstrap DNS name as well as the broker DNS names. But that is something completely different than the client certificates used for client authentication.