Impossible to connect to external kafka SASL_SSL

[VizzleTF]

Issue submitter TODO list

• I've searched for an already existing issues here

Describe the bug (actual behavior)

I cannot connect new components sentry-taskbroker-* to external kafka with SASL_SSL.

Expected behavior

No response

values.yaml

externalKafka:
host: "host.lan"
port: 9092
sasl:
mechanism: SCRAM-SHA-512 # PLAIN,SCRAM-SHA-256,SCRAM-512
username: username
password: password
sslCertificatesSecret:
enabled: true
name: kafka-ssl-certificates
caKey: ca.crt
certKey: kafka_client.crt
keyKey: kafka_client.key
security:
protocol: SASL_SSL

Helm chart version

28.0.2

Steps to reproduce

1. Use external kafka with SASL_SSL
2. Connect sentry-taskbroker-*-0

Screenshots

Logs

taskbroker starting
version: 78a5f141838cd563eda6e68963c12f0ff7832dfd
2026-01-15T14:05:59.867392Z INFO taskbroker: Running full vacuum on database
2026-01-15T14:05:59.868116Z INFO taskbroker: Full vacuum completed.
2026-01-15T14:05:59.869503Z INFO taskbroker: GRPC server listening on 0.0.0.0:50051
2026-01-15T14:05:59.896754Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): host.lan:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 11ms in state APIVERSION_QUERY)
2026-01-15T14:05:59.896803Z WARN rdkafka::producer::base_producer: Ignored event 'Error' on base producer poll
2026-01-15T14:05:59.896818Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2026-01-15T14:05:59.896830Z WARN rdkafka::producer::base_producer: Ignored event 'Error' on base producer poll
2026-01-15T14:05:59.898131Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): host.lan:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 12ms in state APIVERSION_QUERY)
2026-01-15T14:05:59.898170Z ERROR taskbroker::kafka::consumer: Failed to connect to broker, retrying...
2026-01-15T14:05:59.898181Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2026-01-15T14:05:59.898191Z ERROR taskbroker::kafka::consumer: Got unexpected status from consumer client: Err(KafkaError (Message consumption error: AllBrokersDown (Local: All broker connections are down)))
2026-01-15T14:05:59.898237Z INFO taskbroker::kafka::consumer: Cancellation token received, shutting down consumer...
2026-01-15T14:05:59.898277Z INFO handle_events: taskbroker::kafka::consumer: Received event: Shutdown
2026-01-15T14:05:59.976157Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): host.lan:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 9ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
2026-01-15T14:05:59.976198Z WARN rdkafka::producer::base_producer: Ignored event 'Error' on base producer poll
2026-01-15T14:05:59.998670Z INFO taskbroker: Task consumer completed
2026-01-15T14:05:59.998676Z INFO taskbroker: Cancellation token received, shutting down GRPC server
2026-01-15T14:05:59.998747Z INFO taskbroker::upkeep: Cancellation token received, shutting down upkeep

Additional context

No response

[gondaz]

same issue with taskbrokers but using SASL_PLAINTEXT

taskbroker starting
version: 
2026-01-20T12:20:08.548906Z  INFO taskbroker: Running full vacuum on database
2026-01-20T12:20:08.549655Z  INFO taskbroker: Full vacuum completed.
2026-01-20T12:20:08.552810Z  INFO taskbroker: GRPC server listening on 0.0.0.0:50051
2026-01-20T12:20:08.700317Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka-broker-0.kafka.svc.cluster.local:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 136ms in state UP)    
2026-01-20T12:20:08.700375Z  WARN rdkafka::producer::base_producer: Ignored event 'Error' on base producer poll    
2026-01-20T12:20:08.874097Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka-broker-0.kafka.svc.cluster.local:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 163ms in state UP, 1 identical error(s) suppressed)    
2026-01-20T12:20:08.874145Z  WARN rdkafka::producer::base_producer: Ignored event 'Error' on base producer poll    
2026-01-20T12:20:11.679854Z  INFO pre_rebalance: taskbroker::kafka::consumer: Partition assignment event sent, waiting for rendezvous...
2026-01-20T12:20:11.679892Z  INFO handle_events: taskbroker::kafka::consumer: Received event: Assign({("taskworker-products", 0)})
2026-01-20T12:20:11.680852Z  INFO pre_rebalance: taskbroker::kafka::consumer: Rendezvous complete

tried to add extra variables like following:

         - name: TASKBROKER_KAFKA_SECURITY_PROTOCOL                                                                      
           value: SASL_PLAINTEXT                                                                                                    
         - name: TASKBROKER_KAFKA_SASL_MECHANISM                                                                   
           value: SCRAM-SHA-512
         - name: TASKBROKER_KAFKA_SASL_USERNAME
            value: kafka-user
         - name: TASKBROKER_KAFKA_SASL_PASSWORD
            value: superpass

but to no avail, the issue still persists

any ideas what can be wrong?

[gondaz]

I managed to fix the error by adding extra variables for a taskbroker section like that:

        - name: TASKBROKER_KAFKA_DEADLETTER_CLUSTER
          value: kafka:9092
        - name: TASKBROKER_KAFKA_DEADLETTER_TOPIC
          value: taskworker-dlq
        - name: TASKBROKER_KAFKA_DEADLETTER_SECURITY_PROTOCOL
          value: SASL_PLAINTEXT
        - name: TASKBROKER_KAFKA_DEADLETTER_SASL_MECHANISM
          value: SCRAM-SHA-512
        - name: TASKBROKER_KAFKA_DEADLETTER_SASL_USERNAME
          value: user
        - name: TASKBROKER_KAFKA_DEADLETTER_SASL_PASSWORD
          value: password

and it worked! taskbroker containers started correctly, no errors anymore
seems like it requires to have kafka deadletter staff in the inner configuration while rdkafka initializes a connection to the kafka cluster, weird for me cause there is no any word about it either in the sentry chart or in a taskbroker repo
hopefully it helps
I think it must be added to the helm chart as well to the appropriate template

[VizzleTF]

In my case:

  # Main Kafka connection configuration
  - name: TASKBROKER_KAFKA_SECURITY_PROTOCOL
    value: SASL_SSL
  - name: TASKBROKER_KAFKA_SASL_MECHANISM
    value: SCRAM-SHA-512
  - name: TASKBROKER_KAFKA_SASL_USERNAME
    value: ops_sentry
  - name: TASKBROKER_KAFKA_SASL_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kafka-credentials
        key: kafka-password
  - name: TASKBROKER_KAFKA_SSL_CA_LOCATION
    value: /etc/kafka/certs/ca.crt
  - name: TASKBROKER_KAFKA_SSL_CERTIFICATE_LOCATION
    value: /etc/kafka/certs/kafka_broker.crt
  - name: TASKBROKER_KAFKA_SSL_KEY_LOCATION
    value: /etc/kafka/certs/kafka_broker.key
  - name: TASKBROKER_KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM
    value: none
  # Dead Letter Queue configuration
  - name: TASKBROKER_KAFKA_DEADLETTER_CLUSTER
    value: kafka:9092
  - name: TASKBROKER_KAFKA_DEADLETTER_TOPIC
    value: test.taskworker-dlq
  - name: TASKBROKER_KAFKA_DEADLETTER_SECURITY_PROTOCOL
    value: SASL_SSL
  - name: TASKBROKER_KAFKA_DEADLETTER_SASL_MECHANISM
    value: SCRAM-SHA-512
  - name: TASKBROKER_KAFKA_DEADLETTER_SASL_USERNAME
    value: ops_sentry
  - name: TASKBROKER_KAFKA_DEADLETTER_SASL_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kafka-credentials
        key: kafka-password
  - name: TASKBROKER_KAFKA_DEADLETTER_SSL_CA_LOCATION
    value: /etc/kafka/certs/ca.crt
  - name: TASKBROKER_KAFKA_DEADLETTER_SSL_CERTIFICATE_LOCATION
    value: /etc/kafka/certs/kafka_broker.crt
  - name: TASKBROKER_KAFKA_DEADLETTER_SSL_KEY_LOCATION
    value: /etc/kafka/certs/kafka_broker.key
  - name: TASKBROKER_KAFKA_DEADLETTER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM
    value: none

Thank you @gondaz for investigating the problem.