Kafka source (v0.51.0) fails SCRAM-SHA-512 SASL authentication when connecting to AWS MSK (Kafka 4.0) #24310
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
Vector 0.51.0 fails SASL authentication with SCRAM-SHA-512 when connecting to an Amazon MSK Kafka cluster running Apache Kafka 4.0. The failure occurs during the SCRAM handshake in the AUTH_REQ state after a successful socket + TLS handshake (~318ms).
The credentials used (username/password) have been independently validated outside Vector using Kafka CLI tooling (kcat, kafka-metadata-shell, kafka-metadata-shell) and confirmed to:
Authenticate successfully against the same MSK brokers
Return cluster metadata without errors
Confirm the SASL user is active and password is correct
Confirm no ACL, SCRAM user, or broker-side credential rejection issues
This indicates the authentication failure is specific to Vector’s Kafka source client (librdkafka), not the MSK cluster or IAM/SCRAM credentials.
Configuration
Steps to reproduce the behavior:
Configure a Vector Kafka source with SCRAM-SHA-512 + TLS enabled.
Point bootstrap_servers to an AWS MSK TLS/SASL endpoint (port 9096 or configured secure port).
Start Vector and observe authentication failure.
Configuration snippet:
[sources.msk_source]
type = "kafka"
bootstrap_servers = "<MSK_BROKER>:9096"
sasl.enabled = true
sasl.mechanism = "SCRAM-SHA-512"
sasl.username = "<USERNAME>"
sasl.password = "<PASSWORD>"
tls.enabled = true
Error log captured:
vector | 2025-11-24T12:35:03.000615Z ERROR source{component_kind="source" component_id=msk_source component_type=kafka}: kafka_source:
librdkafka: FAIL [thrd:sasl_ssl://<MSK_BROKER>:9096/bootstrap]: SASL authentication error:
Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 318ms in state AUTH_REQ)
Expected behavior
Vector should authenticate successfully using the provided SCRAM-SHA-512 credentials over TLS and start consuming Kafka topic data from the MSK cluster, consistent with behavior exhibited by other Kafka clients.
Version
0.51.0
Debug Output
vector | 2025-11-24T12:35:03.000615Z ERROR source{component_kind="source" component_id=msk_source component_type=kafka}: kafka_source:
librdkafka: FAIL [thrd:sasl_ssl://<MSK_BROKER>:9096/bootstrap]: SASL authentication error:
Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 318ms in state AUTH_REQ)
Example Data
No response
Additional Context
we implemented the config changes recommended in issue 23385 :#23385 but its did work for us.
References
No response