Doris unable to connect to AWS MSK (Kafka) via SASL_SSL

[mq-lionel]

Issue

Doris fails to connect to MSK via SASL_SSL using the public access broker urls.

However, I can connect to and execute commands on MSK using confluent-kafka with python and Kafka CLIs on local machine and various EC2 machine. I'm sure I have created the topic in all 3 partitions and tested it locally and on EC2s. The username and password are correct too.

We follow this "Deploy Docker Cluster Instruction" using apache/doris:2.0.0_alpha-fe-x86_64 image to explore Doris.

Is it a bug or is there some installation or setup missing here?

Code

CREATE ROUTINE LOAD kafka_msk_test.example_routine_load ON example_routine_load_table
WITH APPEND
COLUMNS(id,username,tasks)
PROPERTIES
(
    "format" = "json",
    "jsonpaths" = "[\"$.id\",\"$.username\",\"$.tasks\"]",
    "strip_outer_array" = "false",
    "desired_concurrent_number" = "5",
    "max_batch_interval" = "20",
    "max_batch_rows" = "300000",
    "max_batch_size" = "209715200",
    "strict_mode" = "false"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-2-public.....amazonaws.com:9196,b-1-public.....amazonaws.com:9196,b-3-public.....amazonaws.com:9196",
    "kafka_partitions" = "0,1,2",
    "kafka_topic" = "hello-kafka",
    "property.group.id" = "group",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "SCRAM-SHA-512",
    "property.sasl.username" = "...",
    "property.sasl.password" = "...",
    "property.enable.auto.commit" = "true",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);

Exception:

Execution failed: Error Failed to execute sql: java.sql.SQLException: (conn=41) errCode = 2, detailMessage = Failed to get all partitions of kafka topic: hello-kafka. error: Waited 5 seconds (plus 172678 nanoseconds delay) for io.grpc.stub.ClientCalls$GrpcFuture@11ee7d1[status=PENDING, info=[GrpcFuture{clientCall={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=doris.PBackendService/get_info, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@4fa3c662, responseMarshaller=io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1e0d8fac, schemaDescriptor=org.apache.doris.proto.PBackendServiceGrpc$PBackendServiceMethodDescriptorSupplier@26192bfe}}}}]]

[Neustradamus]

@mq-lionel: Any news about it?