broker-list.sh trouble parsing IP6 port forwards

[dagardner-nv]

When I start up a Kafka cluster using docker-compose up -d --scale kafka=3 I get a port mapping for both IP4 and IP6.

$ docker ps | grep 9092
5cce048aef23   kafka-docker_kafka       "start-kafka.sh"         12 minutes ago   Up 12 minutes   0.0.0.0:49159->9092/tcp, :::49159->9092/tcp                             kafka-docker_kafka_2
868f9c7b2783   kafka-docker_kafka       "start-kafka.sh"         12 minutes ago   Up 12 minutes   0.0.0.0:49161->9092/tcp, :::49161->9092/tcp                             kafka-docker_kafka_3
dafc8b2affe1   kafka-docker_kafka       "start-kafka.sh"         12 minutes ago   Up 12 minutes   0.0.0.0:49160->9092/tcp, :::49160->9092/tcp                             kafka-docker_kafka_1

Such that running docker port "$CONTAINER" 9092 returns:

$ docker port "5cce048aef23" 9092
0.0.0.0:49159
:::49159

This causes broker-list.sh to return:

<KAFKA_ADVERTISED_HOST_NAME>:49159,:::49159,<KAFKA_ADVERTISED_HOST_NAME>:49161,:::49161,<KAFKA_ADVERTISED_HOST_NAME>:49160,:::49160

This could be a mis-configuration on my side but Kafka is unable to connect to5:20.10.17 the :: entries, and even if it could these would be duplicates:

$ ./start-kafka-shell.sh $KAFKA_ADVERTISED_HOST_NAME
root@12839a5ca207:/# $KAFKA_HOME/bin/kafka-topics.sh --create --topic test \
--partitions 4 --replication-factor 2 \
--bootstrap-server `broker-list.sh`
[2022-07-19 18:24:37,199] WARN [AdminClient clientId=adminclient-1] Connection to node -2 (/0:0:0:0:0:0:0:0:49159) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Created topic test.

I was able to work-around the issue by changing broker-list.sh:

diff --git a/broker-list.sh b/broker-list.sh
index 5c5ee2d..8c0d8ba 100755
--- a/broker-list.sh
+++ b/broker-list.sh
@@ -1,5 +1,5 @@
 #!/bin/bash
 
 CONTAINERS=$(docker ps | grep 9092 | awk '{print $1}')
-BROKERS=$(for CONTAINER in ${CONTAINERS}; do docker port "$CONTAINER" 9092 | sed -e "s/0.0.0.0:/$HOST_IP:/g"; done)
+BROKERS=$(for CONTAINER in ${CONTAINERS}; do docker port "$CONTAINER" 9092 | head -n 1 | sed -e "s/0.0.0.0:/$HOST_IP:/g"; done)
 echo "${BROKERS//$'\n'/,}"

but this change relies on the IP4 port appearing prior to the IP6 port.

Version info:

$ docker --version
Docker version 20.10.17, build 100c701
$ docker-compose --version
docker-compose version 1.29.2, build unknown

[dagardner-nv]

Looks like a simpler work-around is to specify the ports as:

ports:
   - "0.0.0.0::9092"

Not sure if this is a known issue or not, feel free to close.

[OneCricketeer]

That's correct.

Or you can try an environment variable - https://stackoverflow.com/a/67071065