Intermittently getting "[41] Not Controller: this is not the correct controller for this cluster" for CreateTopics with AWS MSK

[sachin-rafay]

Describe the bug
I am intermittently getting this error for CreateTopics function : "[41] Not Controller: this is not the correct controller for this cluster"

It works fine with my local kafka cluster but this error comes with AWS MSK.
Also, the topics already exist in MSK, but still the the call shouldn't fail with this error.

Kafka Version

• What version(s) of Kafka are you testing against? 3.5.1

• What version of kafka-go are you using? v0.4.47

To Reproduce

Resources to reproduce the behavior: Use my code block multiple times on an AWS MSK cluster.

go

func CreateTopics(topics []string) error {
	conn, err := kafka.Dial("tcp", "broker:9092")
	if err != nil {

		return err
	}
	defer conn.Close()

	controller, err := conn.Controller()
	if err != nil {
		return err
	}
	var controllerConn *kafka.Conn
	controllerConn, err = kafka.Dial("tcp", net.JoinHostPort(controller.Host, strconv.Itoa(controller.Port)))
	if err != nil {
		return err
	}
	defer controllerConn.Close()

	var topicConfigs []kafka.TopicConfig
	for _, topic := range topics {
		topicConfigs = append(topicConfigs, kafka.TopicConfig{
			Topic:             topic,
			NumPartitions:     20,
			ReplicationFactor: 3,
		})
	}

	err = controllerConn.CreateTopics(topicConfigs...)
	if err != nil {
		return err
	}
	return nil
}

Expected Behavior

The call shouldn't fail.

Observed Behavior

Intermittently getting the error : "[41] Not Controller: this is not the correct controller for this cluster"