Description
Hi there, wondering if it is possible to create multiple readers/consumers?
From this issue #1200 it looks like only 1 reader is possible, but I couldn't find where in the documentation this was stated. Edit: Looks like I misread and 2 readers are allowed, but I'm not sure what I'm doing wrong
I'm trying to have 2 different consumer groups consuming from 2 different topics. I've tried creating 2 readers but keep getting connection issues(Connection refused) to the Broker(Amazon MSK) for the 2nd reader I'm creating. Previously when we only had a single consumer/reader, we were able to connect with no issues. Any references to existing documentations on suggestions on how to do so is appreciated!
Kafka Version
What version of kafka-go are you using?
0.4.40
To Reproduce
Resources to reproduce the behavior:
How I'm initialising reader
func (kReader *SegmentioKafkaReader) NewReader(brokers []string, topic, groupID string) error {
// Check if its a valid reader config
rc := kafkaGo.ReaderConfig{
Brokers: brokers,
GroupID: groupID,
Topic: topic,
Dialer: kafkaGo.DefaultDialer,
MaxBytes: 10e6, // 10MB
}
if err := rc.Validate(); err != nil {
errStr := fmt.Sprintf("kafka.NewConsumer(): invalid kafka reader config!, err %v", err)
log.Printf(errStr)
return err
}
kReader.r = kafkaGo.NewReader(rc)
return nil
}
Errors I am getting
Error connecting to kafka broker abc.kafka.us-west-2.amazonaws.com:9092. Error: failed to dial: failed to open connection to abc.kafka.us-west-2.amazonaws.com:9092: dial tcp: lookup abc.kafka.us-west-2.amazonaws.com on 10.3.0.10:53: dial tcp 10.3.0.10:53: connect: connection refused.
Expected Behavior
I should be able to create 2 readers with different consumer groups consuming from different topics