Issue with Burrow from 1.6.0 to 1.7.0/1.8.0/1.9.1/1.9.2

[grigassukys]

Hello,

I've encountered an issue when updating the Burrow package from version 1.6.0 to 1.7.0/1.8.0/1.9.1/1.9.2. The configuration remains exactly the same as it was in version 1.6.0, but starting from version 1.7.0, it no longer works as expected. If I rollback to 1.6.0, everything works the same as before.

Errors I'm getting when updating to 1.7.0/1.8.0/1.91/1.92:

{"level":"debug","ts":1715166414.740178,"msg":"Error while performing SASL Auth IP:PORT","name":"sarama"} {"level":"debug","ts":1715166414.7402897,"msg":"Closed connection to broker IP:PORT","name":"sarama"} {"level":"debug","ts":1715166414.74035,"msg":"Error while sending ApiVersionsRequest to broker IP:PORT: EOF","name":"sarama"} {"level":"debug","ts":1715166414.740367,"msg":"client/metadata got error from broker -1 while fetching metadata:EOF","name":"sarama"}

Or like this
{"level":"error","ts":1715163313.996046,"msg":"failed to start client","type":"module","coordinator":"cluster","class":"kafka","name":"MYCLUSTER","error":"kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* EOF\n\t* EOF\n\t* EOF\n"}

Steps to Reproduce:

1. Update the Burrow package from version 1.6.0 to 1.7.0/1.8.0/1.9.1/1.9.2
2. Use the same configuration as in version 1.6.0.

Expected Behavior:

The Burrow package should work as expected with the same configuration as in version 1.6.0.

Actual Behavior:

The Burrow package does not work as expected starting from version 1.7.0.

Environment:

• Burrow version: 1.7.0/1.8.0/1.9.1/1.9.2
• Kafka Version: 3.5.2
• Zookeeper version: 3.8.0

[lukasertl]

Is your kafka-version setting in the client-profile up-to-date? I had similar problems and fixed it by updating this setting.

[grigassukys]

@lukasertl kafka-version is set to 3.5.2 as this is our version of the cluster

[nkostoulas]

Having the same issue when using SASL/SCRAM for authentication.

[grigassukys]

@bai maybe you could take a look at this? thank you!

[cdenneen]

Having same issue... had to revert back to 1.6.0

My 1.6.0 config has a fake tls with unverify=true which has worked for years. I thought this was the issue with the EOF so I actually removed that and went back to plaintext instead. This still rendered the same EOF issue on 1.8.0. So I was able to re-enable the TLS back on 1.6.0 but no idea why tls or plaintext both keep getting these EOF issues on latest burrow.

[grigassukys]

Haven't found a solution to this problem.

[grigassukys]

@bai this issue still exists on 1.9.1

[lisikh]

Having the same issue with >1.6.0, not using tls/sasl configuration.
Tried to change "kafka-version" param without success.

On kafka node getting:

[2025-01-30 16:15:01,654] ERROR Exception while processing request from 172.17.0.3:9092-172.16.3.135:59294-337154 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: METADATA, apiVersion: 10, connectionId: 172.17.0.3:9092-172.16.3.135:59294-337154, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
Caused by: java.lang.IllegalArgumentException: Invalid version for API key METADATA: 10

And kafka-broker-api-versions shows:

Metadata(3): 0 to 9 [usable: 9]

UPDATE:
Solved this issue. All was need is to set kafka-version=2.0.0 in client-profile and link this profile to consumer section:

[client-profile.myclient]
kafka-version="2.0.0"

[consumer.test]
class-name="kafka"
cluster="test"
client-profile="myclient"

[grigassukys]

Problem still occurs with the 1.9.2 version

[putz612]

I ran into this as well, the secret is the client profile but you have to include it in cluster and consumer

[client-profile.my_profile]
client-id="my_cluster"
kafka-version="2.0.0"

[cluster.my_cluster]
class-name="kafka"
client-profile="my_profile"
servers=["broker1:9092","broker2:9092","broker3:9092"]
topic-refresh=120
offset-refresh=30
groups-reaper-refresh=600

[consumer.my_consumers]
class-name="kafka"
client-profile="my_profile"
cluster="my_cluster"
servers=["broker1:9092","broker2:9092","broker3:9092"]

[grigassukys]

My configuration has client-profile in both cluster and consumer blocks. The application itself was working with every version until it didn't with >1.7.0
I'm assuming nothing has changed in requirements on configuration side therefore it should work as before?

[nsagnett]

To give more context and after some testing, it seems to be a problem only with SASL SCRAM, when I connect with plain text protocol, everything works fine.

Do you know what changed since 1.7.0 version about that?

[nsagnett]

@grigassukys After some more tests, if you use SASL, you have to set handshake-first=true in your sasl configuration part, it seems to work for me.

[grigassukys]

@nsagnett Thank you so much. Even though handshake-first in documentation is set to true by default, but setting handshake-first=true in my SASL configuration did the trick and now burrow works as before.
Thank you!

[putz612]

Looking over the rest of my config I should have posted that section too, I have handshake-first=true set as well.

Here is my full config

[client-profile.my_profile]
client-id="my_cluster"
kafka-version="2.0.0"

[cluster.my_cluster]
class-name="kafka"
client-profile="my_profile"
servers=["broker1:9092","broker2:9092","broker3:9092"]
topic-refresh=120
offset-refresh=30
groups-reaper-refresh=600

[consumer.my_consumers]
class-name="kafka"
client-profile="my_profile"
cluster="my_cluster"
servers=["broker1:9092","broker2:9092","broker3:9092"]

[sasl.my_cluster]
username="burrow"
handshake-first=true
password="burrow-pass"
mechanism="PLAIN"

[tls.my_cluster]
noverify=true