Getting this error after enabling kafka tier storage

[iamhritik]

What can we help you with?

Getting this error in kafka server.log

[2025-05-06 18:27:36,248] ERROR Encountered error while initializing topic-based RLMM resources (org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager)
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:554)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:147)
	at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.initializeResources(TopicBasedRemoteLogMetadataManager.java:394)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:254)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:189)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538)
	... 3 more
Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:183)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
	at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
	... 5 more
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
	at org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:303)
	at org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:92)
	at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61)
	at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:112)
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:169)
	... 9 more

server.properties kafka tier related changes:

remote.log.storage.system.enable=true
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=INTERNAL
remote.log.metadata.manager.impl.prefix=rlmm.config.
remote.log.storage.manager.class.path=/opt/kafka_2.13-3.8.0/core/*:/opt/kafka_2.13-3.8.0/s3/*
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.impl.prefix=rsm.config.

rlmm.config.remote.log.metadata.topic.replication.factor=1

rsm.config.chunk.size=107374
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=onemind-tier-storage-poc
rsm.config.storage.s3.region=eu-central-1
rsm.config.storage.s3.credentials.default=true
aws.access.key.id=REDACTED
aws.secret.access.key=REDACTED
remote.storage.enable=true

kafka cluster

version: 3.8 with zookeeper

[mapan1984]

You may need to configure access information for the remote log metadata manager, for example:

rlmm.config.remote.log.metadata.common.client.bootstrap.servers=***
rlmm.config.remote.log.metadata.common.client.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.common.client.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.common.client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='***' password='***';

rlmm.config.remote.log.metadata.consumer.bootstrap.servers=***
rlmm.config.remote.log.metadata.consumer.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.consumer.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='***' password='***';

rlmm.config.remote.log.metadata.producer.bootstrap.servers=***
rlmm.config.remote.log.metadata.producer.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.producer.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='***' password='***';

[DDDFiish]

You may need to configure access information for the remote log metadata manager, for example:

rlmm.config.remote.log.metadata.common.client.bootstrap.servers=***
rlmm.config.remote.log.metadata.common.client.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.common.client.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.common.client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='' password='';

rlmm.config.remote.log.metadata.consumer.bootstrap.servers=***
rlmm.config.remote.log.metadata.consumer.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.consumer.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='' password='';

rlmm.config.remote.log.metadata.producer.bootstrap.servers=***
rlmm.config.remote.log.metadata.producer.security.protocol=SASL_PLAINTEXT
rlmm.config.remote.log.metadata.producer.sasl.mechanism=PLAIN
rlmm.config.remote.log.metadata.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule sufficient username='' password='';

This works for me! But could you also add a note about this in the documentation? I think many people still use authentication mechanisms for Kafka rather than PLAIN.

[jeqo]

This is part of the upstream TS configurations. I'm adding references on our docs: #697

[iamhritik]

Thanks