Pekko-based Application Pods Idle at Startup, Not Consuming Kafka Messages Due to Consumer Group Creation Behavior

[dahiyahimanshu]

Issue Description:
During the initial startup of Pekko-based streaming application pods, when the input Kafka topic is created for the first time, and the consumer group is automatically generated by the application, the pods remain idle and do not consume messages from all partitions. This issue does not occur if we manually create the same consumer group using Kafka CLI before starting the application.

Steps to Reproduce:

1. Create a new Kafka topic with multiple partitions using the CLI.
2. Start the Pekko-based application, allowing it to create a consumer group automatically.
3. Observe the logs: the application does not consume messages, and the consumer group has no active members initially.

Observed Behavior:
When the application creates the consumer group itself, the CURRENT-OFFSET remains - initially, and no messages are processed. However, if we manually create the consumer group beforehand, the CURRENT-OFFSET is set to 0, and the application starts consuming messages as expected.

Workaround:
Creating the consumer group manually using Kafka CLI before starting the application resolves this issue, enabling the application to begin consuming messages from all partitions as intended.

Configuration Details:
The application sets the following properties on the Kafka consumer:

AUTO_OFFSET_RESET_CONFIG set to "earliest"
ENABLE_AUTO_COMMIT_CONFIG set to "false"

Additional Information:
We are using Apache Pekko with Scala for streaming applications.
The issue appears specific to when the consumer group is created by the application itself, and not when created manually via CLI.

Expected Behavior:
The application should automatically start consuming messages from all partitions when it creates the consumer group upon startup, without needing the manual step of creating the consumer group through Kafka CLI.

Status:
A workaround exists, but the root cause remains unclear. We need to understand why the consumer group creation by the application results in an idle state, whereas a manually created consumer group does not cause this issue.

[pjfanning]

@dahiyahimanshu the logs would be really useful. It is time consuming and error prone to expect someone to reproduce the scenario themselves.

I think it is sensible for users to create the consumer groups using the Kafka CLI (your workaround).
I don't believe that our automated tests currently do this but I may have missed something. It still makes me wonder whether this issue will be easy to reproduce - that maybe it only occurs if you have a setup where startup is slowed down by running in a containerized environment.