Support Dynamic Topic Subscriptions Using Patterns in brod

[jasontleo]

Description

Currently, the brod client requires explicit topic subscriptions at initialization, making it challenging to handle dynamic topic creation efficiently. This limitation is particularly problematic in multi-tenant environments and scenarios where topics are created and removed dynamically, such as for high-priority and low-priority messaging use cases.

A feature similar to Python’s kafka-python subscribe(pattern=...) would allow brod to automatically subscribe to topics matching a specified regular expression. This capability is crucial for handling use cases like:

• Multi-Tenancy: Dynamically subscribing to tenant-specific topics without restarting consumers.
• Priority-Based Processing: High-priority topics remain static, while low-priority topics (e.g., for backfills) are created on demand and removed when processing is complete.
• Efficient Resource Utilization: Reducing active partitions and consumers for topics that are only needed intermittently.

References

Python Kafka Client supports this feature via subscribe(pattern="^low_priority_.*") (KafkaConsumer Docs).
Similar functionality exists in other Kafka client libraries, enabling more dynamic topic handling.
Would love to hear feedback from the maintainers on feasibility and potential roadmap for this feature. 🚀

[zmstone]

Hi @jasontleo

Do you mean to support topic regex pattern for this config?

brod/src/brod_group_subscriber_v2.erl

Lines 147 to 153 in e1a36e0

	%% <li>`topics': Predefined set of topic names to join the
	%% group. Mandatory
	%%
	%% NOTE: The group leader member will collect topics from all
	%% members and assign all collected topic-partitions to members in
	%% the group. i.e. members can join with arbitrary set of
	%% topics.</li>

[jasontleo]

@zmstone I'm not fully familiar with brod's internal code, but that looks correct.