feat(demo): Auto Balancing for Traffic Load

[daniel-y]

Task

Create a demo to showcase AutoMQ's traffic-based auto-balancing feature.

Background

In real-world scenarios, data distribution is often uneven, causing some brokers to handle significantly more traffic (bytes/sec) than others. This demo will show how AutoMQ's auto-balancer detects this imbalance and redistributes partitions to achieve a more even traffic load across the cluster.

Proposed Solution / Demo Plan

1. Environment Setup:

   • Create a docker-compose.yml with a 3-node AutoMQ cluster.
   • Include a load-generation client container in the compose file (e.g., using the AutoMQ image which contains Kafka tools).

2. Create Imbalance:

   • Create a topic with multiple partitions (e.g., 6 partitions).
   • Use kafka-producer-perf-test.sh or a custom script to produce a high volume of traffic (large record-size).
   • Direct the majority of the traffic to partitions primarily located on a single broker (e.g., Broker 1).

3. Observation:

   • Instruct the user on how to check initial broker metrics (e.g., BytesInPerSec) to confirm the imbalance. This can be done via JMX metrics exposed by the brokers or a pre-configured Grafana dashboard.
   • Allow the auto-balancer to run. The demo should note that this may take a few minutes.

4. Verification:

   • Instruct the user to check the broker metrics again after some time.
   • The BytesInPerSec metric should be more evenly distributed across all three brokers.
   • Show how to verify that partitions have been moved from the hot broker to the other brokers.

Acceptance Criteria

• A runnable docker-compose.yml is provided in the auto-balancing-demos/traffic-based/ directory.
• The README.md contains a clear, step-by-step guide to execute the demo.
• The guide explains how to create the traffic imbalance.
• The guide explains how to observe the "before" and "after" state of broker traffic metrics to verify the auto-balancing action.