feat(demo): Auto Balancing for QPS Load

[daniel-y]

Task

Create a demo to showcase AutoMQ's QPS-based auto-balancing feature.

Background

A broker might be overloaded due to a high rate of small requests (high QPS) even if its total byte-rate is low. This demo will illustrate how AutoMQ can rebalance the load based on the number of requests per second to prevent QPS hotspots.

Proposed Solution / Demo Plan

1. Environment Setup:

   • Create a docker-compose.yml with a 3-node AutoMQ cluster and a client container.

2. Create Imbalance:

   • Create a topic with multiple partitions.
   • Use kafka-producer-perf-test.sh or a custom script to produce a high rate of messages (--throughput) but with a very small record-size (e.g., 16 bytes).
   • Target partitions primarily located on a single broker to create a QPS hotspot.

3. Observation:

   • Instruct the user on how to check initial broker metrics, specifically MessagesInPerSec or a similar QPS-related metric, to confirm the imbalance.
   • Allow the auto-balancer to run.

4. Verification:

   • Instruct the user to re-check the QPS metrics after a few minutes.
   • The MessagesInPerSec metric should be more evenly distributed across the cluster, demonstrating that partitions were moved from the QPS-hot broker.

Acceptance Criteria

• A runnable docker-compose.yml is provided in the auto-balancing-demos/qps-based/ directory.
• The README.md contains a clear, step-by-step guide.
• The guide explains how to create the QPS imbalance.
• The guide explains how to observe the "before" and "after" state of broker QPS metrics to verify the auto-balancing action.