feat(demo): Auto Balancing for Slow Node Isolation

[daniel-y]

Task

Create a demo to showcase AutoMQ's ability to isolate slow nodes.

Background

Hardware degradation or network issues can cause a broker to become a "slow node," impacting overall cluster performance. This demo will show how AutoMQ's auto-balancer detects such a node by its high latency and automatically migrates its partitions away to maintain cluster health.

Proposed Solution / Demo Plan

1. Environment Setup:

   • Create a docker-compose.yml with a 3-node AutoMQ cluster.
   • The setup should allow for network manipulation of a single broker.

2. Simulate a Slow Node:

   • After the cluster is running, use a tool like tc (traffic control) within the Docker network to introduce artificial latency to one of the broker containers (e.g., add 200ms of latency to all incoming and outgoing traffic for automq-broker-3).
   • The README.md will provide the exact docker exec command to apply this latency.
   • Start a consistent load across all brokers.

3. Observation:

   • Instruct the user on how to observe that one broker has significantly higher produce/fetch latency metrics.
   • Allow the auto-balancer to detect the anomaly.

4. Verification:

   • After a few minutes, instruct the user to check the partition distribution.
   • They should observe that most or all partitions have been migrated away from the simulated slow node.
   • The guide should also include the command to remove the artificial latency.

Acceptance Criteria

• A runnable docker-compose.yml is provided in auto-balancing-demos/slow-node-isolation/.
• The README.md provides a clear, step-by-step guide.
• The guide includes the specific commands to simulate a slow node by adding network latency.
• The guide explains how to verify that partitions have been moved off the slow node.