I/O imbalance across bookies

[Jayer23]

Description

We observed an abnormal write pattern in our Pulsar cluster:

The nodes exposed in bookie metrics have little difference in read and write io. Producer has compressed write enabled.

In fact, monitoring at the node level found that io was greatly magnified, and the io differences between nodes were large. Some nodes had io less than 50MB/s, while some nodes had io greater than 450MB/s.

Journal disks on some nodes show very high write throughput (e.g., 400+ MB/s).

Environment

• Pulsar version: 3.0.7
• Storage config:
  • Journal: 3×SSD
  • Ledger：9×HDD
• Replication: 3 replicas, 2 quorum writes

Expected Behavior

Write operations should be balanced between bookies

[thetumbled]

Answered in #24010 (comment).

[Shawyeok]

You probably have some naughty partitions with heavy write throughput, you can check by following Prometheus query:

pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"} > 5 * 1024^2

If you discover partitions with excessive write loads, consider increasing the number of partitions for the affected topics. This will help distribute the write throughput more evenly across your bookie instances.

Notes about Pulsar message publishing:

• Each partition maintains exactly one writable ledger at a time
• With configuration ensembleSize=2, writeQuorum=2:
  • A partition with 100MB/s write throughput will generate approximately 100MB/s write load on each bookie in the ensemble
  • Bookies not in the current ensemble will not receive any writes from this ledger