feat: add block-level partition shuffle

[dantengsky]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Add heuristic-based block-level shuffle for better load balancing when tables have few segments relative to cluster size.

Background

In distributed query scenarios, when a table has few segments relative to the cluster size, the original segment-level Mod distribution strategy causes uneven load balancing.

Problem scenarios:

• 3 segments, 4 nodes, 300 blocks → Segments per node: 1-1-1-0 (1 node idle, max 2x workload difference)
• 10 segments, 4 nodes, 1000 blocks → Segments per node: 3-3-2-2 (50% workload difference)

Solution

Introduce an automatic block-level distribution heuristic:

1. Trigger condition: Activates when segment_count < cluster_nodes * threshold
2. Distribution method: All segments assign to all nodes; each node filters blocks by block_idx % num_nodes == node_idx
3. setting: auto_block_shuffle_threshold (default: 5, set to 0 to disable)

How it improves

┌───────────────────────────────────┬────────────────────────────────────────────┬─────────────────────────┐
│             Scenario              │               Original (Mod)               │     New (BlockMod)      │
├───────────────────────────────────┼────────────────────────────────────────────┼─────────────────────────┤
│ 3 segments, 4 nodes, 300 blocks   │ Segments: 1-1-1-0, blocks: 100-100-100-0   │ Blocks: 75-75-75-75     │
├───────────────────────────────────┼────────────────────────────────────────────┼─────────────────────────┤
│ 10 segments, 4 nodes, 1000 blocks │ Segments: 3-3-2-2, blocks: 300-300-200-200 │ Blocks: 250-250-250-250 │
└───────────────────────────────────┴────────────────────────────────────────────┴─────────────────────────┘

With block-level distribution, workload is evenly distributed regardless of segment count.

New Settings

-- View current threshold
SELECT value FROM system.settings WHERE name = 'auto_block_shuffle_threshold';

-- Adjust threshold (block-level distribution when segment < nodes * threshold)
SET auto_block_shuffle_threshold = 5; -- default

-- Disable automatic block-level distribution
SET auto_block_shuffle_threshold = 0;

Changes

• Add BlockMod shuffle kind for block-level distribution
• Add auto_block_shuffle_threshold setting (default=5, 0 to disable)
• When segment_count < nodes * threshold, use block-level shuffle
• Each executor filters blocks by block_idx % num_executors == executor_idx
• Add info logging for shuffle strategy selection

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[github-actions]

Docker Image for PR

• tag: pr-19311-14cf1ad-1769051528

note: this image tag is only available for internal use.

[github-actions]

Docker Image for PR

• tag: pr-19311-27fa899-1769066148

note: this image tag is only available for internal use.

[github-actions]

ClickBench Report

• hits: https://benchmark.databend.com/clickbench/pr/19311/21238603203/hits.html
• tpch100: https://benchmark.databend.com/clickbench/pr/19311/21238603203/tpch100.html
• tpch1000: https://benchmark.databend.com/clickbench/pr/19311/21238603203/tpch1000.html

[github-actions]

🤖 CI Job Analysis

Workflow: 21243076599

⛔️ CANCELLED

Higher priority request detected - retry cancelled to avoid conflicts.

View Workflow