scheduler: decouple the keyranges with different schedulers #9330

bufferflies · 2025-05-15T11:56:02Z

What problem does this PR solve?

Issue Number: Close #9302

What is changed and how does it work?

Check List

Tests

Unit test
Integration test

Code changes

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

None.

Signed-off-by: 童剑 <[email protected]>

ti-chi-bot · 2025-05-15T11:56:06Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: 童剑 <[email protected]>

codecov · 2025-05-16T03:55:24Z

Codecov Report

Attention: Patch coverage is 95.27027% with 7 lines in your changes missing coverage. Please review.

Project coverage is 76.14%. Comparing base (506cae1) to head (1aa9b40).
Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9330      +/-   ##
==========================================
+ Coverage   76.02%   76.14%   +0.11%     
==========================================
  Files         475      479       +4     
  Lines       74080    74636     +556     
==========================================
+ Hits        56320    56830     +510     
- Misses      14241    14275      +34     
- Partials     3519     3531      +12

Flag	Coverage Δ
unittests	`76.14% <95.27%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: 童剑 <[email protected]>

lhy1024 · 2025-05-27T08:16:28Z

pkg/schedule/schedulers/balance_leader.go

@@ -429,7 +429,12 @@ func makeInfluence(op *operator.Operator, plan *solver, usedRegions map[uint64]s
 // It randomly selects a health region from the source store, then picks
 // the best follower peer and transfers the leader.
 func (s *balanceLeaderScheduler) transferLeaderOut(solver *solver, collector *plan.Collector) *operator.Operator {
-	solver.Region = filter.SelectOneRegion(solver.RandLeaderRegions(solver.sourceStoreID(), s.conf.getRanges()),
+	rs := s.conf.getRanges()
+	if len(rs) == 1 && len(rs[0].StartKey) == 0 && len(rs[0].EndKey) == 0 {


How about adding a function in rs to check len(rs) == 1 && len(rs[0].StartKey) == 0 && len(rs[0].EndKey) == 0?

Signed-off-by: 童剑 <[email protected]>

rleungx · 2025-05-29T07:41:46Z

pkg/schedule/schedulers/balance_region.go

@@ -137,6 +137,11 @@ func (s *balanceRegionScheduler) Schedule(cluster sche.SchedulerCluster, dryRun
 	solver.Step++
 	var sourceIndex int

+	rs := s.conf.Ranges


Does the Ranges always have a value?

yes, it can be seen as

pd/pkg/schedule/schedulers/utils.go

Lines 218 to 237 in 6f886e0

func getKeyRanges(args []string) ([]keyutil.KeyRange, error) {

var ranges []keyutil.KeyRange

for len(args) > 1 {

startKey, err := url.QueryUnescape(args[0])

if err != nil {

return nil, errs.ErrQueryUnescape.Wrap(err)

}

endKey, err := url.QueryUnescape(args[1])

if err != nil {

return nil, errs.ErrQueryUnescape.Wrap(err)

}

args = args[2:]

ranges = append(ranges, keyutil.NewKeyRange(startKey, endKey))

}

if len(ranges) == 0 {

return []keyutil.KeyRange{keyutil.NewKeyRange("", "")}, nil

}

return ranges, nil

}

rleungx · 2025-05-29T07:46:00Z

pkg/schedule/rangelist/key_range_manager.go

+)
+
+// KeyRangeManager is a manager for key ranges.
+type KeyRangeManager struct {


Seems keyutil is a more proper package?

moved to the new pkg pkg/schedule/rangelist

rleungx · 2025-05-29T07:47:10Z

pkg/schedule/schedulers/balance_leader.go

@@ -429,7 +429,12 @@ func makeInfluence(op *operator.Operator, plan *solver, usedRegions map[uint64]s
 // It randomly selects a health region from the source store, then picks
 // the best follower peer and transfers the leader.
 func (s *balanceLeaderScheduler) transferLeaderOut(solver *solver, collector *plan.Collector) *operator.Operator {
-	solver.Region = filter.SelectOneRegion(solver.RandLeaderRegions(solver.sourceStoreID(), s.conf.getRanges()),
+	rs := s.conf.getRanges()
+	if IsDefaultKeyRange(rs) {


How about the hot region scheduler?

Hot region scheduler can schedule all the regions, not restricting this limit.

Signed-off-by: 童剑 <[email protected]>

ti-chi-bot · 2025-06-09T03:02:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lhy1024]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-06-09T03:02:19Z

[LGTM Timeline notifier]

Timeline:

2025-06-09 03:02:19.409732585 +0000 UTC m=+237717.638047833: ☑️ agreed by lhy1024.

bufferflies · 2025-06-10T03:20:47Z

ping @rleungx again

rleungx · 2025-06-10T06:01:17Z

pkg/schedule/keyrange/manager.go

+	for _, r := range rs {
+		s.sortedKeyRanges.Append(r.StartKey, r.EndKey)
+	}
+	s.sortedKeyRanges.SortAndDeduce()


For every time we append, we need to sort?

done， using adjust function to check the two regions

Signed-off-by: 童剑 <[email protected]>

rleungx · 2025-06-11T06:42:55Z

pkg/utils/keyutil/util.go

@@ -41,6 +41,36 @@ func MinKey(a, b []byte) []byte {
 	return a
 }

+// MaxKeyWithBoundary returns the bigger key for the given keys.
+// it is different from MaxKey, if the key is empty and the boundary is right, the empty key is biggest,
+func MaxKeyWithBoundary(a, b []byte, opt boundary) []byte {


How about changing to MaxStartKey/MinEndKey? The boundary is useless.

rleungx · 2025-06-11T06:45:44Z

pkg/utils/keyutil/keyrange.go

+	for i := 1; i < len(rs.krs); i++ {
+		last := res[len(res)-1]
+		if last.IsAdjacent(rs.krs[i]) {
+			last.StartKey = MinKeyWithBoundary(last.StartKey, rs.krs[i].StartKey, left)


It has already been sorted, no need to compare?

rleungx · 2025-06-11T06:47:23Z

pkg/schedule/schedulers/balance_leader.go

+	rs := s.conf.getRanges()
+	if IsDefaultKeyRange(rs) {
+		km := solver.GetKeyRangeManager()
+		rs = km.GetNonOverlappingKeyRanges(&rs[0])


Will it slow down the balance leader's speed?

rleungx · 2025-06-11T06:49:05Z

pkg/schedule/schedulers/balance_region.go

@@ -137,6 +137,11 @@ func (s *balanceRegionScheduler) Schedule(cluster sche.SchedulerCluster, dryRun
 	solver.Step++
 	var sourceIndex int

+	rs := s.conf.Ranges


What if we change the range manually?

bufferflies added 7 commits May 9, 2025 14:30

inferface

e1ec26d

Signed-off-by: 童剑 <[email protected]>

inferface

a39754d

Signed-off-by: 童剑 <[email protected]>

move

fdd5a97

Signed-off-by: 童剑 <[email protected]>

conflict

1270f87

Signed-off-by: 童剑 <[email protected]>

lint

fe030a0

Signed-off-by: 童剑 <[email protected]>

scheduler

1a2d2ab

Signed-off-by: 童剑 <[email protected]>

conflict

8d91e0d

Signed-off-by: 童剑 <[email protected]>

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels May 15, 2025

ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 15, 2025

bufferflies force-pushed the feature/decouple_keyrange branch 2 times, most recently from 7fa0925 to 808b7a1 Compare May 16, 2025 03:40

delete range

dbff105

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the feature/decouple_keyrange branch from 808b7a1 to dbff105 Compare May 16, 2025 03:46

conflict

5bb9eb5

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the feature/decouple_keyrange branch from 885ffe7 to 51f06d1 Compare May 22, 2025 12:24

add more test for balance range

01f2334

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the feature/decouple_keyrange branch from 51f06d1 to 01f2334 Compare May 27, 2025 07:48

bufferflies marked this pull request as ready for review May 27, 2025 07:48

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2025

lhy1024 reviewed May 27, 2025

View reviewed changes

IsDefaultKeyRange

7b54827

Signed-off-by: 童剑 <[email protected]>

rleungx reviewed May 29, 2025

View reviewed changes

move new pkg

338ca04

Signed-off-by: 童剑 <[email protected]>

bufferflies requested review from lhy1024 and rleungx June 4, 2025 08:26

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Jun 5, 2025

lint

0e2a025

Signed-off-by: 童剑 <[email protected]>

bufferflies force-pushed the feature/decouple_keyrange branch from 1aa9b40 to 0e2a025 Compare June 5, 2025 08:18

lhy1024 approved these changes Jun 9, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 9, 2025

ti-chi-bot bot added the approved label Jun 9, 2025

rleungx reviewed Jun 10, 2025

View reviewed changes

use Adjacent

02af066

Signed-off-by: 童剑 <[email protected]>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 10, 2025

default timeout is 30m

7ce9e71

Signed-off-by: 童剑 <[email protected]>

rleungx reviewed Jun 11, 2025

View reviewed changes


	func getKeyRanges(args []string) ([]keyutil.KeyRange, error) {
	var ranges []keyutil.KeyRange
	for len(args) > 1 {
	startKey, err := url.QueryUnescape(args[0])
	if err != nil {
	return nil, errs.ErrQueryUnescape.Wrap(err)
	}
	endKey, err := url.QueryUnescape(args[1])
	if err != nil {
	return nil, errs.ErrQueryUnescape.Wrap(err)
	}
	args = args[2:]
	ranges = append(ranges, keyutil.NewKeyRange(startKey, endKey))
	}
	if len(ranges) == 0 {
	return []keyutil.KeyRange{keyutil.NewKeyRange("", "")}, nil
	}
	return ranges, nil
	}

scheduler: decouple the keyranges with different schedulers #9330

Are you sure you want to change the base?

scheduler: decouple the keyranges with different schedulers #9330

Uh oh!

Conversation

bufferflies commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot bot commented May 15, 2025

Uh oh!

codecov bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rleungx May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jun 9, 2025

Uh oh!

ti-chi-bot bot commented Jun 9, 2025

[LGTM Timeline notifier]

Uh oh!

bufferflies commented Jun 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bufferflies Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bufferflies commented May 15, 2025 •

edited

Loading

codecov bot commented May 16, 2025 •

edited

Loading

rleungx May 29, 2025 •

edited

Loading

bufferflies Jun 10, 2025 •

edited

Loading