planner, core: implement partial order TopN attach2Task and partial order flow #65799

elsa0520 · 2026-01-26T04:13:22Z

What problem does this PR solve?

Issue Number: close #65813

Problem Summary:
Currently, TiDB's optimizer cannot effectively utilize prefix indexes for ORDER BY ... LIMIT queries. When a query uses ORDER BY on a column with a prefix index, the optimizer performs a full table scan, leading to significant performance issues for large tables.

What changed and how does it work?

1. Extended Data Structures

Added partial order support to physical operators:
PhysicalTopN/PhysicalLimit

// PhysicalTopN (TiDB side)
type PhysicalTopN/PhysicalLimit struct {
    // ... existing fields ...
    
    // Partial order fields
    PartialOrderedLimit uint64  // Count + Offset
    PrefixColID         int64   // Column UniqueID
    PrefixLen           int     // Prefix length in bytes
}

2. Planner Flow

The optimization is integrated into the existing planner pipeline:

                 └─> skylinePruning
                       └─> matchPartialOrderProperty: Check if prefix index matches
                             - Match success: Set PrefixColID/PrefixLen, generate candidate
                             - Match failure: Skip this path
                 └─> convertToIndexScan
                       └─> Set KeepOrder=true for IndexScan
                       └─> Generate IndexLookUp if needed
                 └─> PhysicalTopN.Attach2Task
                       └─> handlePartialOrderTopN
                             └─> Two-layer: TiDB TopN + TiKV Limit (if can push down)
                             └─> One-layer: TiDB TopN only (if cannot push down)

3. Key Functions

matchPartialOrderProperty(): Validates prefix index match
- Sets: PrefixColID and PrefixLen in PartialOrderInfo
handlePartialOrderTopN(): Generates execution plan
- Decides two-layer vs one-layer based on CopTask state
- Updates stats for both Limit and TopN operators

4. Two-Layer vs One-Layer Plans

Two-layer mode :

TopN (TiDB: final sort + short-circuit)
└── IndexLookUp
    ├── Limit (TiKV: prefix-based short-circuit)
    │   └── IndexScan (keep_order=true)
    └── TableScan

One-layer mode (when IndexPlanFinished=true or has RootTaskConds):

TopN (TiDB: sort + short-circuit)
└── IndexLookUp
    ├── IndexScan (keep_order=true)
    └── TableScan

5. session variable behavior

tidb_opt_partial_ordered_index_for_topn : range: disable, cost.

disable: Completely disables optimization.
cost: Only supports combining the use index hint and the cost parameter together. To allow users to specify partial order + index.
2.1 ：cost + use index + match partial order property: force use the index with partial order optimize
2.2: cost + use index + not match partial order property: degenerate into normal index use behavior without considering partial order optimization.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

…rder flow

ti-chi-bot · 2026-01-26T04:13:26Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tiprow · 2026-01-26T04:13:39Z

Hi @elsa0520. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hawkingrei · 2026-01-26T10:29:13Z

/ok-to-test

Copilot

Pull request overview

This PR implements core planner changes for partial order TopN optimization with prefix indexes, enabling the optimizer to use prefix indexes for ORDER BY ... LIMIT queries to avoid full table scans.

Changes:

Extended PhysicalTopN and PhysicalLimit operators with partial order fields (PartialOrderedLimit, PrefixColID, PrefixLen)
Added PartialOrderInfo structure to PhysicalProperty to track partial order requirements through the planning pipeline
Implemented two-layer execution mode (TiDB TopN + TiKV Limit) and one-layer mode (TiDB TopN only) in handlePartialOrderTopN
Modified matchPartialOrderProperty to validate and set prefix index parameters
Updated GetOriginalPhysicalIndexScan signature to accept needKeepOrder parameter handling both normal and partial order cases

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/planner/property/physical_property.go	Adds `PartialOrderInfo` structure with prefix column tracking fields and helper methods (`NeedKeepOrder`, `GetSortDesc`, `GetSortItemsForKeepOrder`)
pkg/planner/core/task.go	Implements `handlePartialOrderTopN` for two-layer/one-layer plan generation and `estimateMaxXForPartialOrder` stub
pkg/planner/core/operator/physicalop/physical_topn.go	Adds partial order fields to `PhysicalTopN` and updates `MemoryUsage`, `ExplainInfo` methods
pkg/planner/core/operator/physicalop/physical_limit.go	Adds partial order fields to `PhysicalLimit` and updates `MemoryUsage`, `ExplainInfo` methods
pkg/planner/core/operator/physicalop/physical_index_scan.go	Updates `GetOriginalPhysicalIndexScan` to accept `needKeepOrder` parameter and use `GetSortDesc()` for both cases
pkg/planner/core/find_best_task.go	Refactors `matchPartialOrderProperty` to update `PartialOrderInfo` fields directly; updates `convertToIndexScan` to use `NeedKeepOrder()` and `GetSortItemsForKeepOrder()`

pkg/planner/core/task.go

pkg/planner/property/physical_property.go

pkg/planner/core/task.go

codecov · 2026-01-26T10:54:19Z

Codecov Report

❌ Patch coverage is 84.40367% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.5623%. Comparing base (afbf7ac) to head (820c999).
⚠️ Report is 69 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #65799        +/-   ##
================================================
- Coverage   77.7931%   77.5623%   -0.2309%     
================================================
  Files          1999       1923        -76     
  Lines        544702     541027      -3675     
================================================
- Hits         423741     419633      -4108     
- Misses       119297     121346      +2049     
+ Partials       1664         48      -1616

Flag	Coverage Δ
integration	`41.5297% <80.5555%> (-6.6443%)`	⬇️
unit	`76.7120% <79.8165%> (+0.2962%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`56.7974% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`47.9484% <ø> (-13.0294%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pkg/planner/core/operator/physicalop/physical_limit.go

pkg/planner/core/operator/physicalop/physical_index_scan.go

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

pkg/planner/core/find_best_task.go

pkg/planner/property/physical_property.go

pkg/planner/core/task.go

pkg/planner/core/operator/physicalop/physical_index_scan.go

pkg/planner/core/find_best_task.go

pkg/planner/core/task.go

Co-authored-by: Copilot <[email protected]>

qw4990 · 2026-02-03T01:03:15Z

/retest

elsa0520 · 2026-02-03T02:26:13Z

/unhold

terry1purcell · 2026-02-03T03:40:15Z

/hold

terry1purcell · 2026-02-03T03:40:28Z

@pantheon-bot please review

pantheon-ai · 2026-02-03T03:40:40Z

P2: Partial-order TopN can produce incorrect results when enabled

Severity: P2 because the optimization is opt-in (default DISABLE) via tidb_opt_partial_ordered_index_for_topn, so it won't affect production systems unless explicitly enabled.

Issue: The planner constructs a partial-order Limit with prefix metadata (PrefixCol, PrefixLen) to preserve TopN correctness by reading Count+Offset rows plus all rows sharing the last prefix. However, the pushdown protobuf only encodes a plain Limit count—TiKV will stop mid-prefix-group and drop candidate rows that could belong to the final TopN.

Evidence:

pkg/planner/core/task.go:1412 creates PhysicalLimit{Count: partialOrderedLimit, PrefixCol: p.PrefixCol, PrefixLen: matchResult.PrefixLen}
pkg/planner/core/operator/physicalop/physical_limit.go:155 builds tipb.Limit{Limit: p.Count} without any prefix metadata

Fix: Don't push down the limit until TiKV supports partial-ordered limit semantics, or extend the protobuf to include prefix fields and implement the "read N then continue until prefix changes" behavior in TiKV.

P2: Test incompatibility with new enum sysvar

Severity: P2 because it's a test consistency issue, not a production runtime breakage (invalid SET_VAR hints produce warnings, not errors).

Issue: The sysvar tidb_opt_partial_ordered_index_for_topn was changed to an enum accepting only DISABLE/COST, but test code still uses legacy boolean values (ON/OFF/0/1) which will fail validation.

Evidence:

pkg/sessionctx/variable/sysvar.go:3076 enforces enum-only validation
pkg/planner/core/hint_test.go:151 still uses set_var(tidb_opt_partial_ordered_index_for_topn=ON) expecting boolean behavior

Fix: Update pkg/planner/core/hint_test.go to use DISABLE/COST values and adjust test expectations accordingly.

P2: AccessPath.Clone() drops ForcePartialOrder flag

Severity: P2 because the reachability is unclear (flag is only set during skyline pruning for partial-order TopN, which is gated behind opt-in feature), but it's a completeness bug that could cause inconsistent hint enforcement.

Issue: The AccessPath.Clone() method copies ForceKeepOrder and ForceNoKeepOrder flags but omits the new ForcePartialOrder field, so cloned paths silently lose forced partial-order behavior.

Evidence:

pkg/planner/util/path.go:139 defines ForcePartialOrder field
pkg/planner/util/path.go:193 clone copies other force flags but omits ForcePartialOrder

Fix: Add ForcePartialOrder: path.ForcePartialOrder to the (*AccessPath).Clone() return struct.

elsa0520 · 2026-02-03T05:49:17Z

P2: Partial-order TopN can produce incorrect results when enabled

The connect part code not include this PR because the tipb change has not been merged.

elsa0520 · 2026-02-03T05:49:50Z

P2: Test incompatibility with new enum sysvar

The test has been changed in this PR

elsa0520 · 2026-02-03T05:52:55Z

P2: AccessPath.Clone() drops ForcePartialOrder flag

Has been changed in PR

elsa0520 · 2026-02-03T06:21:45Z

/test mysql-test

ti-chi-bot · 2026-02-03T06:21:49Z

@elsa0520: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-build-next-gen

/test pull-integration-e2e-test

/test pull-integration-realcluster-test-next-gen

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-mysql-client-test-next-gen

/test pull-unit-test-ddlv1

/test pull-unit-test-next-gen

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-br-integration-test-next-gen

/test pull-check-deps

/test pull-common-test

/test pull-e2e-test

/test pull-error-log-review

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-ddl-test

/test pull-integration-ddl-test-next-gen

/test pull-integration-e2e-test-next-gen

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-mysql-test-next-gen

/test pull-sqllogic-test

/test pull-tiflash-integration-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_build_next_gen

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_integration_realcluster_test_next_gen

pingcap/tidb/pull_mysql_client_test

pingcap/tidb/pull_mysql_client_test_next_gen

pingcap/tidb/pull_unit_test_next_gen

pull-check-deps

pull-error-log-review

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2026-02-03T06:22:09Z

@elsa0520: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow

/test tidb_parser_test

Use /test all to run all jobs.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

elsa0520 · 2026-02-03T06:22:12Z

/test mysql-test

tiprow · 2026-02-03T06:22:36Z

@elsa0520: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow

/test tidb_parser_test

Use /test all to run all jobs.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

elsa0520 · 2026-02-03T08:01:51Z

/unhold

0xPoe · 2026-02-03T09:26:46Z

/retest

tiprow · 2026-02-03T09:31:48Z

@elsa0520: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tidb_parser_test	`820c999`	link	true	`/test tidb_parser_test`
fast_test_tiprow	`820c999`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2026-02-03T16:10:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qw4990, winoros, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [qw4990,winoros]
~~pkg/planner/OWNERS~~ [qw4990,winoros]
~~pkg/sessionctx/vardef/OWNERS~~ [yudongusa]
~~pkg/sessionctx/variable/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

planner, core: implement partial order TopN attach2Task and partial o…

dc8eee9

…rder flow

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Jan 26, 2026

fix some refactor

8502689

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Jan 26, 2026

elsa0520 requested a review from winoros January 26, 2026 10:23

elsa0520 marked this pull request as ready for review January 26, 2026 10:23

Copilot AI review requested due to automatic review settings January 26, 2026 10:23

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 26, 2026

Copilot started reviewing on behalf of elsa0520 January 26, 2026 10:23 View session

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

pkg/planner/core/task.go Outdated Show resolved Hide resolved

pkg/planner/property/physical_property.go Outdated Show resolved Hide resolved

pkg/planner/core/task.go Outdated Show resolved Hide resolved

pkg/planner/core/task.go Outdated Show resolved Hide resolved

qw4990 reviewed Jan 27, 2026

View reviewed changes

pkg/planner/core/operator/physicalop/physical_limit.go Outdated Show resolved Hide resolved

qw4990 reviewed Jan 27, 2026

View reviewed changes

pkg/planner/core/operator/physicalop/physical_index_scan.go Outdated Show resolved Hide resolved

Update pkg/planner/core/task.go

e10e107

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings January 27, 2026 04:17

Update pkg/planner/property/physical_property.go

331756d

Co-authored-by: Copilot <[email protected]>

Copilot started reviewing on behalf of elsa0520 January 27, 2026 04:18 View session

Copilot AI reviewed Jan 27, 2026

View reviewed changes

Update pkg/planner/core/find_best_task.go

e7e2bbd

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings January 27, 2026 07:53

elsa0520 added 4 commits February 2, 2026 18:23

fix bug of use the origin index column instead path.idxcol

3eac73a

add hint in all test

f3d7e55

add test

9332588

fix lint

b15eb90

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2026

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2026

fix build and ut

5af179f

elsa0520 force-pushed the partial_order_by branch from 3f0df54 to 5af179f Compare February 3, 2026 05:42

fix clone problem

820c999

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2026

yudongusa approved these changes Feb 3, 2026

View reviewed changes

ti-chi-bot bot added the approved label Feb 3, 2026

ti-chi-bot bot merged commit 715b1cc into pingcap:master Feb 3, 2026
28 of 30 checks passed

planner, core: implement partial order TopN attach2Task and partial order flow #65799

planner, core: implement partial order TopN attach2Task and partial order flow #65799

Uh oh!

Conversation

elsa0520 commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot bot commented Jan 26, 2026

Uh oh!

tiprow bot commented Jan 26, 2026

Uh oh!

hawkingrei commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qw4990 commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

terry1purcell commented Feb 3, 2026

Uh oh!

terry1purcell commented Feb 3, 2026

Uh oh!

pantheon-ai bot commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

ti-chi-bot bot commented Feb 3, 2026

Uh oh!

tiprow bot commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

tiprow bot commented Feb 3, 2026

Uh oh!

elsa0520 commented Feb 3, 2026

Uh oh!

0xPoe commented Feb 3, 2026

Uh oh!

tiprow bot commented Feb 3, 2026

Uh oh!

ti-chi-bot bot commented Feb 3, 2026

Uh oh!

Uh oh!

elsa0520 commented Jan 26, 2026 •

edited

Loading

codecov bot commented Jan 26, 2026 •

edited

Loading

pantheon-ai bot commented Feb 3, 2026 •

edited

Loading