Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memdb: prevent iterator invalidation #1563

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ekexium
Copy link
Contributor

@ekexium ekexium commented Jan 23, 2025

ref pingcap/tidb#59153

To prevent potential misuse and iterator invalidation, modify the iterators provided by ART memdb as follows:

  1. Iter and IterReverse now comes with an extra check: it is invalidated immediately by any write operation to the memdb after the creation of the iterator. Attempting to use such an invalidated iterator will result in a panic.
  2. SnapshotIter and SnapshotIterReverse will be replaced by BatchedSnapshotIter.
    2.1. SnapshotIter is different from Iter that it can be valid after write operations, but only becomes invalid if a write operation modifies the "snapshot".
    2.2. We need to introduce BatchedSnapshotIter instead of directly modifying SnapshotIter because SnapshotIter maintains internal states and pointers. Consider a situation where a write operation causes changes to the internal data structure making the pointers invalid, while the snapshot should remain valid.
    2.3. SnapshotIter and SnapshotIterReverse are not removed now for compatibility.

RBT is unchanged as it is no longer used.
Pipelined MemDB still doesn't support iterators as it was.

Performance

Iterator microbenchmark

go test -run=^$ -bench=BenchmarkSnapshotIter -benchtime=3s
goos: linux
goarch: amd64
pkg: github.com/tikv/client-go/v2/internal/unionstore
cpu: AMD Ryzen 9 9900X 12-Core Processor            
BenchmarkSnapshotIter/RBT-SnapshotIter-24                     1783           1826587 ns/op             144 B/op               2 allocs/op
BenchmarkSnapshotIter/ART-SnapshotIter-24                     1870           1718771 ns/op             496 B/op              11 allocs/op
BenchmarkSnapshotIter/ART-BatchedSnapshotIter-24              1120           2964170 ns/op          417461 B/op             370 allocs/op
BenchmarkSnapshotIter/ART-ForEachInSnapshot-24                1771           1832084 ns/op             496 B/op              11 allocs/op
PASS
ok          github.com/tikv/client-go/v2/internal/unionstore        14.598s

TiDB union scan executor
BatchedSnapshotIter

go test -run=^$ -bench=BenchmarkUnionScanRead -benchtime=10s
    3999           3063035 ns/op          793197 B/op           17578 allocs/op

SnapshotIter

go test -run=^$ -bench=BenchmarkUnionScanRead -benchtime=10s
    3862           2990516 ns/op          386942 B/op           17434 allocs/op

Copy link

ti-chi-bot bot commented Jan 23, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from ekexium, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 23, 2025
@ekexium ekexium force-pushed the memdb-prevent-iter-invalidation branch from 8ac9c7a to 1a6d100 Compare January 23, 2025 08:02
@ekexium ekexium force-pushed the memdb-prevent-iter-invalidation branch from faf4853 to 4c16a14 Compare January 23, 2025 11:30
Signed-off-by: ekexium <[email protected]>
@ekexium ekexium force-pushed the memdb-prevent-iter-invalidation branch from 916238e to e1a3b5a Compare January 23, 2025 12:27
Signed-off-by: ekexium <[email protected]>
@ekexium ekexium force-pushed the memdb-prevent-iter-invalidation branch from 74a0617 to 29fc98e Compare January 24, 2025 05:54
@ekexium ekexium marked this pull request as ready for review January 24, 2025 05:54
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 24, 2025
@ekexium ekexium requested review from you06 and cfzjywxk January 24, 2025 05:54
Signed-off-by: ekexium <[email protected]>
@@ -223,6 +226,19 @@ func (cp *MemDBCheckpoint) IsSamePosition(other *MemDBCheckpoint) bool {
return cp.blocks == other.blocks && cp.offsetInBlock == other.offsetInBlock
}

func (cp *MemDBCheckpoint) LessThan(cp2 *MemDBCheckpoint) bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need comments for exported functions.

@@ -52,6 +52,13 @@ type ART struct {
lastTraversedNode atomic.Uint64
hitCount atomic.Uint64
missCount atomic.Uint64

// The counter of every write operation, used to invalidate iterators that were created before the write operation.
SeqNo int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SeqNo seems not to express meaning of write operation? How about use writeCnt like names.

}
snapshotIter = it.db.SnapshotIter(searchLower, it.upper)
}
defer snapshotIter.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we need to check if the newly created snapshotIter contains the same snapshot compared with the previous one, so the whole snapshotBatchedIter has a consistent snapshot semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants