-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
memdb: prevent iterator invalidation #1563
base: master
Are you sure you want to change the base?
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: ekexium <[email protected]>
8ac9c7a
to
1a6d100
Compare
Signed-off-by: ekexium <[email protected]>
Signed-off-by: ekexium <[email protected]>
faf4853
to
4c16a14
Compare
Signed-off-by: ekexium <[email protected]>
916238e
to
e1a3b5a
Compare
Signed-off-by: ekexium <[email protected]>
74a0617
to
29fc98e
Compare
Signed-off-by: ekexium <[email protected]>
@@ -223,6 +226,19 @@ func (cp *MemDBCheckpoint) IsSamePosition(other *MemDBCheckpoint) bool { | |||
return cp.blocks == other.blocks && cp.offsetInBlock == other.offsetInBlock | |||
} | |||
|
|||
func (cp *MemDBCheckpoint) LessThan(cp2 *MemDBCheckpoint) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need comments for exported functions.
@@ -52,6 +52,13 @@ type ART struct { | |||
lastTraversedNode atomic.Uint64 | |||
hitCount atomic.Uint64 | |||
missCount atomic.Uint64 | |||
|
|||
// The counter of every write operation, used to invalidate iterators that were created before the write operation. | |||
SeqNo int |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SeqNo
seems not to express meaning of write operation
? How about use writeCnt
like names.
} | ||
snapshotIter = it.db.SnapshotIter(searchLower, it.upper) | ||
} | ||
defer snapshotIter.Close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we need to check if the newly created snapshotIter
contains the same snapshot compared with the previous one, so the whole snapshotBatchedIter
has a consistent snapshot semantics.
ref pingcap/tidb#59153
To prevent potential misuse and iterator invalidation, modify the iterators provided by ART memdb as follows:
Iter
andIterReverse
now comes with an extra check: it is invalidated immediately by any write operation to the memdb after the creation of the iterator. Attempting to use such an invalidated iterator will result in a panic.SnapshotIter
andSnapshotIterReverse
will be replaced byBatchedSnapshotIter
.2.1.
SnapshotIter
is different fromIter
that it can be valid after write operations, but only becomes invalid if a write operation modifies the "snapshot".2.2. We need to introduce
BatchedSnapshotIter
instead of directly modifyingSnapshotIter
becauseSnapshotIter
maintains internal states and pointers. Consider a situation where a write operation causes changes to the internal data structure making the pointers invalid, while the snapshot should remain valid.2.3.
SnapshotIter
andSnapshotIterReverse
are not removed now for compatibility.RBT is unchanged as it is no longer used.
Pipelined MemDB still doesn't support iterators as it was.
Performance
Iterator microbenchmark
TiDB union scan executor
BatchedSnapshotIter
SnapshotIter