Add a 'prefetch' option to ParquetRecordBatchStream to load the next row group while decoding
#6676
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
commit 28b2bf1 Author: Mason Hall <[email protected]> Date: Fri Oct 18 15:49:31 2024 -0400 Cleaned up prefetch and added a test commit 3d7e018 Author: Mason Hall <[email protected]> Date: Fri Oct 18 13:32:22 2024 -0400 prefetch working
tustvold
reviewed
Nov 4, 2024
| }, | ||
| } | ||
| StreamState::Prefetch(batch_reader, f) => { | ||
| let mut noop_cx = Context::from_waker(futures::task::noop_waker_ref()); |
There was a problem hiding this comment.
What is the rationale for doing this?
There was a problem hiding this comment.
I wanted to avoid any potential overhead from using the real context when polling the future here. Since we're always returning Poll::Ready out of this state (or transitioning to another state), we don't need to rely on the real context to wake the main stream future.
I'm not an expert at async rust code though so if it would make more sense to do something else here I'm happy to make that change.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6559.
Rationale for this change
This improves performance when reading from filesystems with high latency and/or low bandwidth.
What changes are included in this PR?
This adds an option to
ParquetRecordBatchStreamto load the next row group while decoding the current one. This adds a new state calledPrefetchto the stream state. In this state, a future for the next row group is polled before returning data from the currentParquetRecordBatchReader.Are there any user-facing changes?
There is a new option for
ParquetRecordBatchStreamBuildercalledprefetchthat is set by a method calledwith_prefetch.