-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prepare to support parquet row group skipping #381
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #381 +/- ##
==========================================
+ Coverage 76.86% 77.06% +0.20%
==========================================
Files 47 47
Lines 9436 9524 +88
Branches 9436 9524 +88
==========================================
+ Hits 7253 7340 +87
- Misses 1789 1790 +1
Partials 394 394 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice thanks ryan, really like the new replay_for_*
LGTM!
@@ -122,6 +122,7 @@ fn read_parquet_file_impl( | |||
last_modified: file.last_modified, | |||
size: file.size, | |||
}; | |||
// TODO: Plumb the predicate through the FFI? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
created #382
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm! thanks
@@ -744,7 +748,7 @@ fn predicate_on_number_with_not_null() -> Result<(), Box<dyn std::error::Error>> | |||
"./tests/data/basic_partitioned", | |||
Some(&["a_float", "number"]), | |||
Some(Expression::and( | |||
Expression::not(Expression::column("number").is_null()), | |||
Expression::column("number").is_not_null(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so much nicer :)
In preparation for #362 that actually implements parquet row group skipping, here we make various preparatory changes that can stand on their own:
Expression::is_not_null
helper that does what it saysreplay_for_XXX
methods, so that log replay involving push-down predicates can be tested independently.This should make both changes easier to review.