Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding predicate to protocol and metadata log replay query #336

Merged
merged 13 commits into from
Sep 18, 2024
Merged

Adding predicate to protocol and metadata log replay query #336

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
dfdd756
Pass predicate in snapshot replay to filter metadata or protocol
OussamaSaoudi-db Sep 13, 2024
cf95385
Add clarifying comment for filter, rename filter variable
OussamaSaoudi-db Sep 13, 2024
0dc7fb2
Fix column names for filter
OussamaSaoudi-db Sep 13, 2024
c31a716
Change unary() call to just use not()
OussamaSaoudi-db Sep 13, 2024
59c4988
Use Expr associated type instead of Expression
OussamaSaoudi-db Sep 16, 2024
a9b2375
Validate that predicate is correctly passed to engine
OussamaSaoudi-db Sep 17, 2024
2d763b5
Amend acceptance tests
OussamaSaoudi-db Sep 17, 2024
911437e
Move print test to non-acceptance test
OussamaSaoudi-db Sep 17, 2024
fe77903
Remove unnecessary change
OussamaSaoudi-db Sep 17, 2024
49889e7
Address PR comments
OussamaSaoudi-db Sep 17, 2024
e461695
Remove unecessary changes in default engine
OussamaSaoudi-db Sep 17, 2024
7868f58
Remove unused imports
OussamaSaoudi-db Sep 17, 2024
beb9850
Move to tracing_subscriber
OussamaSaoudi-db Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions kernel/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,4 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [
"env-filter",
"fmt",
] }
env_logger = "0.11.5"
OussamaSaoudi-db marked this conversation as resolved.
Show resolved Hide resolved
7 changes: 5 additions & 2 deletions kernel/src/engine/sync/json.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,12 @@ impl JsonHandler for SyncJsonHandler {
&self,
files: &[FileMeta],
schema: SchemaRef,
_predicate: Option<Expression>,
predicate: Option<Expression>,
) -> DeltaResult<FileDataReadResultIterator> {
debug!("Reading json files: {:#?}", files);
debug!(
OussamaSaoudi-db marked this conversation as resolved.
Show resolved Hide resolved
"Reading json files: {:#?} with predicate {:#?}",
files, predicate
);
if files.is_empty() {
return Ok(Box::new(std::iter::empty()));
}
Expand Down
4 changes: 2 additions & 2 deletions kernel/src/engine/sync/parquet.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ impl ParquetHandler for SyncParquetHandler {
&self,
files: &[FileMeta],
schema: SchemaRef,
_predicate: Option<Expression>,
predicate: Option<Expression>,
) -> DeltaResult<FileDataReadResultIterator> {
debug!("Reading parquet files: {files:#?} with schema {schema:#?}");
debug!("Reading parquet files: {files:#?} with schema {schema:#?} and predicate {predicate:#?}");
if files.is_empty() {
return Ok(Box::new(std::iter::empty()));
}
Expand Down
1 change: 1 addition & 0 deletions kernel/src/scan/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,7 @@ mod tests {

#[test]
fn test_scan_data() {
env_logger::init();
let path =
std::fs::canonicalize(PathBuf::from("./tests/data/table-without-dv-small/")).unwrap();
let url = url::Url::from_directory_path(path).unwrap();
Expand Down
12 changes: 10 additions & 2 deletions kernel/src/snapshot.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
//!

use std::cmp::Ordering;
use std::ops::Not;
use std::sync::Arc;

use itertools::Itertools;
Expand Down Expand Up @@ -71,9 +72,16 @@ impl LogSegment {

fn read_metadata(&self, engine: &dyn Engine) -> DeltaResult<Option<(Metadata, Protocol)>> {
let schema = get_log_schema().project(&[PROTOCOL_NAME, METADATA_NAME])?;
// filter out log files that do not contain metadata or protocol information
use Expression as Expr;
OussamaSaoudi-db marked this conversation as resolved.
Show resolved Hide resolved
let filter = Some(Expr::or(
Expr::not(Expr::is_null(Expr::Column("metaData.id".into()))),
Expr::not(Expr::is_null(Expr::Column(
"protocol.min_reader_version".into(),
))),
));
OussamaSaoudi-db marked this conversation as resolved.
Show resolved Hide resolved
// read the same protocol and metadata schema for both commits and checkpoints
// TODO add metadata.table_id is not null and protocol.something_required is not null
let data_batches = self.replay(engine, schema.clone(), schema, None)?;
let data_batches = self.replay(engine, schema.clone(), schema, filter)?;
let mut metadata_opt: Option<Metadata> = None;
let mut protocol_opt: Option<Protocol> = None;
for batch in data_batches {
Expand Down
Loading