refactor!: update `ScanMetadata` to struct with new `FilteredEngineData` type #768

sebastiantia · 2025-03-26T19:50:13Z

What changes are proposed in this pull request?

Updated ScanMetata from typed tuple to struct. ScanMetadata is now a struct with fields:

filtered_data: A FilteredEngineData instance.
transforms: A vector of transformations to be applied to the data read from the files

Introduction of FilteredEngineData type:
Couples EngineData with a selection vector indicating which rows to process.
This type is returned from thescan_metadata API and the incoming checkpoint API
Updates visit_scan_files parameters to accept ScanMetadata to avoid de-structuring.
Corresponding FFI changes for visit_scan_files to accept ScanMetadata param

All current tests pass.

codecov · 2025-03-26T19:53:07Z

Codecov Report

Attention: Patch coverage is 53.62319% with 32 lines in your changes missing coverage. Please review.

Project coverage is 84.88%. Comparing base (10bdee7) to head (a6bc8d1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
ffi/src/scan.rs	0.00%	25 Missing ⚠️
kernel/src/scan/mod.rs	81.48%	0 Missing and 5 partials ⚠️
kernel/src/scan/log_replay.rs	85.71%	1 Missing ⚠️
kernel/src/scan/state.rs	90.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #768   +/-   ##
=======================================
  Coverage   84.87%   84.88%           
=======================================
  Files          83       83           
  Lines       20316    20311    -5     
  Branches    20316    20311    -5     
=======================================
- Hits        17244    17241    -3     
- Misses       2221     2222    +1     
+ Partials      851      848    -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nicklan · 2025-03-26T21:47:57Z

kernel/src/scan/mod.rs

+                let scan_data = res?;
+                let (data, sel_vec) = scan_data.filtered_data;
                let scan_files = vec![];
                state::visit_scan_files(


Should we have a visit_scan_data_files or similar that just takes the ScanData? Then we don't have to do this decomposition all over the place

How about updating visit_scan_files to just take ScanData?

discussed a little offline: i'm kinda partial to a ScanData.visit(callback, context)?

Implemented @zachschuermann 's approach

zachschuermann

few comments. and we need to ensure all the breaking changes are clearly spelled out in PR description

kernel/src/engine_data.rs

zachschuermann · 2025-03-26T23:09:20Z

kernel/src/scan/mod.rs

+                let scan_data = res?;
+                let (data, sel_vec) = scan_data.filtered_data;
                let scan_files = vec![];
                state::visit_scan_files(


discussed a little offline: i'm kinda partial to a ScanData.visit(callback, context)?

ffi/examples/read-table/read_table.c

kernel/src/engine_data.rs

kernel/src/scan/mod.rs

kernel/src/scan/state.rs

scovich

Approach LGTM. We just need to clean up all the nits.

kernel/src/scan/mod.rs

nicklan

lgtm, just one small nit

kernel/src/scan/mod.rs

zachschuermann

LGTM after a quick rename

ffi/src/scan.rs

## What changes are proposed in this pull request? Rename `ScanData` to `ScanMetadata` and `Scan::scan_data` to `Scan::scan_metadata` (and corresponding FFI). Additionally, renames `TableChangesScanData` to `TableChangesScanMetadata`. Additional docs/refactor coming in #768 ### This PR affects the following public APIs breaking changes: 1. rename `ScanData` to `ScanMetadata` 2. rename `Scan::scan_data()` to `Scan::scan_metadata()` 3. (ffi) rename `free_kernel_scan_data()` to `free_scan_metadata_iter()` 4. (ffi) rename `kernel_scan_data_next()` to `scan_metadata_next()` 5. (ffi) rename `visit_scan_data()` to `visit_scan_metadata()` 6. (ffi) rename `kernel_scan_data_init()` to `scan_metadata_iter_init()` 7. (ffi) rename `KernelScanDataIterator` to `ScanMetadataIterator` 8. (ffi) rename `SharedScanDataIterator` to `SharedScanMetadataIterator` ## How was this change tested? existing resolves #816

refactor

714428d

github-actions bot assigned sebastiantia Mar 26, 2025

sebastiantia added the breaking-change Change that require a major version bump label Mar 26, 2025

sebastiantia changed the title ~~refactor!: make ScanData struct with new FilteredEngineData type~~ refactor!: update ScanData to struct with new FilteredEngineData type Mar 26, 2025

sebastiantia requested review from nicklan and zachschuermann March 26, 2025 20:04

sebastiantia marked this pull request as ready for review March 26, 2025 20:09

nicklan reviewed Mar 26, 2025

View reviewed changes

update doc

92a820e

sebastiantia requested a review from nicklan March 26, 2025 22:57

zachschuermann reviewed Mar 26, 2025

View reviewed changes

sebastiantia added 5 commits March 26, 2025 17:36

wip

773df8f

fixes

2180603

c

83a876e

print selection vector

c281162

free sel vec

473a548

sebastiantia requested a review from zachschuermann March 27, 2025 02:42

remove unneeded method

48c2bdc

sebastiantia mentioned this pull request Mar 28, 2025

feat: add CheckpointLogReplayProcessor in new checkpoints mod #744

Merged

zachschuermann requested a review from scovich April 7, 2025 22:02

zachschuermann reviewed Apr 9, 2025

View reviewed changes

scovich approved these changes Apr 9, 2025

View reviewed changes

kernel/src/scan/mod.rs Outdated Show resolved Hide resolved

sebastiantia added 5 commits April 8, 2025 22:25

fix formatting

f50b299

fix c formatting

bdd8720

nits & review

6c5139b

merge

6e0c9f5

fix c

e676d5e

zachschuermann mentioned this pull request Apr 9, 2025

release 0.9.0 #815

Merged

sebastiantia requested a review from zachschuermann April 9, 2025 17:56

zachschuermann mentioned this pull request Apr 9, 2025

refactor!: rename ScanData to ScanMetadata #817

Merged

zachschuermann reviewed Apr 9, 2025

View reviewed changes

kernel/src/scan/mod.rs Outdated Show resolved Hide resolved

fix doc

12514c6

sebastiantia requested a review from zachschuermann April 9, 2025 18:16

sebastiantia added 3 commits April 9, 2025 11:17

docs

8ffe070

remove getter

c7618fc

clarify

ddba939

nicklan approved these changes Apr 9, 2025

View reviewed changes

kernel/src/scan/mod.rs Show resolved Hide resolved

zachschuermann approved these changes Apr 9, 2025

View reviewed changes

ffi/src/scan.rs Outdated Show resolved Hide resolved

rename to SharedScanData

d75bc64

sebastiantia added 5 commits April 9, 2025 16:18

docs

86d03af

merge

267cffb

fix ffi

90354a3

fmt

1b07cfd

fix

3b55ae1

sebastiantia changed the title ~~refactor!: update ScanData to struct with new FilteredEngineData type~~ refactor!: update ScanMetadata to struct with new FilteredEngineData type Apr 9, 2025

comments

a6bc8d1

sebastiantia merged commit b38bc5d into delta-io:main Apr 10, 2025
20 of 21 checks passed

sebastiantia deleted the seb/scan-data-refactor-with-filtered-engine-data branch April 10, 2025 00:06

refactor!: update ScanMetadata to struct with new FilteredEngineData type #768

refactor!: update ScanMetadata to struct with new FilteredEngineData type #768

Uh oh!

Conversation

sebastiantia commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

Uh oh!

codecov bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

nicklan Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

sebastiantia Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

zachschuermann Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

sebastiantia Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

zachschuermann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zachschuermann Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nicklan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zachschuermann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

refactor!: update `ScanMetadata` to struct with new `FilteredEngineData` type #768

refactor!: update `ScanMetadata` to struct with new `FilteredEngineData` type #768

sebastiantia commented Mar 26, 2025 •

edited

Loading

codecov bot commented Mar 26, 2025 •

edited

Loading