- 
                Notifications
    
You must be signed in to change notification settings  - Fork 119
 
Closed as duplicate of#1177
Description
Change Data Feed is a feature of the Delta protocol that allows users to scan all row-level changes that happened for each commit.
Completed
- Move log segment into separate module #438
 -  Add methods for constructing 
LogSegmentfor Snapshot and for TableChanges #495 - Add table changes constructor #505
 -  Implement Builder for Scans on 
TableChanges#521 -  Add 
Cdcaction for Change Data Feed #506 - Implement Log Replay for Change Data Feed #540
 - ScanFile expression and visitor for CDF #546
 - Resolve deletion vectors to find inserted and removed rows for CDF #568
 - Helper methods for CDF Physical to Logical Transformation #579
 - TableChangesScan::execute and end to end testing for CDF #580
 -  Add 
TableChangesScan::schemamethod to get logical schema #589 - Add fix for sv extension #591
 
Remaining
- additional testing (TODO issue)
 - [In progress] Allow CDF scans with schema evolution #523
 - support column mapping
 - ensure all protocol reader verision 1-3 supported and all reader features (col mapping, DVs, timestampNtz, vacuum protocol check) (note v2checkpoint not supported elsewhere yet)
 - [In progress] Add support for In-Commit Timestamps #559
 - type widening support: feat: Add type widening support to schema compatibility check #623
 - add a user guide (and revisit docs)
 
Future work (out of scope)
- (later) Support schema evolution for tables with column mapping enabled
 - (later) improve error message when performing CDF using a predicate that references a CDF generated column. Currently this fails, but the error message is not very clear.
 
dennyglee
Metadata
Metadata
Assignees
Labels
No labels