feat: Add in-commit timestamp support for change data feed #617

OussamaSaoudi-db · 2024-12-30T22:51:42Z

What changes are proposed in this pull request?

This is a Stacked PR. Please look at the latest commit in the branch!

This adds support for in-commit timestamps when performing change data feed. Now when a commit contains commitInfo with inCommitTimestamp, that timestamp will be the one used for all changed rows in the commit.

Please only review these commits.

How was this change tested?

Add tests to check that the timestamp extracted from commits containing in-commit-timestamps are the ICT instead of file modification time.

codecov · 2024-12-30T22:56:52Z

Codecov Report

Attention: Patch coverage is 91.60305% with 22 lines in your changes missing coverage. Please review.

Project coverage is 84.87%. Comparing base (1f4e4c0) to head (680ba1c).

Files with missing lines	Patch %	Lines
kernel/src/table_changes/log_replay.rs	83.72%	5 Missing and 9 partials ⚠️
kernel/src/actions/visitors.rs	41.66%	0 Missing and 7 partials ⚠️
kernel/src/table_changes/scan.rs	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #617      +/-   ##
==========================================
+ Coverage   84.78%   84.87%   +0.08%     
==========================================
  Files          88       88              
  Lines       22605    22758     +153     
  Branches    22605    22758     +153     
==========================================
+ Hits        19166    19316     +150     
- Misses       2459     2460       +1     
- Partials      980      982       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

scovich

LGTM, one nit

kernel/src/table_changes/log_replay.rs

zachschuermann

few things looks good tho!

zachschuermann · 2025-01-09T16:29:23Z

kernel/src/table_changes/log_replay.rs

                add_paths: &mut add_paths,
                remove_dvs: &mut remove_dvs,
                has_cdc_action: &mut has_cdc_action,
+                commit_timestamp: &mut timestamp,


would this be clearer?

Suggested change

commit_timestamp: &mut timestamp,

in_commit_timestamp: &mut timestamp,

We initialize this field with the file modification timestamp, so it would be inaccurate to call it that. I do like the update you made below tho when we actually read ICT from a commitinfo.

zachschuermann · 2025-01-09T16:32:21Z

kernel/src/table_changes/log_replay.rs

    /// 2. Construct a map from path to deletion vector of remove actions that share the same path
    ///    as an add action.
    /// 3. Perform validation on each protocol and metadata action in the commit.
+    /// 4. Extract the in-commit timestamp from [`CommitInfo`] if it is present.


I can't comment on L130 above but I think we need to do some comment updates?

Good catch! I went through every mention of ICT and I think I got them all.

kernel/src/table_changes/log_replay.rs

zachschuermann · 2025-01-09T16:39:35Z

kernel/src/table_changes/log_replay/tests.rs

+            Action::CommitInfo(CommitInfo {
+                in_commit_timestamp: Some(timestamp),
+                ..Default::default()
+            }),


what happens if commit info isn't first? do we still read it? I know the protocol says it must be first with ICT enabled but I wonder what the expected behavior is when it isn't first? do we do the right thing?

(but probably don't solve here)

Discussed a little here:
#581 (comment)

I'm still quite certain that delta-spark doesn't care about the ordering because it goes through the all actions in the commit looking for CommitInfo

var commitInfo: Option[CommitInfo] = None actions.foreach { case c: AddCDCFile => cdcActions.append(c) totalFiles += 1L totalBytes += c.size case a: AddFile => totalFiles += 1L totalBytes += a.size case r: RemoveFile => totalFiles += 1L totalBytes += r.size.getOrElse(0L) case i: CommitInfo => commitInfo = Some(i) case _ => // do nothing }

I've added a check that only puts in the ICT if it is the first action in the log, but there comes a question: should we fail if it isn't the first action?

I can also revert the check that CommitInfo is first and revisit that in a future PR.

I'd prefer to be strict. ICT isn't widely adopted yet, so hopefully we don't have too many bad actors yet either. If kernel-rs is strict that will deter future bad actors.

zachschuermann · 2025-01-09T16:42:13Z

kernel/src/table_changes/log_replay.rs

            Option::<Cdc>::get_struct_field(CDC_NAME),
            Option::<Metadata>::get_struct_field(METADATA_NAME),
            Option::<Protocol>::get_struct_field(PROTOCOL_NAME),
+            StructField::new("commitInfo", StructType::new([ict_type]), true),


though i wonder if we can do something similar to above like Option<CommitInfo>::get_struct_field(COMMIT_INFO_NAME) and get struct field inCommitTimestamp of that?

but for now at least can use COMMIT_INFO_NAME?

Suggested change

StructField::new("commitInfo", StructType::new([ict_type]), true),

StructField::new(COMMIT_INFO_NAME, StructType::new([ict_type]), true),

i wonder if we can do something similar to above like Option::get_struct_field(COMMIT_INFO_NAME) and get struct field inCommitTimestamp of that?

We would get a StructField of type CommitInfo, which we'd have to 1) get datatype, 2) cast to a struct 3) get the ICT field. So I'll stick with your suggested change 👍

zachschuermann · 2025-01-09T16:44:08Z

kernel/src/table_changes/scan_file.rs

+                Action::Cdc(cdc.clone()),
+                Action::CommitInfo(commit_info.clone()),


are these ordered? should commit info be first?

swapped ordering

scovich · 2025-04-15T18:47:11Z

kernel/src/table_changes/log_replay.rs

+                if self.is_first_batch && i == 0 {
+                    *self.commit_timestamp = in_commit_timestamp;
+                }


Relating to the other thread -- to enforce that commit info is first, we would just need an else here that returns Err?

Suggested change

if self.is_first_batch && i == 0 {

*self.commit_timestamp = in_commit_timestamp;

}

if self.is_first_batch && i == 0 {

*self.commit_timestamp = in_commit_timestamp;

} else {

return Err(...);

}

or even use require! macro?

Suggested change

if self.is_first_batch && i == 0 {

*self.commit_timestamp = in_commit_timestamp;

}

require!(self.is_first_batch && i == 0, Error::Something(...));

*self.commit_timestamp = in_commit_timestamp;

Went with the require macro. 👍

github-actions bot assigned OussamaSaoudi-db Dec 30, 2024

OussamaSaoudi-db requested review from scovich, nicklan and zachschuermann December 30, 2024 22:55

OussamaSaoudi-db marked this pull request as ready for review December 30, 2024 22:55

OussamaSaoudi-db changed the title ~~feat: Add in-commit timestamp support for change data fede~~ feat: Add in-commit timestamp support for change data feed Jan 2, 2025

scovich approved these changes Jan 6, 2025

View reviewed changes

kernel/src/table_changes/log_replay.rs Outdated Show resolved Hide resolved

kernel/src/table_changes/log_replay.rs Outdated Show resolved Hide resolved

kernel/src/table_changes/log_replay.rs Outdated Show resolved Hide resolved

zachschuermann approved these changes Jan 9, 2025

View reviewed changes

OussamaSaoudi-db force-pushed the cdf_ict_impl branch 2 times, most recently from 7242904 to f50c202 Compare January 10, 2025 00:13

OussamaSaoudi added the merge hold Don't allow the PR to merge label Jan 10, 2025

OussamaSaoudi force-pushed the cdf_ict_impl branch from 2831885 to d295ffc Compare February 4, 2025 01:18

github-actions bot added the breaking-change Change that require a major version bump label Feb 4, 2025

OussamaSaoudi force-pushed the cdf_ict_impl branch from d295ffc to 1b16c4c Compare April 12, 2025 22:53

OussamaSaoudi requested a review from zachschuermann April 15, 2025 17:02

scovich reviewed Apr 15, 2025

View reviewed changes

roeap mentioned this pull request Mar 8, 2025

inCommitTimestamps feature delta-io/delta-rs#3253

Open

OussamaSaoudi force-pushed the cdf_ict_impl branch 2 times, most recently from 0230327 to 20228f6 Compare April 30, 2025 22:00

Use table config in cdf

eec0cfc

OussamaSaoudi force-pushed the cdf_ict_impl branch 2 times, most recently from c745a0c to 94bbe54 Compare May 2, 2025 15:41

Add ICT support to table changes

9f44537

OussamaSaoudi force-pushed the cdf_ict_impl branch from 94bbe54 to 9f44537 Compare May 2, 2025 15:51

hntd187 added 4 commits May 6, 2025 09:07

Merge branch 'main' into cdf_ict_impl

fe37b7e

Merge branch 'main' into cdf_ict_impl

4b79bc5

Update log_replay.rs

680ba1c

Merge branch 'main' into cdf_ict_impl

f5280c6

OussamaSaoudi self-assigned this Jul 22, 2025

OussamaSaoudi unassigned OussamaSaoudi-db Jul 22, 2025

OussamaSaoudi mentioned this pull request Aug 18, 2025

[EPIC] Change Data Feed read path improvements #1177

Open

	commit_timestamp: &mut timestamp,
	in_commit_timestamp: &mut timestamp,

	StructField::new("commitInfo", StructType::new([ict_type]), true),
	StructField::new(COMMIT_INFO_NAME, StructType::new([ict_type]), true),

		Action::Cdc(cdc.clone()),
		Action::CommitInfo(commit_info.clone()),

feat: Add in-commit timestamp support for change data feed #617

Are you sure you want to change the base?

feat: Add in-commit timestamp support for change data feed #617

Uh oh!

Conversation

OussamaSaoudi-db commented Dec 30, 2024 • edited by OussamaSaoudi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this change tested?

Uh oh!

codecov bot commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zachschuermann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

OussamaSaoudi-db commented Dec 30, 2024 •

edited by OussamaSaoudi

Loading

codecov bot commented Dec 30, 2024 •

edited

Loading