Skip to content

feat: Add InCommitTimestamp support to ChangeDataFeed #1641

@OussamaSaoudi

Description

@OussamaSaoudi

Please describe why this is necessary.

InCommitTimestamps (ICTs) associates a monotonically increasing timestamp with each commit. Currently, CDF only extracts the file modification timestamp of the commit. The goal is to extract ICT from a commit only if ICT is enabled.

Describe the functionality you are proposing.

In the LogReplayScanner for table_changes, you must do the following:

  1. Apply the InCommitTimestamp visitor to commit action
  2. Query the latest table_configuration. Be mindful to query a newly constructed one if metadata or protocol has been updated.
  3. If the latest table_configuration has ICT enabled, then set the timestamp to be the ICT. Otherwise, default to the last_modified timestamp.

Additional context

No response

Metadata

Metadata

Assignees

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions