feat: `validate_deleted_data_files` #1938

jayceslesar · 2025-04-19T19:42:19Z

Closes #1928

Rationale for this change

Add validate_deleted_data_files which depends on #1935

Are these changes tested?

Added a test!

References

Java deletedDataFiles impl:
https://github.com/apache/iceberg/blob/3a29199e73f2e9ae0f8f92a1a0732a338c66aa0d/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L678

Java ManifestGroup.entries impl:
https://github.com/apache/iceberg/blob/3a29199e73f2e9ae0f8f92a1a0732a338c66aa0d/core/src/main/java/org/apache/iceberg/ManifestGroup.java#L242

…_history`

jayceslesar · 2025-04-23T21:42:42Z

pyiceberg/table/update/validate.py

+            if entry.snapshot_id not in new_snapshot_ids:
+                continue
+
+            if entry.status != ManifestEntryStatus.DELETED:
+                continue
+
+            if data_filter is not None and not evaluator(entry.data_file):
+                continue
+
+            if partition_set is not None and (entry.data_file.spec_id, entry.data_file.partition) not in partition_set:
+                continue


probably cleaner to put these each on a line and wrap in an any call

sungwy · 2025-05-02T01:19:39Z

pyiceberg/table/update/validate.py

+    parent_snapshot: Optional[Snapshot],
+    partition_set: Optional[set[Record]],


This looks out of order:

Suggested change

parent_snapshot: Optional[Snapshot],

partition_set: Optional[set[Record]],

partition_set: Optional[set[Record]],

parent_snapshot: Optional[Snapshot],

It looks like the function call in validate_deleted_data_files is assuming that parent_snapshot is the last argument

conflicting_entries = deleted_data_files(table, starting_snapshot, data_filter, None, parent_snapshot)

I think this needs some more work. Next to @sungwy's comment, we in the code below:

(entry.data_file.spec_id, entry.data_file.partition) not in partition_set

Where it checks if a tuple is in a partition_set, but the partition_set only contains the Record according to the signature.

This triggered me, because if you do:

ALTER TABLE prod.db.taxis REPLACE PARTITION FIELD pickup_timestamp WITH day(pickup_timestamp); -- and then ALTER TABLE prod.db.taxis REPLACE PARTITION FIELD pickup_timestamp WITH day(dropoff_timestamp);

Both of the partitioning strategies will produce a Record[int] because it will contain the number of days since epoch. But the meaning is completely different.

sungwy · 2025-05-02T01:24:15Z

pyiceberg/table/update/validate.py

+        starting_snapshot,
+        parent_snapshot,


This looks out of order to, because validation_history has to_snapshot as the second argument, and from_snapshot as the third argument.

Suggested change

starting_snapshot,

parent_snapshot,

starting_snapshot,

parent_snapshot,

Do you think it would be better to update validation_history function to use the following function signature instead? I think it's a lot more expected to have from_snapshot then to_snapshot

def validation_history( table: Table, from_snapshot: Snapshot, to_snapshot: Snapshot, matching_operations: set[Operation], manifest_content_filter: ManifestContent, )

sungwy · 2025-05-02T01:42:51Z

pyiceberg/table/update/validate.py

+    )
+
+    if data_filter is not None:
+        evaluator = _StrictMetricsEvaluator(table.schema(), data_filter).eval


I'm not too sure if this is correct, because ManifestGroup.entries seems to be using inclusive projection.

Should we be using inclusive_projection here instead?

Summoning @Fokko for a second review

Yes, I agree that this should be inclusive projection, since we want to know if there are any matches. Inclusive projection returns rows_might_match and rows_cannot_match. If they cannot be matched, then we can skip it :)

Fokko

Thanks for following up on the previous PR @jayceslesar. This looks like a great start, but there are some missing elements. It would be good to add some more tests as well, since this is pretty critical code since it might affect correctness.

Fokko · 2025-05-02T08:24:36Z

pyiceberg/table/update/validate.py

@@ -69,3 +75,74 @@ def validation_history(
        raise ValidationException("No matching snapshot found.")

    return manifests_files, snapshots
+
+
+def deleted_data_files(


Can we mark these as private? I don't think we want to expose this directly to the user, and marking it private makes it easier to change the signature later on (since we don't have to go through the deprecation cycle).

Suggested change

def deleted_data_files(

def _deleted_data_files(

Fokko · 2025-05-02T08:29:34Z

pyiceberg/table/update/validate.py

+    parent_snapshot: Optional[Snapshot],
+    partition_set: Optional[set[Record]],


I think this needs some more work. Next to @sungwy's comment, we in the code below:

(entry.data_file.spec_id, entry.data_file.partition) not in partition_set

Where it checks if a tuple is in a partition_set, but the partition_set only contains the Record according to the signature.

This triggered me, because if you do:

ALTER TABLE prod.db.taxis REPLACE PARTITION FIELD pickup_timestamp WITH day(pickup_timestamp); -- and then ALTER TABLE prod.db.taxis REPLACE PARTITION FIELD pickup_timestamp WITH day(dropoff_timestamp);

Both of the partitioning strategies will produce a Record[int] because it will contain the number of days since epoch. But the meaning is completely different.

Fokko · 2025-05-02T08:47:01Z

pyiceberg/table/update/validate.py

+    """
+    conflicting_entries = deleted_data_files(table, starting_snapshot, data_filter, None, parent_snapshot)
+    if any(conflicting_entries):
+        raise ValidationException("Deleted data files were found matching the filter.")


Maybe nice to also add to the Exception which snapshot-id(s) are conflicting

Fokko · 2025-05-02T08:54:33Z

pyiceberg/table/update/validate.py

+        parent_snapshot: Ending snapshot on the branch being validated
+
+    Returns:
+        List of deleted data files matching the filter


Suggested change

List of deleted data files matching the filter

List of conflicting manifest-entries

Fokko · 2025-05-02T08:58:08Z

pyiceberg/table/update/validate.py

+    )
+
+    if data_filter is not None:
+        evaluator = _StrictMetricsEvaluator(table.schema(), data_filter).eval


Yes, I agree that this should be inclusive projection, since we want to know if there are any matches. Inclusive projection returns rows_might_match and rows_cannot_match. If they cannot be matched, then we can skip it :)

Fokko · 2025-05-02T08:58:48Z

pyiceberg/table/update/validate.py

+            if entry.status != ManifestEntryStatus.DELETED:
+                continue
+
+            if data_filter is not None and not evaluator(entry.data_file):


Maybe better to make it explicit:

Suggested change

if data_filter is not None and not evaluator(entry.data_file):

if data_filter is not None and not evaluator(entry.data_file) is ROWS_CANNOT_MATCH:

jayceslesar added 15 commits April 18, 2025 15:53

feat: validation history

beff92b

format

c369720

almost a working test

41bb8a4

allow content_filter in snapshot.manifests

763e9f4

simplify order of arguments to validation_history

f200beb

simplify return in snapshot.manifests

7f6bf9d

tests passing

c63cc55

correct ancestors_between

f2f3a88

fix to/from logic and allow optional to_snapshot arg in `validation…

74d5569

…_history`

remove a level of nesting with smarter clause

167f9e4

fix bad accessor

efe50b4

fix docstring

0793713

[wip] feat: validate_deleted_data_files

89120aa

first dummy test case

a39abd2

first working tests (needs cleanup)

cf5b061

jayceslesar commented Apr 23, 2025

View reviewed changes

jayceslesar added 3 commits May 1, 2025 09:07

Merge branch 'main' into feat/validate-deleted-data-files

645e8df

bring back all code from silly merge

caf7e36

last tweaks

a52c422

jayceslesar changed the title ~~[wip] feat: validate_deleted_data_files~~ feat: validate_deleted_data_files May 1, 2025

jayceslesar marked this pull request as ready for review May 2, 2025 00:40

sungwy reviewed May 2, 2025

View reviewed changes

Fokko reviewed May 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `validate_deleted_data_files` #1938

feat: `validate_deleted_data_files` #1938

jayceslesar commented Apr 19, 2025 •

edited

Loading

jayceslesar Apr 23, 2025

sungwy May 2, 2025

Fokko May 2, 2025

sungwy May 2, 2025

sungwy May 2, 2025

Fokko May 2, 2025

Fokko left a comment

Fokko May 2, 2025

Fokko May 2, 2025

Fokko May 2, 2025

Fokko May 2, 2025

Fokko May 2, 2025

Fokko May 2, 2025

		parent_snapshot: Optional[Snapshot],
		partition_set: Optional[set[Record]],

	List of deleted data files matching the filter
	List of conflicting manifest-entries

	if data_filter is not None and not evaluator(entry.data_file):
	if data_filter is not None and not evaluator(entry.data_file) is ROWS_CANNOT_MATCH:

feat: validate_deleted_data_files #1938

Are you sure you want to change the base?

feat: validate_deleted_data_files #1938

Conversation

jayceslesar commented Apr 19, 2025 • edited Loading

Rationale for this change

Are these changes tested?

References

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fokko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: `validate_deleted_data_files` #1938

feat: `validate_deleted_data_files` #1938

jayceslesar commented Apr 19, 2025 •

edited

Loading