test: Port cdf tests from delta-spark to kernel

[OussamaSaoudi-db]

What changes are proposed in this pull request?

This PR adds several CDF tests from delta-spark. We check the following:

• CDF over various version ranges
• Update operations are read correctly from cdc files
• data_change=false means the action is skipped
• A range with start > end is an error.
• Start version greater than latest table version is an error
• CDF works on partition tables
• CDF works on tables with backticks in the column names
• CDF is correct in deletion cases with unconditional deletes, conditional deletes that remove all rows, and selective conditional deletes.

Table-changes construction is also changed so that CDF version error is checked before snapshots are created. This makes the error message clearer in the case that the start version is beyond the end of the table.

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Project coverage is 83.70%. Comparing base (c1c1dbe) to head (8156f9b).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/table_changes/mod.rs	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #611      +/-   ##
==========================================
+ Coverage   83.67%   83.70%   +0.02%     
==========================================
  Files          75       75              
  Lines       16950    16951       +1     
  Branches    16950    16951       +1     
==========================================
+ Hits        14183    14188       +5     
  Misses       2100     2100              
+ Partials      667      663       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nicklan]

looks great thanks! One small thing, could you recreate the archives with --no-xattrs passed to tar? Otherwise if you look at these zstd files on a non-macos machine you get lots of:
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'

tar -c --no-xattrs [etc] should do the trick

[nicklan]

I tend to prefer putting the expected output along with the data (like we do with dat) rather than spreading it out like this. I think this is okay for now though, and maybe we can make an issue to port each test by recreating the archive with an expected data parquet.

[OussamaSaoudi-db]

done: #626

[zachschuermann]

this reminds me: we should make an issue (or perhaps we already have one?) to fix all the tests and move away from string matching and instead compare underlying (sorted) data/metadata

cc @nicklan

[OussamaSaoudi-db]

@nicklan rebuilt with --no-xattrs. Seems to have caused a diff on github, but I'm still seeing com.apple.provenance in when calling xattr. Does it still cause the error?

[zachschuermann]

did all the expected output come from delta-spark? if yes, then LGTM

[zachschuermann]

this reminds me: we should make an issue (or perhaps we already have one?) to fix all the tests and move away from string matching and instead compare underlying (sorted) data/metadata

cc @nicklan

[zachschuermann]

how are we observing update_pre and update_post here then? aren't we reading the CDF and then filling in our own update_preimage etc.?

[OussamaSaoudi-db]

All update _change_types come directly from a cdc file. We don't insert or modify them. In this test, delta-spark wrote update_pre and update_post directly into the cdc file.

[OussamaSaoudi-db]

@zachschuermann regarding creating an issue, I already put one up for CDF here #626, but I can expand its scope to remove all string matching from our testing.

[OussamaSaoudi-db]

@zachschuermann My methodology is as follows: I used their expected results that looked like this (sample below), constructed the tables by hand, and only then did I verify it against our implementation.

        checkCDCAnswer(
          log,
          CDCReader.changesToBatchDF(log, 0, 2, spark).filter("_change_type = 'insert'"),
          Range(0, 6).map { i => Row(i, "old", i % 2, "insert", 0) })
        checkCDCAnswer(
          log,
          CDCReader.changesToBatchDF(log, 0, 2, spark).filter("_change_type = 'delete'"),
          Seq(0, 2, 3, 4).map { i => Row(i, "old", i % 2, "delete", if (i % 2 == 0) 2 else 1) })
        // ...