[wip] feat: Introduce TableConfiguration to jointly manage metadata, protocol, and column mapping

[OussamaSaoudi]

What changes are proposed in this pull request?

This PR introduces the TableConfiguration struct which is used to perform feature enablement checks on both the table's protocol and metadata.

Problem statement

To check that a feature is enabled, you often must check that a certain reader/writer feature is enabled and that a table property is set to true. For example, a writer must check both the delta.enableDeletionVectors table property, and check that the deletionVectors writer feature is present in the table's Protocol. Probing two disparate structs to do a single check is error-prone and may lead to these metadata/protocol checks to become out of sync. Moreover checks are performed in the CDF path, snapshot scan path, and in the read path. Thus there are many ways in which protocol and metadata checks can diverge with one another.

Put simply, the problems are:

1. Protocol check may diverge with the metadata for the same code path
2. P&M checks may diverge among different code paths
3. There is duplicate work between all of these locations
4. Any update to reader/writer feature check or table property check currently needs to be applied to several locations in the code.

Solution

TableConfiguration consolidates all protocol and metadata checks to one place. It also ensures that the logic for checking feature enablement is kept consistent throughout the codebase. This addresses problem 1-4 outlined above.

Closes: #571

How was this change tested?

existing tests pass.
TODO: add more tests

[OussamaSaoudi]

@zachschuermann @nicklan wdyt of this sort of API? The general pattern would be can_(read | write)_{table_feature}

[OussamaSaoudi]

Note that we'd only create such functions if/when we need them, and if they make sense. For example if there is a feature that is only relevant to readers, there would be no can_write_{tablefeature}

[scovich]

Is there a reason it doesn't return DeltaResult<bool>? Seems like there are cases (most of them?) where we don't want to necessarily throw, we just want to learn (but if something goes wrong while trying to learn, an error response still warranted).

Can always add a corresponding assert_can_XXX method if the error version is actually needed?

[OussamaSaoudi]

I was using the pattern of Ok(()) => true, Err(_) => false, with an explanation why. We use a similar pattern here. Do you think we should rethink this approach?

[codecov]

Codecov Report

Attention: Patch coverage is 42.01681% with 69 lines in your changes missing coverage. Please review.

Project coverage is 83.41%. Comparing base (c1c1dbe) to head (50ad379).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/table_configuration.rs	37.73%	63 Missing and 3 partials ⚠️
kernel/src/snapshot.rs	72.72%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #644      +/-   ##
==========================================
- Coverage   83.67%   83.41%   -0.27%     
==========================================
  Files          75       76       +1     
  Lines       16950    17055     +105     
  Branches    16950    17055     +105     
==========================================
+ Hits        14183    14226      +43     
- Misses       2100     2163      +63     
+ Partials      667      666       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[OussamaSaoudi]

Does this and validate_schema_column_mapping belong here?