feat: Add domain metadata support for create-table transactions #1669
+814
−152
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
breaking-change
sanujbasu
force-pushed
the
stack/create_table_4
branch
2 times, most recently
from
4356a80 to
65f7aed
Compare
github-actions
bot
removed
the
breaking-change
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1669 +/- ##
==========================================
+ Coverage 84.64% 84.69% +0.05%
==========================================
Files 125 126 +1
Lines 34721 35043 +322
Branches 34721 35043 +322
==========================================
+ Hits 29388 29681 +293
- Misses 3983 4009 +26
- Partials 1350 1353 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
sanujbasu
force-pushed
the
stack/create_table_4
branch
from
65f7aed to
88130c8
Compare
1. Adds the ability to explicitly enable table features via table properties
using the delta.feature.<featureName> = supported syntax, matching the
Java Kernel's behavior. Only ALLOWED_DELTA_FEATURES can be set during
create. Features get added to protocol features.
2. Allows the min reader/ writer versions to be updated in the protocol using
signal flags. Only protocol versions (3, 7) are supported.
Key Changes:
- Add SET_TABLE_FEATURE_SUPPORTED_PREFIX and SET_TABLE_FEATURE_SUPPORTED_VALUE
constants to table_features module. Move the feature/ property allow/
deny list to the table property configuration module
- Add TableFeature::from_name() to parse feature names from strings
- Add TableFeature::is_reader_writer() to check feature type
- Add TableCreationConfig struct to encapsulate parsing and validation of
user-provided table properties during CREATE TABLE operations.
- Extract delta.feature.* signal flags into reader/writer feature lists
- Extract delta.minReaderVersion/minWriterVersion into protocol hints
- Strip signal flags from properties, pass remaining to metadata
- Reject unknown features and invalid feature flag values
Usage:
create_table("/path/to/table", schema, "MyApp/1.0")
.with_table_properties([
("delta.minReaderVersion", "3"),
("delta.minWriterVersion", "7"),
])
.build(&engine, Box::new(FileSystemCommitter::new()))?
.commit(&engine)?;
The delta.feature.* properties are consumed during build() and not stored
in the final Metadata configuration, matching Java Kernel behavior.
Adds support for user and system domain metadata (domains with 'delta.' prefix) during table creation. This enables features like clustering to be configured at table creation time. Invariants are validated and feature checks performed. Changes: - Refactors generate_domain_metadata_actions to consume domain metadata actions passed down by the create table builder. - Refactors validate_user_domain_operations() to validate_domain_metadata_operations() which enforces a myriad of domain metadata invariants. - Adds validate_system_domain_feature to make sure relevant features are supported when row tracking and clustering domain metadata are pushed down. Clustered table creation support will be added in a stacked PR. The PR with push down the domain metadata for clustered tables into the log. Integration tests to be added in kernel/tests/create_table.rs once clustered table creation support is implement since validation testing requires feature allow listing.
sanujbasu
force-pushed
the
stack/create_table_4
branch
from
88130c8 to
cf03941
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.
What changes are proposed in this pull request?
Adds support for user and system domain metadata (domains with 'delta.' prefix)
processing during table creation in the transactions module. This enables features
like clustering to be configured at table creation time. Invariants are validated and
feature checks performed.
Changes:
actions passed down by the create table builder.
validate_domain_metadata_operations() which enforces a myriad of
domain metadata invariants.
are supported when row tracking and clustering domain metadata are
pushed down.
Clustered table creation support will be added in a stacked PR. The
PR with push down the domain metadata for clustered tables into the log.
How was this change tested?
-Integration tests to be added in kernel/tests/create_table.rs once
clustered table creation support is implement since validation testing
requires feature allow listing.
-Existing tests around domain metadata to cover regressions.