-
Notifications
You must be signed in to change notification settings - Fork 139
Open
Description
Please describe why this is necessary.
With the incoming single-file checkpoint write support (#797), have the opportunity to implement comprehensive integration tests that encompass both reading and writing operations on Delta tables. Our existing test suite lacks scenarios that perform both reading & writing operations in tandem. This gap means we aren't fully validating the end-to-end functionality of the system, particularly the interplay between writing data (and checkpoints) and subsequently reading it back.
Describe the functionality you are proposing.
wip
Should we first look into buffing our current testing framework (non-existent as of now)?
I propose developing a comprehensive integration test that performs a sequence of diverse commit operations on a Delta table. After each commit, the test would:
- read the table state
- checkpoint at the current commit version
- load the table state from the newly created checkpoint.
- assert that the table state before and after checkpointing is identical, ensuring that the checkpoint accurately represents the table's state.
Additional context
No response
Metadata
Metadata
Assignees
Labels
No labels