Skip to content

tests: Integration tests for checkpointing reads & writes #886

@sebastiantia

Description

@sebastiantia

Please describe why this is necessary.

With the incoming single-file checkpoint write support (#797), have the opportunity to implement comprehensive integration tests that encompass both reading and writing operations on Delta tables. Our existing test suite lacks scenarios that perform both reading & writing operations in tandem. This gap means we aren't fully validating the end-to-end functionality of the system, particularly the interplay between writing data (and checkpoints) and subsequently reading it back.

Describe the functionality you are proposing.

wip

Should we first look into buffing our current testing framework (non-existent as of now)?

I propose developing a comprehensive integration test that performs a sequence of diverse commit operations on a Delta table. After each commit, the test would:​

  1. read the table state
  2. checkpoint at the current commit version
  3. load the table state from the newly created checkpoint.​
  4. assert that the table state before and after checkpointing is identical, ensuring that the checkpoint accurately represents the table's state.​

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions