[Feature Request] Kernel should handle corrupted checkpoints gracefully.

[zikangh]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Test scenario:

1. Write 5 Delta versions (0-4)
2. Create a checkpoint at version 5
3. Write one more version (6)
4. Delete the checkpoint file (simulating corruption)
5. Start a streaming query

DSv1 behavior: The streaming query should successfully read all 6 versions by falling back to JSON commit files, despite the _last_checkpoint file still referencing the deleted
checkpoint at version 5.

Current Kernel behavior: Throws InvalidTableException: Missing checkpoint at version 5 instead of falling back to JSON files.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.

[felipepessoto]

@zikangh would be great if it not only handles missing files but is able to handle corrupted files.

This is something Delta Spark used to support and is broken in 4.0: #5458