delta-rs versions since 1.0.0 seem to be updating _last_checkpoint differently

[tmeskuti]

Hey everyone,
We have recently tried the new versions of delta-rs using the Python bindings, and there seems to be a different behaviour from v0.25.5 on how the _last_checkpoint file is updated. You can find examples below:

• v0.25.5 -> {"size":2,"sizeInBytes":14961,"version":0}
• v1.0.0 and onwards -> {"version":0,"size":2,"parts":1,"sizeInBytes":19419,"numOfAddFiles":0}

Both files were generated by running DeltaTable.create_checkpoint() on the same table, and on both occasions, there is only one checkpoint file being created, although the second example specifies "parts": 1.

This behaviour is causing us issues when trying to query the tables using Trino:
LastCheckpoint[version=899, size=16, parts=Optional[1], v2Checkpoint=Optional. empty] mentions a non-existent checkpoint file for table: our_schema. our_table io. trino. spi. TrinoException: LastCheckpoint[version=899, size=16, parts=Optional[1], v2Checkpoint=Optional. empty] mentions a non-existent checkpoint file for table: our_schema. our_table
This is solved by going into the _last_checkpoint file and removing the parts key. For clarification, the checkpoint file does exist, although it is not multi-part.

Is this behaviour expected? I can't determine if this is a bug, so I'm not opening an issue yet, but any insight you may have would be extremely helpful!
Thanks in advance.

[tmeskuti]

Looks like parts is hardcoded to 1 in delta-kernel-rs:
https://github.com/delta-io/delta-kernel-rs/blob/bbca6261fbfe6d9aaf9bc3f6ad219b8279b76a7a/kernel/src/checkpoint/mod.rs#L477

[ion-elgreco]

@tmeskuti I suggest you create an issue in the delta-kernel-rs repo, since we offloaded the checkpointing to kernel.

[tmeskuti]

Putting it here in case it helps anyone else: delta-io/delta-kernel-rs#1043