Checkpoints not used by write_deltalake?

[VLomonovskis]

Hello.

I have several processes that uses the same code and append data to the same Delta Table. Those processes run in parallel. I append data using write_deltalake and use rust engine to merge schema.

As several processes add data, performance degrading and upload takes more time. As I understand it happens because increases number of transaction log files. However, when I create checkpoint ( using delta_table.checkpoint() ), it does not improve performance and looks like write_deltalake reads all the logs before checkpoint. Can this behaviour be changed?

I did see discussions about checkpoint, but they where about checkpoint creation. In my case, checkpoints not used even when created.

[UffeTorlyn]

I am having the exact same experience!
I have a fairly small test table with 300+ partitions. Now I have tried to write the same data over and over again to the partitions and the write speed just keep increasing from 2 secs pr. partitions to now (as of this moment 6-7 secs per write per partition).

The only way I can explain this behavior is exactly as you said that the write_deltalake reads all logs before the checkpoint aswell.

[VLomonovskis]

I was able to improve performance by deleting old logs prior to creating a new checkpoint.
Since I have multiple parallel runs, I introduced locking and now run cleanup_metadata before invoking create_checkpoint.
With this sequence, the most recent logs remain untouched (in case one of the processes is still using them) and only the logs from before the previous checkpoint are deleted.

1. 🗑️ Old Logs → 📌 Old Checkpoint → 📝 Recent Logs
   delta_table.cleanup_metadata() # deletes "Old Logs"
2. 📌 Old Checkpoint → 📝 Recent Logs
   delta_table.create_checkpoint() # creates "New Checkpoint"
3. 📌 Old Checkpoint → 📝 Recent Logs → ✅ New Checkpoint

There will still be some impact from the issue, as the "Recent Logs" will continue to be read.
If logs are generated rapidly, the "Recent Logs" may still have a noticeable effect on the process.