Multiple data sequence numbers in one commit

[SergeiPatiakin]

We have a use case where we generate N changesets. Each changeset is a (data_files, delete_files) tuple. The changesets should have distinct data sequence numbers so that delete files from later changesets apply to data files from earlier changesets. However, we want to commit all changesets together in one Iceberg commit.

Is there any to do this today in iceberg-rust? Can you think of any interface changes to support this? I was thinking of something like this:

let mut tx = table.new_transaction(None);
tx = tx.append_delete(delete1);
tx = tx.append_data(data1);
tx = tx.new_data_sequence_number();
tx = tx.append_delete(delete2);
tx = tx.append_data(data2);
tx = tx.new_data_sequence_number();
// ...
tx = tx.append_delete(deleteN);
tx = tx.append_data(dataN);
tx.commit().await;

Or maybe like this:

let mut tx = table.new_transaction(None);
tx = tx.append_delete_with_dsn_increment(delete1, 1);
tx = tx.append_data_with_dsn_increment(data1, 1);
tx = tx.append_delete_with_dsn_increment(delete2, 2);
tx = tx.append_data_with_dsn_increment(data2, 2);
// ...
tx = tx.append_delete_with_dsn_increment(deleteN, N);
tx = tx.append_data_with_dsn_increment(dataN, N);
tx.commit().await;

What do you think?

[JanKaul]

And you want to commit all changes at once? Otherwise you could make multiple transactions.

Currently all appends are squashed together, which makes it difficult to achieve your goal

[SergeiPatiakin]

Yes, I want to commit all changes at once so that it's atomic

[JanKaul]

It might be possible to do this with some king of apply function that is similar to commit but doesn't call the catalog in the end. It would look like:

let mut tx = table.new_transaction(None);
tx = tx.append_delete(delete1);
tx = tx.append_data(data1);
tx.apply().await
tx = tx.append_delete(delete2);
tx = tx.append_data(data2);
tx.apply().await;
// ...
tx = tx.append_delete(deleteN);
tx = tx.append_data(dataN);
tx.commit().await;

The last commit would have multiple "AddSnapshot" updates as it would add a snapshot for every apply method call. One would need to check if the REST catalog implementations support adding multiple snapshots in one go.

[SergeiPatiakin]

It can still be a single snapshot. What would change is that appended data files would have non-null data sequence numbers (with different values) in the manifest. Today appended data files have null data sequence numbers in the manifest, and so inherit the sequence number from the snapshot (with all data files inheriting the same value).

Then tx.apply() would be an in-memory operation that doesn't involve any calls to the catalog or to the object store. All writes would still happen in tx.commit().await as they do today. Only the data sequence numbers would change.

If you think this makes sense, I might attempt a PR on this.

[JanKaul]

Generally I think this issue arises because the Iceberg API is used in a way that's not intended. When you want to append a consistent state of data and delete files to a table you should commit a transaction. Consistent meaning that the delete files refer to previous data files and the data files contain new entries. Generally the idea is that if you would like to delete entries in the current transaction, you just don't add them to the data files.

What you would like to do is adding delete files that reference other data files from the same commit. To still get the correct behavior you have to manipulate the sequence numbers so that the delete files apply to the correct data files.

However, Iceberg sequence numbers are only defined at the time of the transaction commit. If you have two competing transactions that both want to increase the sequence number by 1, only one will actually get the sequence number that it proposed. That's the reason why the manifest files inherit the sequence numbers from the manifest list. Because otherwise you would need to rewrite the manifest files when a transaction conflict occurs.

I think writing the sequence numbers directly to the manifest, violates this behavior that is defined in the Iceberg specification.

Generally the correct approach would be to create a transaction for every consistent state of data and delete files that you want to add.

[SergeiPatiakin]

Generally the correct approach would be to create a transaction for every consistent state of data and delete files that you want to add.

Unfortunately this is not feasible for us. Our use case is CDC from Postgres to Iceberg and we need each commit in Iceberg to represent a consistent snapshot from Postgres. At the same time, a large Postgres transaction might have too many changes to hold in memory so we do need intermediate writes to the object store.

If you have two competing transactions that both want to increase the sequence number by 1, only one will actually get the sequence number that it proposed. That's the reason why the manifest files inherit the sequence numbers from the manifest list. Because otherwise you would need to rewrite the manifest files when a transaction conflict occurs.

You are right, this is a valid concern. What if we made multiple manifests per commit if the commit has multiple data sequence groups? Each data sequence group gets its own manifest with all entries having sequence_number = null, but in the manifest list the new manifest entries will have sequence_numbers X + 1, X + 2, etc. That would satisfy our use case while avoid manifest rewrites in case of conflict. How does that sound?

[JanKaul]

This should work. However, would it be possible to create a new Operation like Operation::AppendMutliple for it. I would like to keep the regular Append logic separated. I feel like the current approach with selecting a fitting manifest doesn't work well when multiple manifests will be created.