[Question] How to generate manifest files and manifest list

[dentiny]

Hi community, I'm trying to use iceberg-rust to write columns into parquet files, and generate iceberg table.

From my understand (and what chatgpt told me), to generate manifest files and manifest list, I need to use transaction and commit. The pseudocode looks like

txn = transaction(table)
while True:
  # Write to parquet ourselves.
  data_file = write_new_parquet_file()
  txn.add_new_data_files(data_file)
# Manifest files and manifest list is created automatically.
txn.commit()

So next time when we read the catalog, we're able to retrieve the snapshots.

But reading through the current catalog implementation, I found only REST one implements update_table, which means transaction::commit is at an un-usable state.

I would like to make sure if I understand correctly? Thanks!

[dentiny]

I confirm the following python version works, and would like to find an equivalent solution in rust.

import os
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import IntegerType, NestedField
from pyiceberg.exceptions import NoSuchNamespaceError, NoSuchTableError
import pandas as pd

# 1. Set up catalog
catalog = load_catalog(
    "local",
    **{
        "type": "sql",
        "uri": f"sqlite:///{os.getcwd()}/iceberg_catalog.db",
        "warehouse": f"file://{os.getcwd()}/iceberg_warehouse",
    }
)

# 2. Create namespace if it doesn't exist
namespace = "default"
try:
    catalog.list_tables(namespace)
except NoSuchNamespaceError:
    print(f"Creating namespace: {namespace}")
    catalog.create_namespace(namespace)

# 3. Define schema
schema = Schema(
    NestedField(field_id=1, name="value", field_type=IntegerType(), required=True)
)

# 4. Create or load table
table_name = "default.numbers_table"
try:
    table = catalog.create_table(table_name, schema)
    print(f"Created new table: {table_name}")
except Exception as e:
    table = catalog.load_table(table_name)
    print(f"Loaded existing table: {table_name}")

# 5. Insert data
df = pd.DataFrame({"value": [1, 2, 3, 4, 5]})
table.append(df)

# 6. Read data
print("\nTable contents:")
for record in table.scan().to_arrow():
    print(record["value"].as_py())

[liurenjie1024]

Hi, @dentiny Yes, you are right, updating table is currently under development and not ready yet.

[dentiny]

Thanks you so much for the quick reply!

May I ask two followup questions?

• if we want to store and load snapshot, would you suggest me implementing my own? For example, first writing data files, then manifest files, manifest list, etc. Do you think of anything tricky, appreciate any suggestions!
• may I also ask about the team/community's plan? I used to use pyiceberg and feel it's common and easy to do so, wondering if we have a timeline for the feature?

Thanks again for the help!

[dentiny]

Or it would be nice if you could provide some code pointer, I'm willing to help contribute :)

[liurenjie1024]

if we want to store and load snapshot, would you suggest me implementing my own?

I don't suggest you do it that.

may I also ask about the team/community's plan?

There are some effort undergoing, but currently there is no epic issue to track the progress. I'll try to open one.

[dentiny]

I confirmed rest catalog works with transaction, which correctly generates manifest files and manifest list, example code

let txn = Transaction::new(&iceberg_table);
let mut action = txn.fast_append(/*commit_uuid=*/ None, /*key_metadata=*/ vec![])?;
action.add_data_files(data_files)?;
let txn = action.apply().await?;
txn.commit(&catalog).await?;

Hopefully other catalogs will catchup.

[liurenjie1024]

Yes, rest catalog supports it. Support in other catalog are undergoing.