Apache Iceberg

[amotl]

About

Apache Iceberg is a high performance open-source format for large analytic tables. Iceberg enables the use of SQL tables for big data while making it possible for engines like Spark, Trino, Flink, Presto, Hive, Impala, StarRocks, Doris, and Pig to safely work with the same tables, at the same time.

Details

Iceberg uses the Apache Parquet file format for storing actual data due to its efficient columnar storage structure, optimized for analytical queries. Parquet files in Iceberg store table rows in a compressed, column-oriented format, significantly reducing storage costs and improving read performance through techniques such as predicate pushdown and column pruning. Iceberg references Parquet files in manifest files, facilitating quick identification and access to relevant data during query execution.

Resources

• https://en.wikipedia.org/wiki/Apache_Iceberg
• https://iceberg.apache.org/

[amotl]

Updates 2025

• Apache Iceberg™ | What It Is and Why Everyone’s Talking About It
• https://github.com/crate/cratedb-toolkit/issues/420

[amotl]

All of those gems are using Apache Iceberg™ Rust.

• Databend: An open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake.
• Lakekeeper: An Apache-licensed Iceberg REST Catalog with data access controls.
• RisingWave: A Postgres-compatible SQL database designed for real-time event streaming data processing, analysis, and management.

Is there a way to mix Java and Rust? (discuss)

[amotl]

We are finally making a start to improve interoperability. Thanks, @hlcianfagna.

• I/O: Adapter for Apache Iceberg cratedb-toolkit#444
• RisingWave: Tutorial about Apache Iceberg -> CrateDB cratedb-guide#207
• Use open table format to store dumps: Delta, DuckLake, Hudi, Iceberg #264

/cc @WalBeh, @matriv, @surister, @kneth