Use open table format to store dumps: Delta, DuckLake, Hudi, Iceberg

[amotl]

About

JSON dumps are fine, but maybe something more rich can be useful. We looked at Parquet the other day, and would also like to consider standard open data and table formats like Parquet, Delta, Iceberg, Hudi, or DuckLake.

Introductions

• Open Table Formats — Delta, Iceberg & Hudi
• New kid on the block: DuckLake

Useful packages

https://pypi.org/project/delta-spark/
https://pypi.org/project/hudi/
https://pypi.org/project/pyiceberg/
https://github.com/apache/iceberg-go
https://pypi.org/project/pyspark/
https://github.com/duckdb/ducklake

References

• Integration with polars duckdb/ducklake#121

[amotl]

@mkleen just shared those with me, which could fit into the same box, maybe a bit on the lower-level ends. Thanks!

• https://github.com/vortex-data/vortex
• https://github.com/cwida/FastLanes