[Discussion] Reduce the need for `iceberg-rust` forks / how can we grow the community?

[alamb]

First of all, I hope it goes without saying thank you so much for all your work on this crate 🙏

Problem Statement

Recently, I have heard from multiple people and projects that they are maintaining (or considering maintaining) forks or reimplementations of iceberg-rust because iceberg-rust is not yet suitable for them for some reason.

I think the speaks to a great demand for iceberg functionality in Rust. Having multiple different forks / implementations is of course an option, but I also believe it means the same amount of effort will be split across multiple projects, meaning the available capacity for each will inevitably be smaller than if our efforts were combined somehow

Selfishly, from my DataFusion 🎩 perspective, many projects are interested (understandably) in building systems that use DataFusion to read data stored in Apache Iceberg. I feel that the lack of a super-easy-to-use Iceberg integration crate is holding many of these projects back

Unfortunately I don't have a list of things that would allow these projects to use iceberg-rust, but I hope to generate that list using this ticket.

Ideas to fix:

As I am only relaying this need second hand I don't have many specific ideas to share, and I hope people with firsthand challenges will comment directly on this issue.

One strong strong suspicion is looking at the project commits is that it might help to get some more capacity for reviews / docs / whatever (it appears that @liurenjie1024 is doing the bulk of reviews recently, and @Xuanwo has been helping too)

As always in open source projects, I think the most productive conversation is NOT "people asking the maintainers for things" but rather "how can the people asking for things also contribute to the solution"

Call to Action

Thus I would like to ask:

To people considering a fork:

1. What specific challenges are you facing that means you can't use iceberg-rust
2. What, if anything, are you willing to do / contribute to help get iceberg-rust to the point where you could use it directly

To the existing maintainers:

1. What would help you the most maintaining / moving this crate forward? How can we help?

[Xuanwo]

Thanks for bringing this up! I’m looking forward to hearing more feedback from our users about what they want. I strongly agree that a split community can be a big loss to all of us, so I really want to avoid forking becoming the final decision. If there are anythings I can do, please let know.

---

What would help you the most maintaining / moving this crate forward? How can we help?

The most limited resource for us is review. If experienced developers can help review PRs, it would be really helpful!

[mbutrovich]

Thanks for bringing this up @alamb, as I have recently taken great interest in iceberg-rust (#1749). My Comet PR just today passed all Iceberg Java tests via iceberg-rust, so I think that my POC Comet PR will eventually get merged as an experimental feature.

In my short time focusing on this repo:

1. I have found the reviews to be high quality and the community very welcoming
2. It seems in need of reviewers, so I am trying to spend some time giving review feedback (as a non-committer)
3. I have concerns about how iceberg-rust will track DataFusion and Arrow-rs releases that might make it tough for Comet to keep up with DF, Arrow-rs, and possibly iceberg-rust. However, there is discussion in the Slack on how to better keep up in the future. This is one area I am happy to help contribute/review PRs for since I'm used to bumping DF and Arrow-rs for Comet

I am really exciting about this repo, and the opportunity from my perspective to rapidly mature the ArrowReader via Comet and Iceberg Java tests.

What am I willing to do?

• Review as I find cycles.
• Submit patches, mostly related to ArrowReader at the moment.

[Sl1mb0]

I will share my thoughts as someone who has used iceberg-rust

Current Challenges

One complaint I have is that the Rust implementation relies on the Java implementation as a reference; as opposed to just implementing the spec in Rust. As a result there are a lot of legacy constructs in the iceberg-rust crate that are (in my view) java-flavored; so trying to wield the crate within a Rust project isn't ergonomic at all, and because of that its easy to shoot yourself in the foot.

There are a lot of complexities to iceberg. The metadata serialization, handling, and validation planes are very tightly coupled in iceberg-rust. As an example trying to implement something related to serialization would then require you to understand the handling & validation as well. Not bad in a vacuum, but because the Java implementation serves as the guiding reference - I need to then look at Iceberg java code as a Rust dev in order to understand the context for why things are done the way they are in iceberg-rust, whilst also trying to understand the actual underlying behavior of the implementation. This creates friction for new contributors that I frankly just don't think should be there.

What am I willing to do?

I'm not sure. Personally the biggest roadblock to my ability to contribute is the community agreeing that the above is even a problem. I am sure the core maintainers have valid reasons for using the Java implementation as a reference, but it's something that does not make sense to me AT ALL considering that the entire purpose of having an iceberg specification is to allow implementations to be agnostic in how they uphold it's requirements. In my view there are several fundamental problems with the crate as a result of using the Java impl as the reference.

I would also like to point out that I lack a lot of historical/contextual iceberg knowledge; so while I may view the above as a problem I accept that the community might not agree with me for their own valid reasons.

[kevinjqliu]

Thank you for starting this conversation! I’d love to explore ways we can grow the community together. I’m happy to help in any way I can.

As a contributor to Iceberg

I’ve noticed that PR review can sometimes be a bottleneck for the project. I try to help where possible, but I’d definitely encourage more folks to get involved. For a fast-moving project like iceberg-rust, a more robust test suite could help accelerate reviews. In pyiceberg, we have a solid integration suite with Spark that might be worth bringing over. This could improve review cadence and give everyone more confidence to make forward progress.

Personally, I’m still learning Rust, so there are some language-related challenges for me when contributing to iceberg-rust. However, I’m happy to provide context from the Iceberg side whenever needed.

From a release perspective, we’ve invested a lot of effort in recent releases to automate and streamline the process, as demonstrated in the last release.

I’ll also repost this on the Iceberg dev list to get more visibility. EDIT: https://lists.apache.org/thread/6ovdxgb3dmyq8rwk7kdl5lsdgd1xl9fz

Would love to hear other perspectives from the community!

[JanKaul]

I agree with @Sl1mb0 in that many abstractions feel like they are taken from the Java implementation and don't feel like idiomatic rust. The crate uses many classes where I think lightweight functions would suffice. This was one of the reasons why I stopped contributing to the repo some time ago.

And then there is the interoperability with the Datafusion/Arrow/ObjectStore ecosystem. The crate necessarily duplicates datafusion functionality like expressions, file pruning, parquet reading & writing. This makes total sense for a general library. But if you want to use Datafusion, Datafusion currently just provides more feature complete implementations.

[colinmarc]

Hi, thanks for raising this. We maintain a fork at https://github.com/bauplanlabs/iceberg-rust, but only to stay ahead of patches being merged. The other fork I've seen used is RisingWave's, which I think is in a similar position (although they are more diverged).

We use iceberg-rust in production in concert with datafusion, but we don't use the IcebergTableProvider directly, even though we would really like to. Instead, we use iceberg-rust just for fetching/pruning the manifest lists and then use DataFusion directly. This is awkward, and error prone, and we'd really like to avoid a hack like that. I think there are three particurlarly low-hanging fruit that would really make using IcebergTableProvider feasible for us:

• Fixing the deadlock(s) in the read path: in particular this PR seems excellent and it has gotten zero attention: refactor: TableScan file plan generation now implemented purely in streams rather than channels #1486
• Output partitioning: as I raised in slack, reading from IcebergTableProvider limits you to a single thread, which is obviously not very useful. Here are some benchmarks I created to demonstrate the issue. This issue was closed as not planned. I don't understand the issue 100%, so I hope I'm not misconstruing anything.
• Taking advantage of DataFusion's parquet optimizations: unless I missed something, iceberg-rust doesn't use DataSourceExec/ParquetSource, which means we would automatically lose out on a lot of parquet optimizations already landed or being landed in DataFusion, like metadata caching. I don't understand if that's intended or not. Again, it's possible I'm misunderstanding, apologies if so.

We're more than willing to contribute fixes and features, and we have already, but the ones above are pretty intimidating for me to tackle without any help.

[colinmarc]

One complaint I have is that the Rust implementation relies on the Java implementation as a reference; as opposed to just implementing the spec in Rust. As a result there are a lot of legacy constructs in the iceberg-rust crate that are (in my view) java-flavored; so trying to wield the crate within a Rust project isn't ergonomic at all, and because of that its easy to shoot yourself in the foot.

I'll second this - as a minor point, but something that would definitely help drive adoption were it fixed.

[mbutrovich]

I agree with @Sl1mb0 in that many abstractions feel like they are taken from the Java implementation and don't feel like idiomatic rust. The crate uses many classes where I think lightweight functions would suffice. This was one of the reasons why I stopped contributing to the repo some time ago.

I think this is true of a lot of Rust projects particularly the early days of a “<blah> but in Rust” project. Some of that is due to existing reference implementations, some of that is due to there just being fewer Rust devs and they might be learning on these projects. I’m not dismissing your concerns at all and it’s great to raise them, just affirming that I definitely heard DataFusion described as “Rust written like C++.” We should strive to listen to the Rust community (I am still learning here) about what sort of idiomatic Rust gaps exist in the library, and encourage folks to fix them.

[camuel]

I think rust ecosystem needs something like DuckLake, may be FusionLake? Last time I looked into IcebergTableProvider it looked like building a mini-query-engine or even mini-DBMS within the IcebergTableProvider itself. Is this smart? Those manifestlists and manifests and puffins and json files, it is a lot of transactionally mutable data, is it really smart to work with them as just plain files? Especially by DBMS folks? Every time populate those rust collections, do something, then discard everything per each query? Or develop some sort of ORM cache just for iceberg metadata? Then there is all that advanced functionality like branching operations which only works with Spark. Does rust need to implement it? This is the real complexity!

Given that DataFusion already can natively query avro and json, perhaps DataFusion itself can be used to produce a scan plan with advanced functionality with most logic encoded in SQL or in hand-crafted logical plan and not directly in rust? This will be inline with "canonical iceberg implementation" but going further along with Ducklake (and also Snowflake and Databricks implementation), perhaps DataFusion can use SQLite or Turso or any other OLTP DBMS to maintain its metadata, Iceberg or non-Iceberg, and then have it efficiently dump/ingest to/from canonical Iceberg metadata files and then perhaps even provide IRC out of the box (with the help of LakeKeeker for example) and federation with other IRCs. This will be a full Iceberg implementation on par with "Snowbricks duo".

The thing with Iceberg is to separate "iceberg as data interchange format" where each transaction needs to generate those metadata files on S3 as part of each commit and "iceberg as internal data format" where for most or all transactions no other engine needs to access the data and it is wasteful to generate and regenerate all those numerous metadata files and then run housekeeping to remove them where the only engine accessing the data uses its own metastore.

[adriangb]

@camuel I think that's a bit beyond the scope of what DataFusion as a project wants to pick up and would fragment the ecosystem even more. But I do get your point that Iceberg can be complex, etc. Something that is likely much lower lift and would be a fun project would be to build a DuckLake + DataFusion system where there's a DuckLakeSchemaCatalog type thing so that DuckLake / DuckDB can control the schema catalog and table scans but DataFusion does the rest.

[phillipleblanc]

I agree with many of the issues voiced by @Sl1mb0 and @JanKaul RE: the Rust/Java abstractions and the lack of integration into DataFusion/Arrow/ObjectStore ecosystem. Similar to @colinmarc, we also maintain a fork at https://github.com/spiceai/iceberg-rust, where we mostly apply some of our own patches that we've submitted as PRs (1297, 917, 1673) but haven't yet been merged.

Our ideal state is not using the IcebergTableProvider that is provided by this project out of the box. We went through a similar exercise with delta-rs and found the maintenance burden of coordinating DataFusion versions to delta-rs versions with our usage of DataFusion to be very difficult. We really like the approach that the delta-kernel-rs team took with providing a good set of primitives that can be used during planning, which we then use to hook into the advanced Parquet reading capabilities that DataFusion has (i.e. ParquetExec, ParquetAccessPlan, object_store, etc).

So our wishlist would be:

• A "kernel" (similar to what delta-kernel does) that separates the planning from execution and makes it easy to integrate into a custom query engine.
• Allow using object_store for the kernel IO (ref: Consider Using object_store as IO Abstraction #172) instead of OpenDAL, since we are already heavily invested in it.
• A "reference" implementation of using the kernel (i.e. it could be IcebergTableProvider, but maybe just an example) that shows how to separate the planning of which files to read (and which rows to mask) with a deep integration into the DataFusion ParquetExec machinery. I think its fine to leave the IcebergTableProvider as a "batteries-included" provider that does everything using OpenDAL, as long as we had the primitives above.

If there was appetite to take the project in more of this direction, we would definitely be interested in contributing.

[camuel]

@camuel I think that's a bit beyond the scope of what DataFusion as a project wants to pick up and would fragment the ecosystem even more. But I do get your point that Iceberg can be complex, etc. Something that is likely much lower lift and would be a fun project would be to build a DuckLake + DataFusion system where there's a DuckLakeSchemaCatalog type thing so that DuckLake / DuckDB can control the schema catalog and table scans but DataFusion does the rest.

Thanks @adriangb for the elaboration. That would be a fun project for sure and I think the utility could be in making DuckLake extensible in rust something not possible with Ducklake. May be it isn't too much effort just reimplement Ducklake in rust, not necessarily as part of DataFusion project but using DataFusion in the way Ducklake uses DuckDB, with a roundtrip metadata only interop, it is a full iceberg implementation, not a canonical one but still a fully compatible one. The bulk of the scan planning in DuckLake is done by SQL statements, not by DuckDB C++ code anyway.

[ZENOTME]

One strong strong suspicion is looking at the project commits is that it might help to get some more capacity for reviews / docs / whatever (it appears that @liurenjie1024 is doing the bulk of reviews recently, and @Xuanwo has been helping too)

There is indeed a need for more review capacity. The lack of review resources has slowed down progress on some issues. In this situation, forking might be a faster way to see results.

Thanks for bringing this up! I’m looking forward to hearing more feedback from our users about what they want. I strongly agree that a split community can be a big loss to all of us, so I really want to avoid forking becoming the final decision. If there are anythings I can do, please let know.

What would help you the most maintaining / moving this crate forward? How can we help?

The most limited resource for us is review. If experienced developers can help review PRs, it would be really helpful!

+1. There is a need for more review capacity. The lack of review resources has slowed down progress on some feature request. In this situation, forking might be a faster way to see results.

[twuebi]

I am currently contributing to iceberg-go and have contributed to iceberg-rust and its ecosystem a while ago. I was also contributing to an iceberg-rust fork. I've been mostly on the metadata side of things, having been involved in multiple catalog efforts.

I think the reason why we use iceberg-java as the source of truth alongside the spec is that the spec is underspecified in some places, or iceberg-java's validations went beyond the spec in others, for example, these iceberg-go PRs: apache/iceberg-go#575 (comment), apache/iceberg-go#605 (comment).

Ultimately, iceberg-java has the widest adoption and is the core of the iceberg integration of spark, trino, etc. A lot of users of iceberg-rust / iceberg-go will come from these and have tables written by those engines.

[gsoundar]

@alamb I am willing to help write and review PRs.

What specific challenges are you facing that means you can't use iceberg-rust
The SDK is incomplete at the moment. So while I use the Rust SDK, I have to supplement it with Java SDK.
Issues get started but take a long time to finish to it's hard to depend on it -- for example, #1236

What, if anything, are you willing to do / contribute to help get iceberg-rust to the point where you could use it directly
I am willing to help write and review PRs. I also want to focus on correctness of the code so let me know how we want to proceed.