Arrow2 -> Arrow-rs Migration

[universalmind303]

Background:

Back in the early days of the Rust Arrow ecosystem, two contenders vied for dominance: arrow2 and arrow-rs. Arrow2 made some enticing promises - better safety, improved performance, and had an active community at the time. We placed our bet on arrow2 and built Daft around it.

Fast-forward to today: arrow2 is completely abandoned, leaving projects like ourselves and Polars to vendor it internally just to keep things running. This creates a massive technical burden and prevents us from adopting new features readily available in the thriving arrow-rs ecosystem.

Why Migrate?

Switching to arrow-rs would unlock several key improvements:

f16 support

Would unblock multiple issues (#5293, #3126, #2889) by leveraging arrow-rs's native f16 support.

Dictionary encoding

Eliminates need to write our own dictionary encoding array (#1346).

Native Rust Lance integration

Simplifies implementation (#4841) by using arrow-rs directly with Lance APIs, removing arrow2 → arrow conversion overhead.

Improved count(*) performance

Benefits from arrow-rs's optimized parquet readers (#4450, #2298), which are up to 9x faster for metadata decoding.

Easier contribution

Using a well-documented and actively maintained library would lower the barrier for community contributions, as contributors won't need to understand our vendored arrow2 fork.

Reduced binary size & compile time

Currently we are compiling both arrow2 and arrow-rs, which increases both the binary size and our compile times. Removing arrow2 means we only need to compile one arrow crate instead of two.

Additional benefits

• Better decimal support (arrow-rs is substantially more complete)
• Improved internal representation for file datatype
• Potential for Result<T,E> datatype
• Run-length encoded arrays for scalar values instead of broadcasting
• Native arrow tensor types rather than our custom implementation
• Variant datatype

Proposed migration guide

Part 1: Encapsulation

Create a new daft-arrow crate that will act as a bridge while we are undergoing the migration.

Initially daft-arrow will just reexport everything from arrow2. From there, we can ban all usage of arrow2 outside of that crate and make everything use daft_arrow instead.

This reduces our surface area and makes all arrow usage encapsulated inside this single crate.

Acceptance Criteria:

This stage would be considered complete once there is no usage of arrow2 anywhere outside of this newly introduced crate.

---

Part 2: Interface Migration

In this phase, we'll modify the core Array trait to match the arrow-rs interface while maintaining compatibility with existing code.

Once all arrow usage is encapsulated inside daft_arrow, we'll update the daft_arrow::Array trait to align with arrow_rs::Array. This foundational change requires either:

• Creating a compatibility layer to bridge between our new interface and arrow2 implementations, or
• Updating specific implementations to match the new interface

We're prioritizing the Array trait first since it's the cornerstone of our interactions with Arrow data structures. Related components like TrustedLen and Bitmap will be addressed as they come up.

Acceptance Criteria:

1. The daft_arrow::Array trait definition mirrors the arrow_rs::Array interface
2. A compatibility mechanism exists to maintain functionality with existing arrow2-based code
3. All existing code continues to function without modification

---

Part 3: Physical Array Migration

At this point, we’re now using the new daft_arrow::Array trait throughout the codebase, but all of the concrete array implementations are still arrow2.

We’ll start by moving over the physical arrays

Now that we are using the equivalent of the arrow_rs::Array trait, we can start swapping out the concrete implementations from arrow2 → arrow-rs.

Acceptance Criteria:

All Physical arrays (DataArray ) are now backed by arrow-rs arrays instead of arrow2 arrays.

---

Part 4: Logical Array Migration

Same as part 3, but for logical arrays

---

Part 5: Final Cleanup

Now that everything is fully backed by arrow-rs instead of arrow2, we can start removing the compatibility layer (daft_arrow ), and remove our vendored arrow2 crate entirely

Note: we may want to keep daft_arrow just to keep the encapsulation layer.

Acceptance Criteria

arrow2 is entirely removed from our codebase

---

Post-Migration Work

As mentioned above, there are various issues that become much easier to implement once we are using arrow-rs

To reiterate a few key work streams:

• fp16
• arrow tensors instead of DIY tensors
• dictionary and run end encodings
• better decimal support
• swap out parquet2 for parquet

Success Metrics

• Zero regressions in functionality
• Performance equal to or better than arrow2 implementation
• Reduction in codebase size through elimination of vendored code

[plotor]

LGTM, Thanks @universalmind303