RowId for Daft-Lance Integration

[Jay-ju]

Technical Proposal for Daft-Lance Integration

Current Implementation

Daft effectively utilizes URL-based lazy processing to perform joins, aggregations, and complex operations (e.g., window functions) in multi-modal workflows. This paradigm delays downloading content until its actual use, which offers significant flexibility.

Identified Storage-Side Challenges

1. Excessive small files: High S3 QPS due to fragmented requests
2. Limited encoding capabilities: Cannot leverage columnar compression or advanced encoding techniques
3. Underutilized Lance features: Point queries and dynamic column addition (add_column) remain inaccessible

Proposed Integration Approach

To harness the complementary strengths of Daft and Lance, we suggest:
Replace URL-based download interfaces with row_id-based point queries via lance_take(),
Substitute upload interfaces with Lance's atomic update() operations.

Anticipated Advantages

• Unified execution leveraging Daft's distributed processing and Lance's storage optimizations
• Elimination of small-file overhead through batched columnar access
• Full utilization of Lance features: compression, point queries, and schema evolution

@jaychia @kevinzwang @universalmind303 WDYT？

[kevinzwang]

HI @Jay-ju , thanks for writing this up. I think this approach makes sense.

As a first step, could you share some sample workloads that would benefit from this? I'd like to set up some baseline benchmarks that we can work towards improving.

[Jay-ju]

Benefit 1: Lance can apply columnar storage compression to data managed via URLs, resulting in significant cost reduction. This is particularly advantageous for large-scale unstructured data such as point clouds and serialized pickle files.

Benefit 2: Lance can serve as a PyTorch MapDataset during training, leveraging its efficient random point lookups for optimized data loading.

Regarding Daft: Substantial gains can be achieved when joining tables containing multi-modal columns (e.g., image + text). In Daft's current implementation, images and text are represented by URL references (as illustrated above). When filtering based on URL sources and subsequently persisting the filtered multi-modal data, the workflow involves:

1. Processing URLs alongside metadata
2. Fetching the actual multi-modal content post-filtering
   The integration with Lance replaces the conventional row identifier approach while maintaining this operational pattern.

[Jay-ju]

@kevinzwang I made a simple design here; please take a look at this content. https://docs.google.com/document/d/1dFTjB3DEkBFwEKuPJthuJ7QfIMLzIPfLGGcG1RkyfQ4/edit?tab=t.0

[kevinzwang]

Hi @Jay-ju , thanks for working on this! Tagging @rchowell because this is very close to the catalogs and volumes concepts that he has worked on. Conner what do you think?

[Jay-ju]

@rchowell ， If you have time here, could you please take a look at the idea mentioned here? It mainly involves providing a kv namespace to wrap some point query operations. The priority is to implement lance, and later we will also implement the lmdb form.

[Jay-ju]

A demo implementation has been done here, but it's not the final version and can't be reviewed yet. However, it roughly conveys the functionality that wants to be achieved. #5081