Comparison of the development paradigms of daft and ray data

[caican00]

daft natively supports the sql/dataframe api. Compared with the map/map_batches of ray data, this will offer a better development experience for users.

But now it is learned that ray data is also following up on sql/dataframe capabilities. Will this weaken the advantages of daft?
ray-project/ray#53693

[caican00]

gently ping @colin-ho

[caican00]

They'll try to catch up to Daft but we're laser focused on the workloads that matter to us so I'm sure Daft development is going to always be one step ahead :)

that's great.
May I ask what the current moat of daft is, that is, the important work of daft mentioned here.

[jaychia]

We remain incredibly focused on how we can use models to process unstructured/multimodal data and will be the best possible way to do this. The Daft team doesn't get distracted to also have to build for RL, training, serving, etc - we are only building for this use-case and have no dependency needing to use Ray itself.

At some point we will be able to reduce our dependency on Ray for distributed execution which means better fault tolerance, less object store spilling/tuning, OOMs and overall speedups.

[caican00]

We remain incredibly focused on how we can use models to process unstructured/multimodal data and will be the best possible way to do this. The Daft team doesn't get distracted to also have to build for RL, training, serving, etc - we are only building for this use-case and have no dependency needing to use Ray itself.

At some point we will be able to reduce our dependency on Ray for distributed execution which means better fault tolerance, less object store spilling/tuning, OOMs and overall speedups.

thanks for your reply.

that is to say, the core scenario of daft is batch processing inference. Can it be understood in this way? However, ray data has developed in batch inference scenarios for many years. Will this also lead to competition?

furthermore, is ETL one of the main application scenarios of daft?

[caican00]

May I ask if are there any blogs or official docs for detailed comparisons between daft and ray data?

[jaychia]

Daft is made only for ETL, whereas Ray is meant as a general-purpose distributed system

And yes! Feel free to peruse our work at: https://www.daft.ai/blog