why is daft so fast?

[raybellwaves]

Are there any technical papers, blogs, docs you can put me to that explains why daft is so fast?

Thanks

[colin-ho]

You can find our blogs at https://blog.getdaft.io/. Our documentation also has a page detailing our technical architecture: https://www.getdaft.io/projects/docs/en/latest/faq/technical_architecture.html, though we are in the process of revamping our execution engine so it is a little outdated.

But a quick TLDR, Daft is fast because of

• Vectorized Execution with SIMD
• Parquet/CSV/JSON readers written in Rust
• Built-in integration with Ray for distributed computation

Appreciate the question!

[raybellwaves]

Thanks. I did a quick write up on Daft here to summarize my understanding https://sites.google.com/view/raybellwaves/blog/what-data-processing-tool-should-i-use#h.hqilyig5fmfo

[samster25]

Thanks for including us @raybellwaves and for the feedback! We'll get on it to improve the content out there for how Daft works and some more examples!