v0.5.19

[srilman]

What's Changed 🚀

We have a pretty crazy release this time around. Some especially notable features include:

• Interactive DataFrames in Jupyter Notebooks, with special support for some multimodal types
• An async API for LLM text generation, particularly with OpenAI
• A new .into_batches DataFrame API, the modern alternative to .into_partitions
• Adding support for .offset / OFFSET operator across the engine. Thanks @plotor for the great work!
• Various Flotilla performance and reliability improvements
• Various casting improvements

✨ Features

• feat: Async open ai llm generate @colin-ho (feat: Async open ai llm generate #4879)
• feat: Add offset syntax support to SQL @plotor (feat: Add offset syntax support to SQL #4707)
• feat: adds support for SQL GROUP BY column position @rchowell (feat: adds support for SQL GROUP BY column position #4955)
• feat: better dtype type inference @universalmind303 (feat: better dtype type inference #4973)
• feat: Casting from Python into struct or list types @srilman (feat: Casting from Python into struct or list types #4957)
• feat: support creating partitioned tables in Iceberg via the Catalog interface. @redpheonixx (feat: support creating partitioned tables in Iceberg via the Catalog interface. #4951)
• feat: implement into_batches operator on flotilla distrubted engine @ohbh (feat: implement into_batches operator on flotilla distrubted engine #4958)
• feat: literal variants for (pretty much) all types @kevinzwang (feat: literal variants for (pretty much) all types #4947)
• feat: Add offset support to Flotilla Engine @plotor (feat: Add offset support to Flotilla Engine #4918)
• feat: implement into_batches on the swordfish native daft runner @ohbh (feat: implement into_batches on the swordfish native daft runner #4935)
• feat: Flotilla broadcast join @colin-ho (feat: Flotilla broadcast join #4867)

🐛 Bug Fixes

• fix: Always just use actor for flotilla scheduler @colin-ho (fix: Always just use actor for flotilla scheduler #4978)
• fix: Add handle for swordfish runtime stats manager @colin-ho (fix: Add handle for swordfish runtime stats manager #4970)
• fix: Dudep lance read required columns @xloya (fix: Dudep lance read required columns #4967)
• fix: Don't use wildcard for logical plan match in pushdown rules @colin-ho (fix: Don't use wildcard for logical plan match in pushdown rules #4945)
• fix: Coerce arrow schema for parquet decoding @colin-ho (fix: Coerce arrow schema for parquet decoding #4948)
• fix: use associate type for swordfish into_batches operator state @ohbh (fix: use associate type for swordfish into_batches operator state #4956)
• fix: raise error on invalid cross join parameters @rchowell (fix: raise error on invalid cross join parameters #4952)
• fix: interactive html fixes @colin-ho (fix: interactive html fixes #4943)

♻️ Refactor

• refactor: Use associate type for swordfish operator state @colin-ho (refactor: Use associate type for swordfish operator state #4921)

📖 Documentation

• docs: update links in document processing example @ccmao1130 (docs: update links in document processing example #4946)
• docs: improve daft.func documentation and type inference @universalmind303 (docs: improve daft.func documentation and type inference #4942)
• docs: fix link for pandas @universalmind303 (docs: fix link for pandas #4941)

👷 CI

• ci: add 3.11 to matrix and always include typing_extensions @rchowell (ci: add 3.11 to matrix and always include typing_extensions #4950)

🔧 Maintenance

• chore: Remove Err Case for Series and RecordBatch size_bytes @srilman (chore: Remove Err Case for Series and RecordBatch size_bytes #4954)

👋 New Contributors

• @redpheonixx made their first contribution in feat: support creating partitioned tables in Iceberg via the Catalog interface. #4951

Full Changelog: v0.5.18...v0.5.19

---

This discussion was created from the release v0.5.19.