v0.6.6

[srilman]

What's Changed 🚀

💥 Breaking Changes

• docs!: update docstrings for various functions @universalmind303 (docs!: update docstrings for various functions  #5344)

✨ Features

• feat: Explicit AWS vs. HTTP mode for common crawl dataset @malcolmgreaves (feat: Explicit AWS vs. HTTP mode for common crawl dataset #5379)
• feat: pydantic model type conversion @kevinzwang (feat: pydantic model type conversion #5370)
• feat(dashboard): Individual Query Page @srilman (feat(dashboard): Individual Query Page #5367)
• feat(flotilla): Flotilla sort merge join @colin-ho (feat(flotilla): Flotilla sort merge join #5369)
• feat: batch UDF with @daft.func.batch @kevinzwang (feat: batch UDF with @daft.func.batch #5362)
• feat(dashboard): Queries Page @srilman (feat(dashboard): Queries Page #5257)
• feat: more tensor conversions @kevinzwang (feat: more tensor conversions #5357)
• feat: @daft.cls decorator for new class UDFs @kevinzwang (feat: @daft.cls decorator for new class UDFs #5350)
• feat: Detect concurrency / num gpus for model apis @colin-ho (feat: Detect concurrency / num gpus for model apis #5342)
• feat: Flotilla linear scheduler @colin-ho (feat: Flotilla linear scheduler #4378)
• feat: Lazy from_glob_path @colin-ho (feat: Lazy from_glob_path #5235)

🐛 Bug Fixes

• fix: Use sum supertype for list_sum type inference @colin-ho (fix: Use sum supertype for list_sum type inference #5366)
• fix: Use default io config in read video if not passed in @colin-ho (fix: Use default io config in read video if not passed in #5364)
• fix: support serialize and deserialize LazyImport @stayrascal (fix: support serialize and deserialize LazyImport #5361)
• fix(file): python expects bytes instead of None @universalmind303 (fix(file): python expects bytes instead of None #5348)
• fix: read_video_frames handles EOF gracefully @rchowell (fix: read_video_frames handles EOF gracefully #5343)

🚀 Performance

• perf(flotilla): Throttle worker refresh and autoscaling @colin-ho (perf(flotilla): Throttle worker refresh and autoscaling #5351)
• perf: Elide shuffle for distinct if input is already partitioned @colin-ho (perf: Elide shuffle for distinct if input is already partitioned #5354)
• perf: use bincode instead of python for io_conf serialization in FileArray @universalmind303 (perf: use bincode instead of python for io_conf serialization in FileArray #5340)
• perf: Only Serialize Required Cols in Process UDFs @srilman (perf: Only Serialize Required Cols in Process UDFs #5069)

📖 Documentation

• docs: Update Common Crawl Dataset docs to make AWS region explicit @desmondcheongzx (docs: Update Common Crawl Dataset docs to make AWS region explicit #5373)
• docs: add Flotilla blog post links to AI benchmarks @ykdojo (docs: add Flotilla blog post links to AI benchmarks #5359)
• docs: Revamp optimization docs @colin-ho (docs: Revamp optimization docs #5347)
• docs!: update docstrings for various functions @universalmind303 (docs!: update docstrings for various functions  #5344)

👷 CI

• ci: Fix ai benchmark workflow @colin-ho (ci: Fix ai benchmark workflow #5363)
• ci: Pin pydantic version in provision.py for iceberg tests @colin-ho (ci: Pin pydantic version in provision.py for iceberg tests #5352)
• ci: Add ai benchmarks ci @colin-ho (ci: Add ai benchmarks ci #5337)

Full Changelog: v0.6.5...v0.6.6

---

This discussion was created from the release v0.6.6.