v0.7.0

[colin-ho]

What's Changed 🚀

💥 Breaking Changes

• chore!: remove spark connect @universalmind303 (chore!: remove spark connect #5743)

✨ Features

• feat(lance): row-level schema evolution support @Jay-ju (feat(lance): row-level schema evolution support #5749)
• feat: Improve model api typing @colin-ho (feat: Improve model api typing #5809)
• feat(deltalake): allow users to ignore deletion vectors on read @kevinzwang (feat(deltalake): allow users to ignore deletion vectors on read #5758)
• feat: Capture UDF argument names and values in structured logging. @rohitkulshreshtha (feat: Capture UDF argument names and values in structured logging. #5771)
• feat: support split and merge jsonl/ndjson files @caican00 (feat: support split and merge jsonl/ndjson files #5695)
• feat(tools): add markdown-to-notebook converter for documentation @ykdojo (feat(tools): add markdown-to-notebook converter for documentation #5691)
• feat: support label_selector specification in ray actor/task creation @Jay-ju (feat: support label_selector specification in ray actor/task creation #5042)
• feat: support native csv writer @stayrascal (feat: support native csv writer #5706)
• feat: Update optional dependencies for daft[postgres] @desmondcheongzx (feat: Update optional dependencies for daft[postgres] #5586)
• feat(lance): add distributed compaction for Lance @huleilei (feat(lance): add distributed compaction for Lance #5699)
• feat: Use JSON Serialization for Plans in Subscribers @srilman (feat: Use JSON Serialization for Plans in Subscribers #5709)
• feat: extended hash function to take and hash multiple inputs @rahulkodali (feat: extended hash function to take and hash multiple inputs #5692)
• feat: Add Apache Gravitino catalog in catalog module @shaofengshi (feat: Add Apache Gravitino catalog in catalog module #5694)
• feat: Better errors when lazy imports fail @samstokes (feat: Better errors when lazy imports fail #5753)
• feat: Added structured logging for UDF errors. @rohitkulshreshtha (feat: Added structured logging for UDF errors. #5688)
• feat: Add statistics info to ScanTask when reading Lance dataset @plotor (feat: Add statistics info to ScanTask when reading Lance dataset #5727)
• feat: dynamic batching per operator @universalmind303 (feat: dynamic batching per operator #5676)
• feat: Allow users to disable the suffix range request @TheR1sing3un (feat: Allow users to disable the suffix range request #5188)
• feat: Streaming sample by size @colin-ho (feat: Streaming sample by size #5663)
• feat: Allow dashboard to show query canceled/failed/dead information when query exited abnormally @VOID001 (feat: Allow dashboard to show query canceled/failed/dead information when query exited abnormally #5576)
• feat: Add Google AI provider with prompt @everettVT (feat: Add Google AI provider with prompt #5640)
• feat: Add pow expression @kliwongan (feat: Add pow expression #5237)
• feat: No truncate in .collect preview @colin-ho (feat: No truncate in .collect preview #5632)
• feat: sample api supports precise sampling by size params @caican00 (feat: sample api supports precise sampling by size params #5600)
• feat: audio file subtype @universalmind303 (feat: audio file subtype #5602)
• feat(tos): enhance the retry logic to aware response @stayrascal (feat(tos): enhance the retry logic to aware response #5569)
• feat: emit selectivity metric to OTel in swordfish filter op @samstokes (feat: emit selectivity metric to OTel in swordfish filter op #5584)
• feat: Adding otel logger collector for collecting UDF errors. @rohitkulshreshtha (feat: Adding otel logger collector for collecting UDF errors. #5624)

🐛 Bug Fixes

• fix: handle Windows paths and query params in local_path_from_uri @ykdojo (fix: handle Windows paths and query params in local_path_from_uri #5819)
• fix: use pytest.importorskip for lance in test_limit_offset @ykdojo (fix: use pytest.importorskip for lance in test_limit_offset #5818)
• fix: support skip empty json/jsonl files @caican00 (fix: support skip empty json/jsonl files #5660)
• fix: minor doc fix @yuchaoran2011 (fix: minor doc fix #5814)
• fix: CountRows with Limit returns unexpected result when reading Lance dataset @plotor (fix: CountRows with Limit returns unexpected result when reading Lance dataset #5550)
• fix: Check for missing dependencies in OpenAI provider @everettVT (fix: Check for missing dependencies in OpenAI provider #5747)
• fix: fix btree index invalid issue when reading lance for point lookup @caican00 (fix: fix btree index invalid issue when reading lance for point lookup #5673)
• fix: Combine deltalake with unity extra @everettVT (fix: Combine deltalake with unity extra #5785)
• fix: enhance unit tests @caican00 (fix: enhance unit tests #5787)
• fix: patch CVE-2025-66478 update next dependencies to 16.0.7 @everettVT (fix: patch CVE-2025-66478 update next dependencies to 16.0.7 #5786)
• fix: use single consolidated progress bar in Jupyter notebooks @ykdojo (fix: use single consolidated progress bar in Jupyter notebooks #5774)
• fix: CuPy → NumPy needs explicit conversion @Jay-ju (fix: CuPy → NumPy needs explicit conversion #5680)
• fix: Fix Pydantic cloudpickle serialization in Google Colab @ykdojo (fix: Fix Pydantic cloudpickle serialization in Google Colab #5705)
• fix: update AI integration tests for new Subscriber interface @ykdojo (fix: update AI integration tests for new Subscriber interface #5763)
• fix(optimizer): Prevent limits from being pushed below explodes in non-top-level projections @desmondcheongzx (fix(optimizer): Prevent limits from being pushed below explodes in non-top-level projections #5292)
• fix(io): load all splits in read_huggingface fallback path @ykdojo (fix(io): load all splits in read_huggingface fallback path #5757)
• fix(test): use read_huggingface instead of read_parquet for HF test @ykdojo (fix(test): use read_huggingface instead of read_parquet for HF test #5755)
• fix: add disk cleanup to nightly integration-test-io job @ykdojo (fix: add disk cleanup to nightly integration-test-io job #5711)
• fix: Postgres overwrite table should enable RLS and set up pgvector automatically @desmondcheongzx (fix: Postgres overwrite table should enable RLS and set up pgvector automatically #5657)
• fix: make it easier to enable different logging levels @Abyss-lord (fix: make it easier to enable different logging levels #5661)
• fix: Dashboard logo animation. @j3nkii (fix: Dashboard logo animation. #5672)
• fix(ci): add disk cleanup to integration-test-ai job @ykdojo (fix(ci): add disk cleanup to integration-test-ai job #5733)
• fix: update hypothesis test to use new expression API @ykdojo (fix: update hypothesis test to use new expression API #5723)
• fix: Fix type annotation check on Python 3.14 @srilman (fix: Fix type annotation check on Python 3.14 #5721)
• fix: add fallback mechanism for HuggingFace datasets without parquet files @ykdojo (fix: add fallback mechanism for HuggingFace datasets without parquet files #5650)
• fix: Import or skip lance @colin-ho (fix: Import or skip lance #5662)
• fix: Add missing trailing slashes to S3-compatible endpoint urls @desmondcheongzx (fix: Add missing trailing slashes to S3-compatible endpoint urls #5575)
• fix: Add outer try-finally block in executor generator @colin-ho (fix: Add outer try-finally block in executor generator #5633)
• fix: test_explain @universalmind303 (fix: test_explain #5656)
• fix: Unify the naming and type of URI parameter for Lance-related APIs @plotor (fix: Unify the naming and type of URI parameter for Lance-related APIs #5634)
• fix: Fix blocked and oom issues for scan lance @caican00 (fix: Fix blocked and oom issues for scan lance #5592)
• fix: Executing explain will panic when ScanTask is empty @plotor (fix: Executing explain will panic when ScanTask is empty #5582)
• fix: Embed text dropping texts @colin-ho (fix: Embed text dropping texts #5641)
• fix: limit(n) return n rows directly @caican00 (fix: limit(n) return n rows directly #5597)
• fix: Upgrade to deltalake 1.2.1 @colin-ho (fix: Upgrade to deltalake 1.2.1 #5580)
• fix: add disk cleanup to integration-test-io-credentialed job @ykdojo (fix: add disk cleanup to integration-test-io-credentialed job #5610)
• fix: add disk cleanup to doctests job @ykdojo (fix: add disk cleanup to doctests job #5609)
• fix: Hashable identifier @colin-ho (fix: Hashable identifier #5598)

🚀 Performance

• perf: Lazy udf worker @colin-ho (perf: Lazy udf worker #5542)
• perf: Use growable for build side @colin-ho (perf: Use growable for build side #5613)
• perf: optimize setting lance schema @Jay-ju (perf: optimize setting lance schema #5704)

♻️ Refactor

• refactor(arrow2): values_iter removals for primitive array @universalmind303 (refactor(arrow2): values_iter removals for primitive array #5802)
• refactor(arrow2): remove arrow2 from daft-functions-binary @universalmind303 (refactor(arrow2): remove arrow2 from daft-functions-binary #5799)
• refactor(arrow2): remove deprecated usages from daft-functions-utf8 @universalmind303 (refactor(arrow2): remove deprecated usages from daft-functions-utf8 #5800)
• refactor(arrow2): remove deprecated methods from daft-functions-uri crate @universalmind303 (refactor(arrow2): remove deprecated methods from daft-functions-uri crate #5798)
• refactor(arrow2): remove arrow2 from daft-functions-tokenize @universalmind303 (refactor(arrow2): remove arrow2 from daft-functions-tokenize #5797)
• refactor(arrow2): rename and deprecate to_arrow and as_arrow functions @universalmind303 (refactor(arrow2): rename and deprecate to_arrow and as_arrow functions #5796)
• refactor: write empty dataframe to parquet/json files via native IO @stayrascal (refactor: write empty dataframe to parquet/json files via native IO #5682)
• refactor(arrow-rs): Move temporal conversions from arrow2 to arrow-rs @srilman (refactor(arrow-rs): Move temporal conversions from arrow2 to arrow-rs #5782)
• refactor(arrow-rs): Remove arrow2 Index generics usages @srilman (refactor(arrow-rs): Remove arrow2 Index generics usages  #5761)
• refactor(arrow2): rename and deprecate .to_arrow methods @universalmind303 (refactor(arrow2): rename and deprecate .to_arrow methods #5789)
• refactor(arrow): use arrow for ffi instead of arrow2 @universalmind303 (refactor(arrow): use arrow for ffi instead of arrow2 #5775)
• refactor(arrow): remove daft-arrow from daft-sql & daft-context crates @universalmind303 (refactor(arrow): remove daft-arrow from daft-sql & daft-context crates #5773)
• refactor(arrow-rs): Move all validity daft_arrow::bitmap::Bitmaps to daft_arrow::buffer::NullBuffer @srilman (refactor(arrow-rs): Move all validity daft_arrow::bitmap::Bitmaps to daft_arrow::buffer::NullBuffer #5750)
• refactor: abstract MultipartWriter to write data to object store @stayrascal (refactor: abstract MultipartWriter to write data to object store #5702)
• refactor: Remove Unloaded MicroPartitions @srilman (refactor: Remove Unloaded MicroPartitions #5710)
• refactor(arrow-rs): Add daft-arrow middleman crate for Rust & Arrow usage @srilman (refactor(arrow-rs): Add daft-arrow middleman crate for Rust & Arrow usage #5730)

📖 Documentation

• docs: Update slack invite @everettVT (docs: Update slack invite #5813)
• docs: add logging settings @Jay-ju (docs: add logging settings #5671)
• docs: fix broken Bodo benchmark link @ykdojo (docs: fix broken Bodo benchmark link #5762)
• docs: add voice-analytics-example and update index @everettVT (docs: add voice-analytics-example and update index #5737)
• docs: fix broken Lance documentation link @ykdojo (docs: fix broken Lance documentation link #5724)
• docs: remove redundant About Daft section from README @ykdojo (docs: remove redundant About Daft section from README #5689)
• docs: remove redundant Table of Contents from README @ykdojo (docs: remove redundant Table of Contents from README #5684)
• docs: add Daft Cloud mentions to distributed execution docs @ykdojo (docs: add Daft Cloud mentions to distributed execution docs #5686)
• docs: fix quickstart connector links formatting @ykdojo (docs: fix quickstart connector links formatting #5687)
• docs: update README to reflect AI/multimodal positioning @ykdojo (docs: update README to reflect AI/multimodal positioning #5677)
• docs: Improve mkdocstrings template for Python examples rendering @ykdojo (docs: Improve mkdocstrings template for Python examples rendering #5642)
• docs: changed dev url to a live link to prevent 404 @j3nkii (docs: changed dev url to a live link to prevent 404 #5669)
• docs: add Python version requirement to README @ykdojo (docs: add Python version requirement to README #5655)
• docs: update index overview page @ykdojo (docs: update index overview page #5627)
• docs: remove Python tabs from quickstart @ykdojo (docs: remove Python tabs from quickstart #5626)
• docs: update contributor policy, add contributing section, remove old… @madvart (docs: update contributor policy, add contributing section, remove old… #5251)
• docs: add tip to find your dylib @universalmind303 (docs: add tip to find your dylib #5625)
• docs: add data persistence section to quickstart @ykdojo (docs: add data persistence section to quickstart #5607)
• docs: revamp quickstart with Amazon product dataset example @ykdojo (docs: revamp quickstart with Amazon product dataset example #5585)

✅ Tests

• test: fix flaky OpenAI test by using pattern constraint for hex color format @ykdojo (test: fix flaky OpenAI test by using pattern constraint for hex color format #5808)
• test(io): remove flaky test_read_huggingface_http_urls test @ykdojo (test(io): remove flaky test_read_huggingface_http_urls test #5795)

👷 CI

• ci: increase unit-test timeout to 75 minutes for macOS @ykdojo (ci: increase unit-test timeout to 75 minutes for macOS #5731)
• ci: exclude Kaggle from link checker @ykdojo (ci: exclude Kaggle from link checker #5725)

🔧 Maintenance

• chore(deps): bump the minor group across 1 directory with 45 updates @dependabot[bot] (chore(deps): bump the minor group across 1 directory with 45 updates #5734)
• chore: Provide query end state to RuntimeStatsManager on query end @colin-ho (chore: Provide query end state to RuntimeStatsManager on query end #5791)
• chore: update uvlock to remove tensorflow @kevinzwang (chore: update uvlock to remove tensorflow #5780)
• chore: Codeowners @colin-ho (chore: Codeowners #5502)
• chore: Don't install pytorch in iceberg test docker compose @colin-ho (chore: Don't install pytorch in iceberg test docker compose #5781)
• chore: remove arrow dep from common-image @universalmind303 (chore: remove arrow dep from common-image #5772)
• chore: Pin dependencies @colin-ho (chore: Pin dependencies #5667)
• chore!: remove spark connect @universalmind303 (chore!: remove spark connect #5743)
• chore: remove ir and proto crates @universalmind303 (chore: remove ir and proto crates #5742)
• chore(deps): bump actions/checkout from 5 to 6 in the all group @dependabot[bot] (chore(deps): bump actions/checkout from 5 to 6 in the all group #5713)
• chore(deps): bump ctor from 0.5.0 to 0.6.1 @dependabot[bot] (chore(deps): bump ctor from 0.5.0 to 0.6.1 #5717)
• chore: Cleanup additional Ray runner artifacts @srilman (chore: Cleanup additional Ray runner artifacts #5714)
• chore: Remove the old Ray Runner @srilman (chore: Remove the old Ray Runner #5375)
• chore: Remove expression namespaces @colin-ho (chore: Remove expression namespaces #5619)
• chore: Remove runner from context @colin-ho (chore: Remove runner from context #5628)
• chore: add deprecation to daft.udf @kevinzwang (chore: add deprecation to daft.udf #5665)
• chore(deps): bump the all group with 13 updates @dependabot[bot] (chore(deps): bump the all group with 13 updates #5480)
• chore: update bug report template @universalmind303 (chore: update bug report template #5652)
• chore: remove checklist from pr template @universalmind303 (chore: remove checklist from pr template #5653)
• chore: Remove deprecated agg methods and series split @colin-ho (chore: Remove deprecated agg methods and series split #5630)
• chore: remove broken docpublish job from build-docs workflow @ykdojo (chore: remove broken docpublish job from build-docs workflow #5631)
• chore: Hint users to use .collect when printing empty dataframes @colin-ho (chore: Hint users to use .collect when printing empty dataframes #5616)
• chore: reduce binary size by feature flagging derive(Debug) @universalmind303 (chore: reduce binary size by feature flagging derive(Debug) #5622)
• chore: deprecate llm_generate function @universalmind303 (chore: deprecate llm_generate function #5603)
• chore: reduce sdist size @universalmind303 (chore: reduce sdist size #5606)

⬆️ Dependencies

4 changes

• chore(deps): bump the minor group across 1 directory with 45 updates @dependabot[bot] (chore(deps): bump the minor group across 1 directory with 45 updates #5734)
• chore(deps): bump actions/checkout from 5 to 6 in the all group @dependabot[bot] (chore(deps): bump actions/checkout from 5 to 6 in the all group #5713)
• chore(deps): bump ctor from 0.5.0 to 0.6.1 @dependabot[bot] (chore(deps): bump ctor from 0.5.0 to 0.6.1 #5717)
• chore(deps): bump the all group with 13 updates @dependabot[bot] (chore(deps): bump the all group with 13 updates #5480)

Full Changelog: v0.6.14...v0.7.0