v0.6.0

[colin-ho]

What's Changed 🚀

v0.6.0 marks the official release of our new ray-based distributed engine, Flotilla! If you are already using the ray runner, you do not need to change anything. Setting the DAFT_RUNNER=ray environment variable, or within your python program via daft.context.set_runner_ray(), will use Flotilla by default.

All operations except cross join, sort merge join, and pivot are currently supported. We will be working on adding support for them soon! If you need to use the legacy ray runner, please set daft.set_execution_config(use_legacy_ray_runner=True)

💥 Breaking Changes

SQLCatalog was deprecated in v0.5 and is now removed, in favor of the bindings kwargs.

Before:

catalog = SQLCatalog({"test_data": df})
result = daft.sql("SELECT * FROM test_data", catalog=catalog)

After:

bindings = {"test_data": df}
result = daft.sql("SELECT * FROM test_data", **bindings)

---

• feat!: revert daft.func behavior on literal arguments @kevinzwang (feat!: revert daft.func behavior on literal arguments #5087)
• revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" #5084)

✨ Features

• feat(embed_text): Support LM Studio as a provider @desmondcheongzx (feat(embed_text): Support LM Studio as a provider #5103)
• feat: Implement embed_image() @desmondcheongzx (feat: Implement embed_image() #5101)
• feat!: revert daft.func behavior on literal arguments @kevinzwang (feat!: revert daft.func behavior on literal arguments #5087)
• feat: Automatically grab embedding dimensions for sentence transformers @desmondcheongzx (feat: Automatically grab embedding dimensions for sentence transformers #5078)
• feat: add mcap datasource reader @Jay-ju (feat: add mcap datasource reader #4727)

🐛 Bug Fixes

• fix: Undo skipcheck change @srilman (fix: Undo skipcheck change #5131)
• fix: fix youtube video reading @rchowell (fix: fix youtube video reading #5126)
• fix: Remove flotilla fallback @colin-ho (fix: Remove flotilla fallback #5114)
• fix: Add nulls in json reads if a line doesn't contain the field from the schema @colin-ho (fix: Add nulls in json reads if a line doesn't contain the field from the schema #4993)
• fix: Check if UDFs are Serializable @srilman (fix: Check if UDFs are Serializable #5091)
• fix: nightly property test @malcolmgreaves (fix: nightly property test #5076)
• fix: Handle Unserializable Errors in Process UDFs @srilman (fix: Handle Unserializable Errors in Process UDFs #5075)
• fix: Implement Multi-Column Aggregations with List-like columns @srilman (fix: Implement Multi-Column Aggregations with List-like columns  #5017)

🚀 Performance

• perf: Implement count pushdown for parquet @desmondcheongzx (perf: Implement count pushdown for parquet #5038)
• perf(flotilla): Use Worker Affinity with Pre-Shuffle Merge @srilman (perf(flotilla): Use Worker Affinity with Pre-Shuffle Merge #5112)
• perf: Split UDFs from Filters @srilman (perf: Split UDFs from Filters #5070)
• perf(embed_text): Let Sentence Transformers select the best available device @desmondcheongzx (perf(embed_text): Let Sentence Transformers select the best available device #5082)

♻️ Refactor

• refactor: proper organization for providers and protocols in daft.ai @rchowell (refactor: proper organization for providers and protocols in daft.ai #5125)

📖 Documentation

• docs: fix navigation labels to match section names @ykdojo (docs: fix navigation labels to match section names #5121)
• docs: fix flickering typewriter animation on overview page @ykdojo (docs: fix flickering typewriter animation on overview page #5118)
• docs: Add batch inference use case @desmondcheongzx (docs: Add batch inference use case #5116)
• docs: Add docs for custom data sources and sinks @desmondcheongzx (docs: Add docs for custom data sources and sinks #5115)
• docs: add dark mode support for Algolia DocSearch @ykdojo (docs: add dark mode support for Algolia DocSearch #5109)
• docs: add noindex tag to non-stable pages @jaychia (docs: add noindex tag to non-stable pages #5105)
• docs: Add text guide @desmondcheongzx (docs: Add text guide #5102)
• docs: Improve installation instructions @desmondcheongzx (docs: Improve installation instructions #5094)
• docs: More fixes to the overview page in light mode @desmondcheongzx (docs: More fixes to the overview page in light mode #5095)
• docs: Document write_turbopuffer in the user guide @desmondcheongzx (docs: Document write_turbopuffer in the user guide #5092)

👷 CI

• ci: fix test-wheels job in build-wheel.yml @kevinzwang (ci: fix test-wheels job in build-wheel.yml #5134)
• ci: Truncate the # of concurrent jobs in PR CI @srilman (ci: Truncate the # of concurrent jobs in PR CI #5122)
• ci: Run tests before publish @colin-ho (ci: Run tests before publish #5009)
• ci: Always run the unit-tests required check @colin-ho (ci: Always run the unit-tests required check #5119)
• ci: Do not skip postmerge tests @desmondcheongzx (ci: Do not skip postmerge tests #5096)

🔧 Maintenance

• chore: Add AGENTS.md @srilman (chore: Add AGENTS.md #5124)
• chore: Remove docs codeowners @desmondcheongzx (chore: Remove docs codeowners #5111)
• chore: Clean up write_turbopuffer guide @desmondcheongzx (chore: Clean up write_turbopuffer guide #5093)

⏪ Reverts

• revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" #5084)

Full Changelog: v0.5.22...v0.5.23

---

This discussion was created from the release v0.6.0.

[ngendahimanasimioni-ctrl]

Tell me what you are doing

[colin-ho]

Me? Well, let me first tell you a story.

---

Long ago, the simple data types lived together in harmony...

Numbers. Dates. Strings. SQL engines knew exactly what to expect: COUNT this, SUM that, JOIN these tables.

Then, everything changed when the AI wave attacked.

Suddenly, people wanted custom embedding generations, vision model inferences, and other complex transformations that nobody could have anticipated. User-defined functions invaded every pipeline. Images, audio, and video—reduced down to blobs that don't belong.

---

But we believe Daft can save these pipelines.

We want the simplicity of declarative APIs like dataframes and SQL, with the power to run LLMs and embedding models over our documents, images, video, etc.

So, to answer your question, what am I doing?

I'm building the engine to do this:

import daft
from daft.functions.ai import embed_text

df = daft.read_huggingface(
    "togethercomputer/RedPajama-Data-1T"
)

df = df.select(
    embed_text(
        df["text"],
        provider="sentence_transformers",
        model="Qwen/Qwen3-Embedding-0.6B"
    )
)

df.show()
╭────────────────────────────────┬────────────────────────────────┬───────────────────┬─────────────────────────╮
│ text                           ┆ meta                           ┆ red_pajama_subset ┆ embeddings              │
│ ---                            ┆ ---                            ┆ ---               ┆ ---                     │
│ Utf8                           ┆ Utf8                           ┆ Utf8              ┆ Embedding[Float32; 384] │
╞════════════════════════════════╪════════════════════════════════╪═══════════════════╪═════════════════════════╡
│ Григорианският календар (поня… ┆ {'title': 'Григориански кален… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ GNU General Public License (н… ┆ {'title': 'GNU General Public… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Лицензът за свободна документ… ┆ {'title': 'Лиценз за свободна… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Репу̀блика Бълга̀рия е държав…   ┆ {'title': 'България', 'url': … ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Историята на българските земи… ┆ {'title': 'История на Българи… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Септември е името на деветия … ┆ {'title': 'Септември', 'url':… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 18 септември е 261-вият ден в… ┆ {'title': '18 септември', 'ur… ┆ wikipedia         ┆ <Embedding>             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Този списък съдържа имената н… ┆ {'title': 'Списък на страните… ┆ wikipedia         ┆ <Embedding>             │
╰────────────────────────────────┴────────────────────────────────┴───────────────────┴─────────────────────────╯

(Showing first 8 rows)

[1089243263]

What's Changed 🚀

v0.6.0 marks the official release of our new ray-based distributed engine, Flotilla! If you are already using the ray runner, you do not need to change anything. Setting the DAFT_RUNNER=ray environment variable, or within your python program via daft.context.set_runner_ray(), will use Flotilla by default.

All operations except cross join, sort merge join, and pivot are currently supported. We will be working on adding support for them soon! If you need to use the legacy ray runner, please set daft.set_execution_config(use_legacy_ray_runner=True)

💥 Breaking Changes

SQLCatalog was deprecated in v0.5 and is now removed, in favor of the bindings kwargs.

Before:

catalog = SQLCatalog({"test_data": df})

result = daft.sql("SELECT * FROM test_data", catalog=catalog)

After:

bindings = {"test_data": df}

result = daft.sql("SELECT * FROM test_data", **bindings)

---

• feat!: revert daft.func behavior on literal arguments @kevinzwang (feat!: revert daft.func behavior on literal arguments #5087)

• revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" #5084)

✨ Features

• feat(embed_text): Support LM Studio as a provider @desmondcheongzx (feat(embed_text): Support LM Studio as a provider #5103)

• feat: Implement embed_image() @desmondcheongzx (feat: Implement embed_image() #5101)

• feat!: revert daft.func behavior on literal arguments @kevinzwang (feat!: revert daft.func behavior on literal arguments #5087)

• feat: Automatically grab embedding dimensions for sentence transformers @desmondcheongzx (feat: Automatically grab embedding dimensions for sentence transformers #5078)

• feat: add mcap datasource reader @Jay-ju (feat: add mcap datasource reader #4727)

🐛 Bug Fixes

• fix: Undo skipcheck change @srilman (fix: Undo skipcheck change #5131)

• fix: fix youtube video reading @rchowell (fix: fix youtube video reading #5126)

• fix: Remove flotilla fallback @colin-ho (fix: Remove flotilla fallback #5114)

• fix: Add nulls in json reads if a line doesn't contain the field from the schema @colin-ho (fix: Add nulls in json reads if a line doesn't contain the field from the schema #4993)

• fix: Check if UDFs are Serializable @srilman (fix: Check if UDFs are Serializable #5091)

• fix: nightly property test @malcolmgreaves (fix: nightly property test #5076)

• fix: Handle Unserializable Errors in Process UDFs @srilman (fix: Handle Unserializable Errors in Process UDFs #5075)

• fix: Implement Multi-Column Aggregations with List-like columns @srilman (fix: Implement Multi-Column Aggregations with List-like columns  #5017)

🚀 Performance

• perf: Implement count pushdown for parquet @desmondcheongzx (perf: Implement count pushdown for parquet #5038)

• perf(flotilla): Use Worker Affinity with Pre-Shuffle Merge @srilman (perf(flotilla): Use Worker Affinity with Pre-Shuffle Merge #5112)

• perf: Split UDFs from Filters @srilman (perf: Split UDFs from Filters #5070)

• perf(embed_text): Let Sentence Transformers select the best available device @desmondcheongzx (perf(embed_text): Let Sentence Transformers select the best available device #5082)

♻️ Refactor

• refactor: proper organization for providers and protocols in daft.ai @rchowell (refactor: proper organization for providers and protocols in daft.ai #5125)

📖 Documentation

• docs: fix navigation labels to match section names @ykdojo (docs: fix navigation labels to match section names #5121)

• docs: fix flickering typewriter animation on overview page @ykdojo (docs: fix flickering typewriter animation on overview page #5118)

• docs: Add batch inference use case @desmondcheongzx (docs: Add batch inference use case #5116)

• docs: Add docs for custom data sources and sinks @desmondcheongzx (docs: Add docs for custom data sources and sinks #5115)

• docs: add dark mode support for Algolia DocSearch @ykdojo (docs: add dark mode support for Algolia DocSearch #5109)

• docs: add noindex tag to non-stable pages @jaychia (docs: add noindex tag to non-stable pages #5105)

• docs: Add text guide @desmondcheongzx (docs: Add text guide #5102)

• docs: Improve installation instructions @desmondcheongzx (docs: Improve installation instructions #5094)

• docs: More fixes to the overview page in light mode @desmondcheongzx (docs: More fixes to the overview page in light mode #5095)

• docs: Document write_turbopuffer in the user guide @desmondcheongzx (docs: Document write_turbopuffer in the user guide #5092)

👷 CI

• ci: fix test-wheels job in build-wheel.yml @kevinzwang (ci: fix test-wheels job in build-wheel.yml #5134)

• ci: Truncate the # of concurrent jobs in PR CI @srilman (ci: Truncate the # of concurrent jobs in PR CI #5122)

• ci: Run tests before publish @colin-ho (ci: Run tests before publish #5009)

• ci: Always run the unit-tests required check @colin-ho (ci: Always run the unit-tests required check #5119)

• ci: Do not skip postmerge tests @desmondcheongzx (ci: Do not skip postmerge tests #5096)

🔧 Maintenance

• chore: Add AGENTS.md @srilman (chore: Add AGENTS.md #5124)

• chore: Remove docs codeowners @desmondcheongzx (chore: Remove docs codeowners #5111)

• chore: Clean up write_turbopuffer guide @desmondcheongzx (chore: Clean up write_turbopuffer guide #5093)

⏪ Reverts

• revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" @desmondcheongzx (revert!: "revert: Temporarily revert "Remove deprecated APIs for 0.6" #5084)

Full Changelog: v0.5.22...v0.5.23

---

This discussion was created from the release v0.6.0.