Skip to content

Integrating VegaFusion with distributed DataFrames #567

Answered by jonmmease
OlegWock asked this question in Q&A
Discussion options

You must be logged in to vote

Hi @OlegWock, as you noted we had some architecture in place in version 1 for generating SQL for different dialects. This was removed for v2 because it was a significant source of complexity (given how it worked) and wasn't being used beyond duckdb. And after the optimizations in VegaFusion 2 (and probably in DataFusion as well), the duckdb path wasn't faster, so I didn't put the work into exploring the unparse workflow after all.

That said, I'm definitely interested in bringing this functionality back if there's demand and we develop a cleaner architecture.

Here's what I'm thinking might work.

We still convert the Vega transforms to DataFusion DataFrame operations, but instead of only ev…

Replies: 1 comment 4 replies

Comment options

You must be logged in to vote
4 replies
@OlegWock
Comment options

@OlegWock
Comment options

@jonmmease
Comment options

@OlegWock
Comment options

Answer selected by OlegWock
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants