Query on C Interface Arrow Data

[kesavkolla]

Can someone provide an example of how to seemlessly pass the arrow to duckdb in rust? I have the location of parquet file(s) in Azure ADLSv2. I am currently using object_store and arrow-rs to read those parquet files. I do see duckdb and arrow work very seemless in Python. Is there some example that someone can point how to use in rust.

Thanks in advance.

[wangfenjin]

you mean something like this? https://duckdb.org/docs/guides/python/sql_on_arrow

Better share your python examples

[kesavkolla]

Yes that is correct. The example given in that URL. Ideally the recordbatchreader querying on it would be super useful from rust.

[wangfenjin]

Haven't support that in rust api. For now what you can do is load parquet directly into duckdb

#80

[kesavkolla]

Hmmm... unfortunately I can't load parquet in duckdb because duckdb doesn't have support for Azure ADLS. Currently duckdb has no pluggable mechanism for loading parquet files.

Is it possible to write an extension for duckdb in rust?

[wangfenjin]

What about download parquet first and then load it into duckdb?

I haven't spent too much time to dive deep into the duckdb extension, need time to investigate

[kesavkolla]

Yeah I could do that. Ideally it would be nice to provide Arrow C interface so we can easily integrate between rust and duckdb

[wangfenjin]

Yes, the API would be much simpler.

But the logic still the same, download the file, and load into duckdb. Read into arrow and then bind the data into duckdb might be slower than directly read parquet file into duckdb

[kesavkolla]

I am looking at this example in Python:

from adlfs import AzureBlobFileSystem
import duckdb
import pyarrow.parquet as pq
import pyarrow.dataset as ds

account_name = "xxxxx"
account_key = "xxxxx"
abfs = AzureBlobFileSystem( account_name = account_name, account_key = account_key, container_name = "data")
pqdata = ds.dataset("path/inside/abfs", filesystem=abfs)


conn = duckdb.connect(":memory:")
conn.execute("SELECT * from pqdata LIMIT 10")

Something similar in rust would be greatly helpful

[wangfenjin]

Acctually this example may not work because it missed one important steps:

       conn.register("pqdata",pqdata)

And this register API is what we need to implement in rust.

And this register API is only useful when you already have some data in memory and want to use duckdb query it. If you only want to load parquet and query using duckdb, using parquet extension is much more straightforward and faster.

-- load parquest files in data folder
SELECT * FROM read_parquet('./data/*.parquet');

[kesavkolla]

@wangfenjin Can you provide some high level flow for writing register API. I want to implement that and provide PR

[wangfenjin]

hi @kesavkolla , seems we don't have capi yet, I create an issue in duckdb repo, let's wait for the response.

[kesavkolla]

I'm just reading arrow-rs and found they have something called ffi. Arrow ffi has implementation for stream producer and consumer. looks like duckdb has this arrow stream reader this is what even Python duckdb is using

[wangfenjin]

Don't understand your question.

So you have in-memory data in arrow format, you want to query it using duckdb, or query it using arrow-rs?

• If it's arrow-rs, then you can use something like arrow-rs ffi or arrow-datafusion.
• if you want use duckdb's sql engine to query on it, you have no choice but make the data available to duckdb first, that's why we need the register api