Add support for reading Delta tables

[gdubya]

Describe your feature request

The Delta format is gaining popularity. The Polars project has recently added _read_delta methods for the Python and Rust APIs. It would be great if the node implementation could also support the Delta format.

I am not aware of any native Delta kernel implementation for Node, but perhaps a first version for nodejs-polars could use the Rust implementation behind-the-scenes?

[Bidek56]

@gdubya Delta tables are just versioned Parquet files. node-polars supports readParquet already.
Are there specific features of Delta that you need? Thx

[universalmind303]

@gdubya I do like the idea of adding delta support, and think it'd be a good fit for the project (especially considering py & rust have it). In the short term, you could use GlareDB (another arrow based system) to read delta, then convert to polars.

> npm install nodejs-polars
> npm install @glaredb/glaredb

const pl = require("nodejs-polars");
const glaredb = require("@glaredb/glaredb");
const con = await glaredb.connect();
const df = await (
	await con.sql(
		"select * from delta_scan('path/to/delta/table')",
	)
).toPolars();

console.log(df);

[eitsupi]

The Polars project has recently added _read_delta methods for the Python and Rust APIs.

I think Rust don't have this feature.
Related issues: pola-rs/r-polars#221, pola-rs/polars-cli#14

[TheKnightCoder]

What is the update on this? Would also love this to use delta with nodejs

[Bidek56]

We had #190 but it was decided that it's too heavy and we shelved it for now.

[Bidek56]

This issue is related

[Bidek56]

This issue is in the backlog, so there's hope.