[Feature]: Node.js Bindings

[RyanGuild]

Is your feature request related to a problem?

Node.js has a Spark Client but a more light weight solution would be to expose delta-rs to the node.js runtime as a node extension.

Describe the solution you'd like

I started creating bindings for this library this afternoon. So far it has been simple to implement using napi-derive I would like to volunteer to create this as a PR rather than a stand alone project if anyone would like to help guide me through the contribution process. I think there could be huge applications for deltalake in edge computing if made available to node.

Describe alternatives you've considered

I have not considered any alternatives I was just hacking this afternoon and wanted a really really cheap to operate database. whats cheaper than a file?

Priority

Medium - Would be helpful

Additional context

I am a pretty seasoned rust, typescript, and node developer and I have a few days in the new year to get another release pipeline and everything set up.

Contribution

• I'm willing to submit a pull request for this feature
• I can help with testing this feature
• I can help with documentation for this feature

[roeap]

@RyanGuild - awesome!! I was looking into something similar a while ago and think this would be pretty cool to have!

If you need some inspiration, I found that the lancedb repo has a pretty neat setup and is architecturally quite similiar project-wise!

In any case, happy to help this along. We do probably need a quick vote with the maintainers, if we want this, personally I am very favourable though :).

[abhiaagarwal]

Not that I'm opposed to this at all, as someone who increasingly is writing more TS professionally: but what advantage does this have for edge compute over, say, using rust lambdas/workers directly?

[ion-elgreco]

I would be against having this in the delta-rs repo.. Similar to delta-dotnet we can have this in the incubator project instead.

[RyanGuild]

@abhiaagarwal The coolest benefit is that the napi-derive lets you export an async function to javascript and have it return a JS promise that will poll the rust future. I use delta-rs all the time for my rust based projects but I was trying to get good delta support working in a nextjs project. Without really knowing my execution environment other than node will be installed I need a node-gyp based install to provide delta support.

@ion-elgreco I think I should start this project in a personal repo and we can chat about getting it into the delta-rs project latter.

Thanks for all the quick responses I will LYK when I have something good to look at.

EDIT: I know I should probably use sqlite for the purpose of edge file-based db but I really like the way deltalake encourages a feed-forward design. I know that delta will degrade as an application database but I work mostly on small user count analytics dashboards with a little bit of end user functionality I wish that my frontend and my databricks didn't need an application database in between in my corporate projects so I can stop explaining the difference between spark sql and postgres every 20 min. In my personal projects I am interested in using deltalake as an ultra low cost cloud state store the optimistic concurrency pattern could allow a lambda to respond before the commit even goes through and then use the waitUntil pattern to leave the lambda open to negotiate the actual state commit.

EDIT 2: the napi async feature ships tokio

[abhiaagarwal]

@abhiaagarwal The coolest benefit is that the napi-derive lets you export an async function to javascript and have it return a JS promise that will poll the rust future. I use delta-rs all the time for my rust based projects but I was trying to get good delta support working in a nextjs project. Without really knowing my execution environment other than node will be installed I need a node-gyp based install to provide delta support.

Interesting point. I've made somewhat of a career of using delta lakes as a database, so I'm not necessarily opposed to this.

If you're looking for collaborators, give me a shout abhiaagarwal01@gmail.com: def would be interested in contributing!