DuckDB-Wasm fails to load in electron with nodeIntegrationInWorker enabled

[MasterOdin]

What happens?

When I have the nodeIntegrationInWorker flag enabled in electron, then when I go load/use duckdb-wasm, I get the following error:

TypeError: ic(...).dirname is not a function
    at cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:10:59232
    at Go.instantiateImpl (cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:11:25177)
    at Go.instantiate (cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:11:15869)
    at lc.instantiate (cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:24:10893)
    at lc.onMessage (cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:10:53913)
    at Qu.globalThis.onmessage (cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.27.0/dist/duckdb-browser-eh.worker.js:24:10976)

If I disable the flag, then the library loads fine and seems to work fine, though I've not heavily tested the functionality beyond a simple query.

To Reproduce

See https://github.com/MasterOdin/electron-duckdb-wasm. Changing this line:

https://github.com/MasterOdin/electron-duckdb-wasm/blob/1cd5f871eac0fa1a24d9807b0f8a9ab4eac3c78f/main.js#L14

where if it's true then will see the above error, and if it's false then the query runs and I get my result.

Browser/Environment:

Electron 26 (Chrome 116.0.5845.62), though have seen this for Electron 19 as well.

Device:

Macbook Pro

DuckDB-Wasm Version:

1.27.0

DuckDB-Wasm Deployment:

JS Delivr

Full Name:

Matthew Peveler

Affiliation:

PopSQL

[MasterOdin]

Looking further into this, the root issue seems to be that the bundler breaks compiling the duckdb-eh.js binding into the actual duckdb-browser-eh.worker.js.

From the map file, the duckdb-eh.js lines of interest are:

        var ENVIRONMENT_IS_WEB = typeof window == "object";
        var ENVIRONMENT_IS_WORKER = typeof importScripts == "function";
        var ENVIRONMENT_IS_NODE = typeof process == "object" && typeof process.versions == "object" && typeof process.versions.node == "string";
        var scriptDirectory = "";
        function locateFile(path) {
            if (Module["locateFile"]) {
                return Module["locateFile"](path, scriptDirectory)
            }
            return scriptDirectory + path
        }
        var read_, readAsync, readBinary;
        if (ENVIRONMENT_IS_NODE) {
            var fs = require("fs");
            var nodePath = require("path");
            if (ENVIRONMENT_IS_WORKER) {
                scriptDirectory = nodePath.dirname(scriptDirectory) + "/"
            } else {
                scriptDirectory = __dirname + "/"
            }
            // ...

with nodeIntegrationInWorker enabled, process.versions.node is defined, so ENVIRONMENT_IS_NODE is true, and we fall into the if statement where path is required and since we're also in a worker ENVIRONMENT_IS_WORKER is true, and we hit nodePath.dirname.

Looking at the bundled worker code (https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.28.0/dist/duckdb-browser-eh.worker.js) and run through a beautifier:

                        f = typeof window == "object",
                        m = typeof importScripts == "function",
                        _ = typeof process == "object" && typeof process.versions == "object" && typeof process.versions.node == "string",
                        g = "";

                    function B(c) {
                        return e.locateFile ? e.locateFile(c, g) : g + c
                    }
                    var Z, wt, Ft;
                    if (_) {
                        var At = bu(),
                            Vt = gu();
                        m ? g = Vt.dirname(g) + "/" : g = __dirname + "/", Z = (c, u) => (c = oc(c) ? new URL(c) : Vt.normalize(c), At.readFileSync(c, u ? void 0 : "utf8")), // ... omitted

where we see the require('path') is replaced by a call to gu() which is defined:

    var ns = (r, t) => () => (t || r((t = {
            exports: {}
        }).exports, t), t.exports),
    var gu = ns(() => {});

I'm not exactly sure what ns is doing, but given that the word "path" is not mentioned at all, I'm going to go out on a limb and say that it's not actually doing require('path'), and adding a cosnole.log(Vt) gives me an empty object ({}) back, which would then explain why the error complains about dirname not being a function.

As far as I can tell the code is originating from the emscripten project (shell.js), but I'm not sure exactly how it's getting bundled, or exactly what the build process is that's resulting in this broken setup.

I did find that by doing a replace of _ = typeof process == "object" && typeof process.versions == "object" && typeof process.versions.node == "string", with _ = false and the WASM seemed to work fine for me.

At this point I'd need some pointers from the duckdb-wasm team to dig deeper into trying to find a fix (if one is desired).

[MasterOdin]

emscripten-core/emscripten#22619 fixed the detection issue for emscripten and was part of the duckdb-wasm@1.29.0 release.

I've raised emn178/js-sha256#49 to fix similar issue there which should then close out this issue once merged there and version bumped here.