Skip to content

Download in-memory database as duck or sqlite file #1987

Open
@andy-wrks

Description

@andy-wrks

What happens?

I'm using DuckDB Wasm 1.29.0 through JavaScript in the browser (Chrome).
I create some tables in the "memory" database and populate them with data from various sources. Now I'd like to download the in-memory db to a single file (i.e. preferrably not as single csv, json or parquet tables).
There seems to be the possibility to download the db as a duck or sqlite file, however I struggle to succeed with this.

Here's what I tried for sqlite:

async exportSQLiteFile() {

    const fileName = "my.db";
    const aliasName = "mydb";

    await this.#db.registerEmptyFileBuffer(fileName);

    await this.#conn.query(`ATTACH '${fileName}' AS ${aliasName} (READ_WRITE, TYPE SQLITE);`);
    // Copy the in-memory database to the new database file
    await this.#conn.query(`COPY FROM DATABASE memory TO ${aliasName};`);
    await this.#conn.query(`DETACH ${aliasName};`);

    const buffer = await this.#db.copyFileToBuffer(fileName);
    fileIo.downloadFile(buffer, fileName);

    await this.#db.dropFile(fileName);
}

The COPY process seems to work. When I change to mydb, the tables and data from the memory db are there. However, the downloaded file "my.db" is 1 byte in size...

I then tried to download a duckdb file:

async exportDuckFile() {

    const fileName = "my.db";
    const aliasName = "mydb";

    await this.#db.registerEmptyFileBuffer(fileName);

    await this.#conn.query(`ATTACH '${fileName}' AS ${aliasName} (READ_WRITE, TYPE DUCKDB);`);
    // Copy the in-memory database to the new database file
    await this.#conn.query(`COPY FROM DATABASE memory TO ${aliasName};`);
    await this.#conn.query(`DETACH ${aliasName};`);
	
    const buffer = await this.#db.copyFileToBuffer(fileName);
    fileIo.downloadFile(buffer, fileName);
	
    await this.#db.dropFile(fileName);
}

This results in an "Uncaught (in promise) Error" when using copyFileToBuffer.

The only thing that has worked for me so far, is exporting the tables to single parquet-files like this:

async exportParquetFiles(dbName) {

    // Note: This is a custom function for getting the names of existing tables.
    // Not to be confused with conn.getTableNames from the API, which does something else.
    const tableNames = await this.getTableNames(dbName);

    for (const tableName of tableNames) {
        const fileName = `${tableName}.parquet`;
        await this.#db.registerEmptyFileBuffer(fileName);
        await this.#conn.query(`COPY (SELECT * FROM ${tableName}) TO '${fileName}' (FORMAT PARQUET);`);
        const buffer = await this.#db.copyFileToBuffer(fileName);
        fileIo.downloadFile(buffer, fileName);
        await this.#db.dropFile(fileName);
    }
}

This still trows a warning for every exported file: "Buffering missing file: *.parquet"
I'm not sure if this is a problem or what needs to be done in order to not get this warning.

What am I missing in the first 2 examples, how is it possible to export the in.memory db to a duckdb or sqlite file?

To Reproduce

See code examples and description above.

Browser/Environment:

Chrome 134.0.6998.178

Device:

Win11 Laptop, i7-8850-H CPU

DuckDB-Wasm Version:

1.29.0

DuckDB-Wasm Deployment:

locally via VSCode Live Server

Full Name:

Andy

Affiliation:

none

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions