Skip to content

Internal errors while inserting from geojson #1995

Open
@controversial

Description

@controversial

What happens?

I’m using duckdb-wasm with the spatial extension loaded, and repeatedly running a query of the form:

INSERT INTO segments (id, geom, a, b, recorded_at_z)
SELECT id, geom, a, b, z FROM ST_Read('previously-registered-file.geojsonl')
ON CONFLICT (id) DO UPDATE SET
  geom = EXCLUDED.geom,
  recorded_at_z = EXCLUDED.recorded_at_z
WHERE segments.recorded_at_z < EXCLUDED.recorded_at_z;

I’m frequently seeing errors from within duckdb-wasm like the following:

  • TransactionContext Error: Failed to commit: Trying to read past end of buffer
  • TransactionContext Error: Conflict on tuple deletion!
  • RuntimeError: memory access out of bounds
  • TransactionContext Error: Cannot perform IO in in-memory database — CreateBlock!

the purpose is to ingest a large amount of geospatial data into a queryable form as it loads incrementally from map vector tiles. Each geojson file often contains hundreds (up to a few thousand) of features, and each feature becomes its own record in duckdb.

I’m not sure whether these errors arise as a result of one issue or multiple, but I am seeing all of these errors from the same query (when run with different data)

To Reproduce

I “recorded” the registered files and the executed queries executed for a single session that displays several of these errors in order to create a reliable minimal reproduction.

memory-access-repro.zip

Here,

  • files.json contains content that was “registered” as files to read
  • queries.json contains queries that were run that produced the error(s)
  • the included index.js loads duckdb, registers each of the files, and runs the recorded queries in sequence. This is a simple entrypoint that “replays” the sequence of queries that leads to the errors.

Running index.js reliably produces the errors I saw earlier; the set of conditions leading to the errors appears to be deterministic.

Browser/Environment:

Chrome 136

Device:

macOS 15.5 (Darwin 24.5.0 arm64 arm)

DuckDB-Wasm Version:

1.29.1-dev132.0

DuckDB-Wasm Deployment:

bundled

Full Name:

Luke Taylor

Affiliation:

N/A; using for my own purposes for personal projects

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions