Skip to content

Conversation

@ddwalias
Copy link
Contributor

@ddwalias ddwalias commented Nov 1, 2025

Reproduce:

  1. Use SQLite to initialize an auto-vacuum DB (to produce a live pointer-map page at page 2):

sqlite3 /tmp/av_corruption.db "PRAGMA auto_vacuum=FULL; VACUUM; IF NOT EXISTS __sim_autovacuum_seed__(id INTEGER PRIMARY KEY, value TEXT); INSERT OR IGNORE INTO __sim_autovacuum_seed__ VALUES (1,'seed');"

  1. Let Turso mutate it:

tursodb /tmp/av_corruption.db "CREATE TABLE t(a);" \ for i in $(seq 1 2000); do target/debug/tursodb /tmp/av_corruption.db "INSERT INTO t VALUES ($i);"; done

  1. Back in SQLite, run PRAGMA integrity_check; -> it reports Tree … Failed to read ptrmap key=….. However, Turso’s own PRAGMA integrity_check; still prints ok, so the engine doesn’t notice the damage.

Why the simulator misses it:

The deterministic runs always start from Turso-built databases, and never import a file produced by SQLite, so this path is never exercised.

Reference: #3895

@ddwalias
Copy link
Contributor Author

ddwalias commented Nov 1, 2025

The cause seem to be when Turso allocates leaf or overflow pages, it never writes the corresponding pointer‑map entries.

I attempted to fix this issue with this commit, but seems like I introduced a new bug in the process. For this PR, I will focus only on updating the simulator. The fix will be on another PR.

@penberg
Copy link
Collaborator

penberg commented Nov 2, 2025

Hey @ddwalias! Good catch! I am disabling autovacuum by default in #3899. Let's have other people review this PR too, but assuming the improvement is good, this is eligible for the data corruption challenge reward, although functionality is disabled now.

@ddwalias ddwalias force-pushed the ptrmap-data-corruption branch 7 times, most recently from ed4a35e to 036d820 Compare November 20, 2025 20:53
@ddwalias ddwalias changed the title simulator: catch ptrmap data corruption with pre-initialize autovacuum database Fix ptrmap data corruption with pre-initialize autovacuum database Nov 20, 2025
@ddwalias ddwalias force-pushed the ptrmap-data-corruption branch from 036d820 to 20c75be Compare November 20, 2025 21:30
@ddwalias
Copy link
Contributor Author

I managed to make the integration test passes, however, the simulator is still failing, likely from another issue

@ddwalias ddwalias force-pushed the ptrmap-data-corruption branch from 29bd984 to 20c75be Compare November 23, 2025 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants