sqld: disable checkpoint on primary conn create #1898
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LucioFranco
force-pushed
the
lucio/disable-checkpointing-bottomless
branch
2 times, most recently
from
c8673f4 to
a807731
Compare
haaawk
reviewed
Jan 6, 2025
| // TODO: we shouldn't leak the connection here but for some reason when this connection get | ||
| // dropped it seems to checkpoint the database | ||
| if std::env::var("LIBSQL_DISABLE_INIT_CHECKPOINTING").is_ok() { | ||
| std::mem::forget(conn); |
haaawk
reviewed
Jan 6, 2025
haaawk
reviewed
Jan 6, 2025
LucioFranco
force-pushed
the
lucio/disable-checkpointing-bottomless
branch
from
a807731 to
0bc55b5
Compare
haaawk
reviewed
Jan 6, 2025
haaawk
reviewed
Jan 6, 2025
LucioFranco
force-pushed
the
lucio/disable-checkpointing-bottomless
branch
from
0bc55b5 to
f79525d
Compare
This commit changes our primary connection initialization code in two ways to achieve the ability to disable checkpointing the wal. 1) We ignore the initial checkpoint that we call directly into sqlite3 before we restore from bottomless. There is a fixme above that explains why we need this but to me right now its not totally clear why without digging deeper into the internals of bottomless. We should do this but for the moment this unblocks us and from the fixme comment it does not sound unsafe rather doing extra work potentially. 2) When bottomless needs to get the local change counter is creates a sqlite connection. When this connection drops it seems like it checkpoints the wal. I took a brief look at the `sqlite3_close` code and did not find anything obvious, I have been told in the past that sqlite3 likes to checkpoint at weird points so this could be one of those. For the moment, the temporary fix like above is to `std::mem::forget` the connection so that `Drop` never gets called and thus `sqlite3_close` never gets called. With both of these changes we now don't checkpoint the wal unless we hit the max size or interval (which for testing I have set very high). This changes are not enabled by default but must be enabled by setting the following env var: ``` LIBSQL_DISABLE_INIT_CHECKPOINTING=1 LIBSQL_BOTTOMLESS_DISABLE_INIT_CHECKPOINTING=1 ```
LucioFranco
force-pushed
the
lucio/disable-checkpointing-bottomless
branch
from
f79525d to
d95fe5d
Compare
|
Merging and releasing this since its behind a feature flag. Platform testing will follow. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit changes our primary connection initialization code in two ways to achieve the ability to disable checkpointing the wal.
We ignore the initial checkpoint that we call directly into sqlite3
before we restore from bottomless. There is a fixme above that
explains why we need this but to me right now its not totally clear
why without digging deeper into the internals of bottomless. We
should do this but for the moment this unblocks us and from the fixme
comment it does not sound unsafe rather doing extra work potentially.
When bottomless needs to get the local change counter is creates a
sqlite connection. When this connection drops it seems like it
checkpoints the wal. I took a brief look at the
sqlite3_closecodeand did not find anything obvious, I have been told in the past that
sqlite3 likes to checkpoint at weird points so this could be one of
those. For the moment, the temporary fix like above is to
std::mem::forgetthe connection so thatDropnever gets calledand thus
sqlite3_closenever gets called.With both of these changes we now don't checkpoint the wal unless we hit the max size or interval (which for testing I have set very high).