`ERROR: vector::_M_default_append` while reading `.update-triples` at server start

[ddeboer]

When restarting our server (probably after a crash) we’re getting this error. Our server now seems broken.

│ Command: start │
│ │
│ nohup ServerMain -i dataset-register -j 8 -p 7001 -m 5G -c 2G -e 1G -k 200 -s 30s -a z_Y4mitgG8dujZ9iHBBU-3Tb2BXkf3 --persist-updates > dataset-register.server-log.txt 2>&1 & │
│ │
│ Follow dataset-register.server-log.txt until the server is ready (Ctrl-C stops following the log, but NOT the server) │
│ │
│ 2025-10-11 06:40:01.159 - INFO: QLever Server, compiled on Mon Sep 29 02:25:50 UTC 2025 using git hash 10fc91 │
│ 2025-10-11 06:40:01.164 - INFO: Initializing server ... │
│ 2025-10-11 06:40:01.167 - INFO: The git hash used to build this index was "8b0856" │
│ 2025-10-11 06:40:01.168 - INFO: Reading vocabulary from file dataset-register.vocabulary.words ... │
│ 2025-10-11 06:40:01.173 - INFO: Done, number of words: 102,475 │
│ 2025-10-11 06:40:01.173 - INFO: Number of words in internal vocabulary (these are also part of the external vocabulary): 125 │
│ 2025-10-11 06:40:01.194 - INFO: Registered PSO permutation: #relations = 17, #blocks = 4, #triples = 80,805 │
│ 2025-10-11 06:40:01.197 - INFO: Registered PSO permutation: #relations = 44, #blocks = 54, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.199 - INFO: Registered POS permutation: #relations = 17, #blocks = 4, #triples = 80,805 │
│ 2025-10-11 06:40:01.202 - INFO: Registered POS permutation: #relations = 44, #blocks = 54, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.204 - INFO: Registered OPS permutation: #relations = 260,698, #blocks = 37, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.207 - INFO: Registered OSP permutation: #relations = 260,698, #blocks = 37, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.209 - INFO: Registered SPO permutation: #relations = 64,336, #blocks = 34, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.210 - INFO: Registered SOP permutation: #relations = 64,336, #blocks = 34, #triples = 1,569,572 │
│ 2025-10-11 06:40:01.210 - INFO: Reading patterns from file dataset-register.index.patterns ... │
│ 2025-10-11 06:40:01.211 - INFO: Reading and processing persisted updates from "dataset-register.update-triples" ... │
│ 2025-10-11 06:40:01.219 - ERROR: vector::_M_default_append │
│

Apparently it's an old issue, but it’s unclear how that was resolved: #840.

When I remove the dataset-register.update-triples, the server starts, but of course I’m then missing data. Has our update file become corrupt? Can I repair it?

Will QLever keep appending update sto that file? Will that file then grow indefinitely?

[ddeboer]

I worked around this issue by restoring from a backup, by which we lost only a day of updates.

However, it would be nice if this corruption could be prevented in QLever. Unfortunately, the logs didn’t show anything. I noticed I only had 105 MB left on the data volume. Could low disk space cause errors like this?

[hannahbast]

@ddeboer Currently, the .update-triples file is written from scratch after an update request has completed. That process is quite fast because the data is written in binary. So yes, the file always contains all update triples. I can't tell you what happened in your case, but yes, it is possible that the file became corrupted. Here are some comments and recommendations regarding QLever's handling of persistence of updates:

1. The current mechanism for persisting update triples works but is simplistic
2. We strongly recommend to store update triples so that you can always replay them
3. The way we do it for Wikidata is a good example, see the qlever update-wikidata command: there is an update SSE stream, and the information until which point in time the dataset has been updated is part of the dataset; that way, the command can always figure out by itself from which point in time it needs to replay triples
4. When you have many update triples or a continuous stream of updates, you should rebuild the index from time to time; we are currently working on Rebuild index based on current data, including updates #2408, which does exactly that
5. Rebuild index based on current data, including updates #2408 rebuilds the whole index; this is quite fast because it is done in binary; our next step will be to rebuild the index continuously

We would be interested to hear more details about your use case, in particular: (1) How many triples does your initial dataset have? (2) How many triples do you add in which time intervals? (3) Do you start from a fresh dump from time to time?

[coret]

We would be interested to hear more details about your use case, in particular: (1) How many triples does your initial dataset have? (2) How many triples do you add in which time intervals? (3) Do you start from a fresh dump from time to time?

We (=the Dutch Digital Heritage Network) use Qlever as database for our data-register service. A big part of the 1.5M triples (see Index Information via https://qlever-ui.demo.netwerkdigitaalerfgoed.nl/datasetregister/) is the result of a daily crawl to fetch the dataset descriptions published by Dutch heritage organisations. There are currently over 10 thousand dataset descriptions which are searchable via https://datasetregister.netwerkdigitaalerfgoed.nl/, of course via SPARQL. Each dataset description is stored as a graph, each graph is updated (=deleted+created) daily. There are also so some graphs which hold process information (inserts generated by the crawler) and administrative lists (which see only sporadic manual inserts). So currently, no restarts with a "fresh dump".

[ddeboer]

@hannahbast Thank you for your careful response. To add to @coret above, I’ve followed your advice to rebuild the index by running qlever index --overwrite-existing. That results in:

Should I throw away the .update-triples file after reindexing or must it be kept?

[ddeboer]

When you have many update triples or a continuous stream of updates, you should rebuild the index from time to time

Is it enough to (re)start the server like so?

qlever index --overwrite-existing
qlever start --persist-updates --access-token "$ACCESS_TOKEN"

I’m not sure if that correctly indexes the SPARQL UPDATEs, because the qlever start command will then read and re-apply the update-triples file again.