Feature request: Merge <index_name>.update-triples

[MadsHolten]

Background

I’m new to QLever and recently started using libqlever to run an embedded database inside a Google Cloud Run instance with GCS Fuse as the persistence layer.

After experimenting for about a week, I’m getting very promising results so far—thanks for the fantastic work on this project!

Feature request

We use QLever in a pipeline that needs to “saturate” datasets by running a series of update queries. To explore the performance characteristics, I tested how queries perform when executed over the delta triples (i.e., the entries tracked in <index_name>.update-triples).

Unfortunately, performance drops significantly once update triples are involved. I was wondering if you would consider adding a function that merges the delta triples into the main index, effectively compacting the dataset again.

This would help maintain performance while still enabling incremental updates between rebuilds.

Test Results

1. Baseline count query:
   Dataset size: 9,003,298 triples
   Execution time: 0.386 s

2. After update query:
   245k triples added to <index_name>.update-triples

3. Count query after update:
   Dataset size: 9,248,305 triples
   Execution time: 4.885 s (≈13× slower)

Additional context

To enable update queries through libqlever, I had to add a small helper function.
It’s quite minimal (and admittedly vibe-coded since I don’t write C++), but it works well for quick experiments.

I’d be happy to open a PR if you think this could be useful to other users or serve as a starting point for a more robust interface.

[RobinTF]

See #2408 we're currently working on this, but it might still take a while until it is ready for production. (There are still some bugs, most of which stem from bugs inside UPDATE functionality, but eventually this should allow QLever to keep performance high even after a lot of updates.)

[hannahbast]

@MadsHolten I am surprised by your numbers and would like to understand them better. When the update triples (we call them "delta triples") are just a fraction of the total number of triples, query performance should change only minimally. For example, here is an example where I removed a random subset of 10% of all triples from a large predicate and it slows down the query time by 30% https://github.com/ad-freiburg/qlever/wiki/First-tests-with-SPARQL-1.1-Update . Deleting a random subset from a predicate is the worst case.

When I ingest the Wikidata update stream, which adds or deletes around 1 million triples per day, and I run this for a week, then query time increases by 10 - 20%.

I would understand your results if all the added triples were in new predicates or new graphs and your query mostly queries those new predicates or new graphs. Is something like that the case?

As @RobinTF said, we are working on rebuilding the index in the background. There are four stages to this:

1. Rebuilding the index explicitly from an explicit full export of the dataset. This can be done already now and with 9 million triples, this should be a matter of seconds. For example, I can rebuild the index for the Olympics dataset (2M triples) like this in a few seconds on my PC: curl -s https://qlever.dev/api/olympics --data-urlencode query="CONSTRUCT WHERE { ?s ?p ?o }" | IndexBuilderMain -i export -s <(echo "{}") -f - -F nt -p 1

2. The next step is avoiding the serialization/deserialization involved in 1 (the CONSTRUCT query serializes are triples, thenIndexBuilderMain has to parse them again). This is what Rebuild index based on current data, including updates #2408 does. It speeds up the approach from 1 by a factor of 5.

3. With Rebuild index based on current data, including updates #2408, the rebuild has to be triggered manually using curl -s host:port -d cmd=rebuild-index -d access-token=..., and then you have to restart the server explicitly using the new index yourself. The next step is to rebuild and restart with a single simple command, without affecting any queries (queries might still be buy being processed by the old index when switching to the new one).

4. The final step is to continuously rebuild the index incrementally in the background. Until then, 1 and 2 are pretty good workarounds, just a little more work to set it up.

[MadsHolten]

@hannahbast in the particular example it is a count query and most predicates are probably new. It is also important to mention that I query the index through a CLI command that takes the index and the query. This means that nothing is kept in memory which probably has a significant impact on query time.

Will the index rebuilding also be added to libqlever? In our setup we have one dataset per project and potentially thousands of projects so the 1 server = one index doesn't work for us.

[hannahbast]

@MadsHolten In principle, all QLever features that do not require HTTP can also be added to libqlever. It's just a matter of adding the corresponding wrapper functions to https://github.com/ad-freiburg/qlever/blob/master/src/libqlever/Qlever.h .

It's hard to say why exactly your queries are slow without seeing the exact queries and dataset. It could be because of QLever, it could als very well be because you use it in a suboptimal way that could easily be fixed.