QLever Docker setup - memory requirements for 2 billion triples dataset with autocompletion

[LorenzBuehmann]

Hi again,

I'm trying to make use of the QLever UI and also came across the autocomplete feature which consistently makes the Docker container crashing. So I'm wondering if I messed up the memory settings or our server is to weak? It has only 128GB RAM and there is already some other stuff working on it.

---

Qleverfile memory settings:

WITH_TEXT_INDEX   = false
SETTINGS_JSON     = '{ "num-triples-per-batch": 5000000, "ascii-prefixes-only": false }'
#GET_DATA_CMD      = "Write a one-liner here for how obtain your data (or comment this line out)"
INDEX_DESCRIPTION = "RDF data for ${DB}"
STXXL_MEMORY_GB   = 50

# Server settings
HOSTNAME                       = $(hostname -f)
SERVER_PORT                    = 7001
ACCESS_TOKEN                   = ${DB}_2415228838
MEMORY_FOR_QUERIES             = 80
CACHE_MAX_SIZE_GB              = 50
CACHE_MAX_SIZE_GB_SINGLE_ENTRY = 10
CACHE_MAX_NUM_ENTRIES          = 100

---

Docker start log:

2023-07-12 10:39:48.241	- INFO:  Requested media type of result is "application/qlever-results+json"
2023-07-12 10:39:48.328	- INFO:  Query planning done in 12 ms, additional time spend on index scans during query planning: 85 ms
2023-07-12 10:40:25.250	- INFO:  QLever Server, compiled on Sat Jun 17 20:22:30 UTC 2023 using git hash 04cccc
2023-07-12 10:40:25.255	- INFO:  Initializing server ...
2023-07-12 10:40:25.256	- INFO:  The git hash used to build this index was 04cccc
2023-07-12 10:40:25.256	- INFO:  Reading vocabulary from file tiva.vocabulary.internal ...
2023-07-12 10:40:36.588	- INFO:  Done, number of words: 507,749,948
2023-07-12 10:40:36.588	- INFO:  Number of words in external vocabulary: 0
2023-07-12 10:40:36.595	- INFO:  Registered PSO permutation: #relations = 9, #blocks = 56,281, #triples = 1,758,655,692
2023-07-12 10:40:36.601	- INFO:  Registered POS permutation: #relations = 9, #blocks = 56,281, #triples = 1,758,655,692
2023-07-12 10:40:36.607	- INFO:  Registered OPS permutation: #relations = 359,338,173, #blocks = 52,056, #triples = 1,758,655,692
2023-07-12 10:40:36.616	- INFO:  Registered OSP permutation: #relations = 359,338,173, #blocks = 52,056, #triples = 1,758,655,692
2023-07-12 10:40:36.620	- INFO:  Registered SPO permutation: #relations = 507,749,928, #blocks = 37,519, #triples = 1,758,655,692
2023-07-12 10:40:36.624	- INFO:  Registered SOP permutation: #relations = 507,749,928, #blocks = 37,519, #triples = 1,758,655,692
2023-07-12 10:40:36.624	- INFO:  Reading patterns from file tiva.index.patterns ...
2023-07-12 10:40:37.823	- INFO:  Sorting random result tables to estimate the sorting performance of this machine ...
2023-07-12 10:40:40.637	- INFO:  Access token for restricted API calls is "tiva_2415228838"
2023-07-12 10:40:40.637	- INFO:  The server is ready, listening for requests on port 7001 ...

---

SPARQL query on autocompletion request:

Processing the following SPARQL query:
PREFIX tiva: <https://schema.coypu.org/tiva#>

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>





SELECT ?qleverui_entity
              (MIN(?name) as ?qleverui_name)
              (MIN(?alias) as ?qleverui_alias)
              (SAMPLE(?count_2) as ?qleverui_count) WHERE {
  { SELECT ?qleverui_entity (COUNT(?qleverui_entity) AS ?count_2)
    WHERE { ?s a tiva:ExgrBsci . ?s ql:has-predicate ?qleverui_entity }
    GROUP BY ?qleverui_entity }
  { SELECT ?qleverui_entity ?name ?alias ?count_1 WHERE {
  { { SELECT ?qleverui_entity (COUNT(?qleverui_entity) AS ?count_1) WHERE { ?x ql:has-predicate ?qleverui_entity } GROUP BY ?qleverui_entity }
    BIND(?qleverui_entity AS ?name) BIND(?qleverui_entity AS ?alias) .
    FILTER (?qleverui_entity != <QLever-internal-function/langtag>)
  } UNION {
    { SELECT ?qleverui_entity (COUNT(?qleverui_entity) AS ?count_1) WHERE { ?x ql:has-predicate ?qleverui_entity } GROUP BY ?qleverui_entity }
    BIND(?qleverui_entity AS ?name) BIND(?qleverui_entity AS ?alias) .
    FILTER (?qleverui_entity != <QLever-internal-function/langtag>)
  } } }
  
} GROUP BY ?qleverui_entity ORDER BY DESC(?qleverui_count)


LIMIT 40
OFFSET 0

What happens

• we start the Qlever Docker container and the UI
• context sensitive suggestions is selected
• we write

PREFIX tiva: <https://schema.coypu.org/tiva#>
SELECT * WHERE {
  ?s a tiva:ExgrBsci .
  ?s 
}
limit 10

and the ctrl + Tab for autocomplete

Watching at the memory I can see that it grows until the max. size of the server, i.e. from 50GB after QLever has been started to our max of 128GB RAM:

---

So can we somehow estimate how much memory we have to assign to the server? It's rather difficult for us at the moment to configure it. Thanks in advance