indexing fails on deployment in kubernetes cluster

[sennierer]

We are deploying Qlever in a Kubernetes cluster and it already worked well.
Since recently we are getting errors on indexing that I cant make any sense of:

025-04-09 12:45:44.489 - INFO: QLever IndexBuilder, compiled on Tue Apr  8 01:36:37 UTC 2025 using git hash 968650
2025-04-09 12:45:44.490 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2025-04-09 12:45:44.490 - INFO: You specified "num-triples-per-batch = 100,000", choose a lower value if the index builder runs out of memory
2025-04-09 12:45:44.490 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2025-04-09 12:45:44.490 - INFO: Processing triples from 13 input streams ...
2025-04-09 12:45:44.491 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2025-04-09 12:49:03.186 - INFO: Triples parsed: 5,109,538 [average speed 0.0 M/s] 
2025-04-09 12:49:03.886 - INFO: Number of triples created (including QLever-internal ones): 6,464,970 [may contain duplicates]
2025-04-09 12:49:03.886 - INFO: Number of partial vocabularies created: 49
2025-04-09 12:49:03.886 - INFO: Merging partial vocabularies ...
Building the index failed: /usr/bin/bash: line 1:    11 Illegal instruction     (core dumped) IndexBuilderMain -i pfp_2024-12-4 -s pfp_2024-12-4.settings.json -f <(cat datasets/pmb.nt) -g http://pmb.acdh.oeaw.ac.at -F nt -p false -f <(cat datasets/wmp1.nt) -g http://wmp1.acdh.oeaw.ac.at -F nt -p false -f <(cat datasets/tillich.nt) -g http://tillich.acdh.oeaw.ac.at -F nt -p false -f <(cat datasets/hanslick.nt) -g http://hanslick.acdh.oeaw.ac.at -F nt -p false -f <(cat datasets/emt.nt) -g http://kaiserin-eleonora.oeaw.ac.at -F nt -p false -f <(cat datasets/staribacher.nt) -g http://staribacher.acdh.oeaw.ac.at -F nt -p false -f <(cat datasets/pio.ttl) -g http://oebl-pfp.acdh-ch-dev.oeaw.ac.at -F ttl -p false -f <(cat datasets/provided_entities.ttl) -g http://pfp-schema.acdh-ch-dev.oeaw.ac.at/provided-entities -F ttl -p false -f <(cat datasets/amp.nt) -g http://amp.acdh.oeaw.ac.at -F nt -p false -f <(cat cidoc-crm/CIDOC_CRM_v7.1.3.ttl) -g http://www.cidoc-crm.org -F ttl -p false -f <(cat cidoc-crm/CIDOC_CRM_v7.1.3_PC.ttl) -g http://www.cidoc-crm.org -F ttl -p false -f <(cat cidoc-crm/CIDOC_CRM_v7.1.3_Supplement.ttl) -g http://www.cidoc-crm.org -F ttl -p false -f <(cat prov-o.ttl) -g http://pfp-sources.acdh.oeaw.ac.at -F ttl -p false --text-words-from-literals         12 Done                    | tee pfp_2024-12-4.index-log.txt

Indexing the same datasets with the same Qleverfile on my local Docker just works. Does anyone have an idea what the problem could be?
Thanks very much!

[ktk]

Can you pull latest version and see if it still traps?

[sennierer]

it deploys now with the latest image. Did you change something?

[ktk]

No but good to hear it worked.

Indexing is still a problem for reasons. It can trigger issues on the host system related to how many files are open and related operations. The way the underlying system reacts is not consistent. Which means sometimes you get a clear error, sometimes zero feedback. The QLever team still works on trying to capture those errors better so it's at least clear what the reason is, but it's not easy.

These errors happen faster on VMs or Kubernetes deployments because you might get an older machine for the particular job. So maybe the only thing that changed was that you got a more powerful machine so the problem did not get triggered.

If you have that problem on a regular base you can tweak the "num-triples-per-batch" parameter, as in increase it.

[sennierer]

yes, we actually got a (very) old server in the cluster we thought could have caused the problem. However, with the newest image it also ran through on the old machine (pined it to the specific node to be sure).
Thanks!