Having Qlever production ready, api description, multi-tenancy

[imsdu]

Hello,
We are considering Qlever as a replacement for Blazegraph.

Is there documentation to make it production ready which containes (among other things):

• Recommended hardware documentation
• Recommended configuration options
• Disaster recovery procedures (backups and restore, ...)
• Monitoring / alerting
• ...

Also is there a description of the API ?

Finally, what would be the kind of deployment to make it multi-tenant ?
Is it possible to have isolated datasets in the same instance ?
Or should each dataset live in their dedicated instance ?

Thanks :)

[ktk]

Is there documentation to make it production-ready which contains (among other things):

Have a look at the pages linked in the README and the wiki—most pointers come from there.

• Recommended hardware documentation

Check out the performance evaluations for DBLP here and PubChem here.

In general, QLever runs well on modern but affordable raw hardware. Every layer of virtualization costs performance. Don’t overthink it—start with what you have and see how it performs.

What dataset size are you working with?

• Recommended configuration options

Start with the defaults and adjust as needed. If you run out of memory for queries, it likely means QLever isn't optimizing that query type efficiently. The team is interested in such cases to improve performance.

• Disaster recovery procedures (backups and restore, ...)

Write support isn’t persistent yet, so the best approach is to keep your original data elsewhere and rebuild the index if needed. This will change within the next six months, but for now, that’s the way to go.

• Monitoring / alerting

Not much available yet. We're considering adding tracing and heartbeat monitoring in the future.

Also, is there a description of the API?

The API is SPARQL, so it depends on what exactly you’re looking for.

Finally, what kind of deployment would make it multi-tenant?
Is it possible to have isolated datasets in the same instance?
Or should each dataset have its own instance?

Each dataset needs its own instance.

[imsdu]

Thanks a lot.

[imsdu]

Any plan to allow isolated datasets on the same instance ?

[ktk]

Not right now, no.

[valentinedwv]

To fake multi-tenancy we are just using traefik routing and multiple container. This is the stack for each 'namespace'

version: '3.9'

networks:
  qlever_network:
      name: qlever-network-${QLEVER_NET:-base}
      external: true
  traefik_proxy:
      name: traefik_proxy
      external: true
#export USER_ID=$(id -u)
#export GROUP_ID=$(id -g)
volumes:
  qleverflow_data:
    name: qleverflow_data_${QLEVER_VOL:-main}
    #name: qlever
configs:
  qlever-config:
    name: qlever-config-${QLEVER_CONFIG:-deepoceans}
    external: true
  qlever-ui:
    name: qlever-ui-${QLEVER_CONFIG_UI:-deepoceans}
    external: true
services:

 qlever-server-project:
    image: docker.io/adfreiburg/qlever:latest
    hostname: qlever.server.${PROJECT:-deepoceans}
    init: true
    user: "root:root"  # Will be set via .env file
    #user: "1000:1000"  # Will be set via .env file
    working_dir: /data
#    ports:
#      - "7019:7019"
    environment:
      - PATH="/qlever:/qlever/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      - LANG="C.UTF-8"
      - LC_ALL="C.UTF-8"
      - LC_CTYPE="C.UTF-8"
      - DEBIAN_FRONTEND="noninteractive"
      - QLEVER_PROFILE="/etc/profile.d/qlever.sh"
      - QLEVER_ARGCOMPLETE_ENABLED="1"
      - QLEVER_IS_RUNNING_IN_CONTAINER="1"
    volumes:
      - /etc/localtime:/etc/localtime:ro
    configs:
      - source: qlever-config
        target: /data/Qleverfile
    entrypoint: /qlever/docker-entrypoint.sh
    command: >
         ' ls -l && qlever get-data  --log-level DEBUG && qlever index --text-index from_text_records_and_literals && qlever start --use-text-index yes --run-in-foreground '

    restart: unless-stopped
    networks: &network
      - qlever_network
      - traefik_proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}.entrypoints=http"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}.rule=Host(`qlever.${HOST}`) && PathPrefix(`/graphspace/${PROJECT}`)"
      - "traefik.http.middlewares.qlever-project-${PROJECT:-deepoceans}-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}.middlewares=qlever-project-${PROJECT:-deepoceans}-https-redirect"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}-secure.entrypoints=https"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}-secure.rule=Host(`qlever.${HOST}`) && PathPrefix(`/graphspace/${PROJECT}`)"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}-secure.tls=true"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}-secure.tls.certresolver=httpresolver"
      - "traefik.http.routers.qlever-project-${PROJECT:-deepoceans}-secure.service=qlever-project-${PROJECT:-deepoceans}"
      - "traefik.http.services.qlever-project-${PROJECT:-deepoceans}.loadbalancer.server.port=7019"
      - "traefik.docker.network=traefik_proxy"

 qlever-ui-config-namespace:
    image: docker.io/adfreiburg/qlever-ui:latest
    depends_on:
      qlever-server-project:
        condition: service_started
    volumes:
      - type: volume
        source: qleverflow_data
        target: /app/db/
    configs:
      - source: qlever-ui
        target: /app/Qleverfile-ui-deploy.yml
    # ports:
    #   - "8176:7000"
   #    entrypoint: bash
    command: bash -c "python manage.py copy default ${PROJECT:-deepoceans} && python manage.py config ${PROJECT:-deepoceans} /app/Qleverfile-ui-deploy.yml"

    restart: on-failure:1
    networks: *network