Combined Spatial and Text search/indexing

[recalcitrantsupplant]

I've been playing with text and spatial datasets - the performance of the text search has been great, for the spatial I've had mixed results.

I got combined spatial and text filtering to work, but I had to run the indexer twice, once with on-disk-compressed-geo-split and another time with VOCABULARY_TYPE = "on-disk-compressed" and TEXT_INDEX = from_literals. I then ran qlever start like: qlever start --use-text-index yes. I'm hoping I've overcomplicated this but other methods didn't seem to have both indexes available.

So before raising any issues I wanted to clarify what the current status of text and geospatial search is.

My questions are:

• When using the GeoSPARQL predicates and filter functions, is this hitting a spatial index?
• More generally, what is the status of the different index types (I see S2, libspatial join and others - when are each used)
• Can I generate a combined spatial and text index(es) and what would the configuration for this look like?

My preference is to use GeoSPARQL, as we have applications which generate code in this format, and SERVICE clauses can be a little awkward at times, however I can use whichever format.

I'm happy to contribute some examples to documentation if the future direction of these types of queries is known/can be clarified.

Some example queries I've been using at various points, only for context:

Spatial search with GeoSPARQL - I last tried this some time ago and the performance seemed poor

SELECT *
WHERE {
  {
    SELECT DISTINCT ?focus_node ?pred ?match ?weight (URI(CONCAT("urn:hash:", SHA256(CONCAT(STR(?focus_node), STR(?pred), STR(?match), STR(?weight))))) AS ?hashID) ?focus_node
    WHERE {
      {
        ?focus_node ?pred ?match .
        SERVICE <https://qlever.cs.uni-freiburg.de/textSearch/> {
          ?t <https://qlever.cs.uni-freiburg.de/textSearch/contains> [<https://qlever.cs.uni-freiburg.de/textSearch/word> "boulders";<https://qlever.cs.uni-freiburg.de/textSearch/score> ?weight] .
          ?t <https://qlever.cs.uni-freiburg.de/textSearch/contains> [<https://qlever.cs.uni-freiburg.de/textSearch/word> "large";<https://qlever.cs.uni-freiburg.de/textSearch/score> ?weight] .
          ?t <https://qlever.cs.uni-freiburg.de/textSearch/contains> [<https://qlever.cs.uni-freiburg.de/textSearch/entity> ?match;<https://qlever.cs.uni-freiburg.de/textSearch/score> ?entity_score] .
        }
      }?geom_bnode <http://www.opengis.net/ont/geosparql#asWKT> ?geom_var .
      ?focus_node <http://www.opengis.net/ont/geosparql#hasGeometry> ?geom_bnode .
      FILTER <http://www.opengis.net/def/function/geosparql/sfIntersects>(?geom_var, "<http://www.opengis.net/def/crs/OGC/1.3/CRS84> POLYGON ((121.547957555144200 -22.203073584506715, 119.253348181453150 -24.627507058047406, 117.306406894685010 -21.880817343984596, 121.547957555144200 -22.203073584506715))"^^<http://www.opengis.net/ont/geosparql#wktLiteral>)
    }ORDER BY DESC( ?weight )
    LIMIT 11 OFFSET 0
  }?focus_node <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?prof_1_node_1 .
}

Non geosparql spatial + text search - from memory this performed well but the text query needed exact matches when multiple words were used. Perhaps trying again with my third query with UNION would work.

PREFIX qlss: <https://qlever.cs.uni-freiburg.de/spatialSearch/>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX textSearch: <https://qlever.cs.uni-freiburg.de/textSearch/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX sdo: <https://schema.org>
PREFIX dcterms: <http://purl.org/dc/terms/>
SELECT *
WHERE {
  VALUES ?bbox_wkt { "POLYGON((113.338953 -43.634597, 153.569469 -43.634597, 153.569469 -10.668185, 113.338953 -10.668185, 113.338953 -43.634597))"}
  SERVICE qlss: {
    _:config qlss:algorithm qlss:libspatialjoin ;
             qlss:left ?bbox_wkt ;
             qlss:right ?wkt ;
             qlss:payload ?focus_node , ?match , ?weight ;
             qlss:joinType qlss:intersects .
    {
      ?focus_node geo:hasGeometry / geo:asWKT ?wkt .
      SERVICE textSearch: {
        ?t textSearch:contains [
            textSearch:word "breccia" ;
            textSearch:score ?weight
        ] .
        ?t textSearch:contains [
            textSearch:word "under" ;
            textSearch:score ?weight
        ] .
        ?t textSearch:contains [
            textSearch:entity ?match ;
            textSearch:score ?entity_score
        ] .
      }
      ?focus_node ?pred ?match
    }
  }
} ORDER BY DESC(?weight)

Non spatial search - multiple keywords - would be good if there was a way to not need the UNION clauses - I couldn't figure out a way/see any documentation on how to do multiple word searches which didn't require an exact match of all words

PREFIX qlss: <https://qlever.cs.uni-freiburg.de/spatialSearch/>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX textSearch: <https://qlever.cs.uni-freiburg.de/textSearch/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX sdo: <https://schema.org/>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX prez: <https://prez.dev/>
SELECT *
  {
    {
      SELECT (SUM(?weight) AS ?totalWeight) ?match ?focus_node (URI(CONCAT("urn:hash:", SHA256(CONCAT(STR(?focus_node), STR(?totalWeight))))) AS ?hashID)
      WHERE {
        {
        {
          SERVICE textSearch: {
            ?t textSearch:contains [
                textSearch:word "geological" ;
                textSearch:score ?weight
            ] .
            ?t textSearch:contains [
                textSearch:entity ?match ;
                textSearch:score ?entity_score
            ] .
          }
        }
        UNION
        {
          SERVICE textSearch: {
            ?t textSearch:contains [
                textSearch:word "survey" ;
                textSearch:score ?weight
            ] .
            ?t textSearch:contains [
                textSearch:entity ?match ;
                textSearch:score ?entity_score
            ] .
          }
        }
        UNION
        {
          SERVICE textSearch: {
            ?t textSearch:contains [
                textSearch:word "tasmania" ;
                textSearch:score ?weight
            ] .
            ?t textSearch:contains [
                textSearch:entity ?match ;
                textSearch:score ?entity_score
            ] .
          }
        }
          ?focus_node ?pred ?match .
          ?focus_node a sdo:Dataset .
          FILTER(?pred != prez:searchResultMatch)
        }
      }
      GROUP BY ?focus_node ?match
      ORDER BY DESC(?totalWeight)
      LIMIT 10
    }
    ?focus_node sdo:name|rdfs:label ?label
    OPTIONAL {?focus_node geo:hasGeometry ?geom_bn . ?geom_bn geo:asWKT ?wkt}
    OPTIONAL {?focus_node sdo:distribution ?distrib . ?distrib ?p ?o}
  }

[hannahbast]

@recalcitrantsupplant Here is the documentation regarding QLever's current GeoSPARQL support: https://docs.qlever.dev/geosparql

Regarding efficiency, there is a lot to say and we should take it one at a time. QLever's core spatial library is extremely efficient. But its integration into QLever is not completed yet. An example of a query that is very fast already now (all streets in Australia, click on "Analyze" to see the progress while the query is computing): https://qlever.dev/osm-planet/WvzjIg

Do you have a very concrete example of a (simple) query that you think should be fast but it's currently not?

[hannahbast]

PS: I just saw that your original post already contained some queries. It would be good to have the dataset on which these queries work. And it would be good to simplify the queries. Almost always, when a query is slow or does not work, you can reproduce that behavior with a relatively simple query

[recalcitrantsupplant]

Unfortunately I cannot share the dataset those queries are used with, however I may soon have other similar ones which I can share.

The main pattern for our queries is a user panning around a web map, or selecting an area of interest on a map to find features within it. So one geometry in our queries is supplied at runtime.

[hannahbast]

@recalcitrantsupplant Did you also read my previous comment, regarding https://docs.qlever.dev ?

The use case you mention (display all objects that match a keyword and are contained in a given viewport) should work fine with QLever already now. And we are currently working on some improvements that should make this kind of query super fast.

There are some peculiarities in the queries from your first comment in this thread. First, prefixes would help to make the queries much more readable. Second, you are using the same ?weight variable for two different keywords, that probably does not make sense. Third, the query as it is written currently does not work with QLever because both arguments to geof:sfIntersects have to be variables (which you can always achieve using a VALUES clause); so we don't understand what you mean when you say that the query is slow

[recalcitrantsupplant]

I'm coming back to this having not touched it for a few months - I did read those docs (docs.qlever.dev) some time ago - when I came back yesterday I only found https://github.com/ad-freiburg/qlever/tree/master/docs - it would be good to have a link from the main readme to docs.qlever.dev.

The first query above would be my preferred format for the spatial part - it probably didn't run (consistent with your comment), and so I went back to the other SERVICE qlss format. I'll have another go writing some queries with your feedback.

Am I right in thinking multiple word text search is not currently supported, other than exact matches? For example something like searching for "large regions" weights results with just large or just regions as 1 point, both words in the text 5 points.