How coupled are SPARQL-Anything and Fuseki TDB?

[mielvds]

Hi! This is a question mostly in context of using the ondisk cache: how easy is it to replace Fuseki TDB2 with some other triple store and gain performance/scalability when using the ondisk setting?
Is it a matter of implementing or using an implementation of the DatasetGraph interface or does it use more low-level features? And in that sense, how much processing can you push down to the triple store and how? Thanks!

[justin2004]

here is where the DatasetGraph is initialized:
https://github.com/SPARQL-Anything/sparql.anything/blob/v1.0-DEV/sparql-anything-model/src/main/java/io/github/sparqlanything/model/BaseFacadeXGraphBuilder.java#L52

i think if you didn't want to use native jena TDB2 you could see if you get get a connection to some sparql endpoint that could be represented by an instance of a jena DatasetGraph

[enridaga]

Having alternative on-disk cache methods would be nice.

[mielvds]

Definitely, but, given that DatasetGraph is a pretty restricted interface (triple patterns & graphs), to really benefit from another engine, meaning indexes, joins and the like, you will have to go higher up the SA stack where the queries are executed, which is non-trivial I guess. On the DatasetGraph level, TDB2 will be hard to beat being native and all.

[justin2004]

@mielvds , if you are thinking about the cache as a performance booster for large input files i think what might be more useful is a different slicing method:

• chunk input files
• run query against chunks in parallel and output quads (or a line oriented serialization) to disk
• when all chunks are done concatenate all the files together
• optionally re-serialize (which would dedup)

that is how we've gotten the best performance out of sparql anything in production for large input files but we scripted that around sparql anything. it would be nice if it was all internal.

[mielvds]

wow @justin2004 that's neat. If you can share anything; as you might have noticed, I'm looking for ways and examples to use sparql anything at scale. This would also be helpful for taking things internal.
And how do you deal with inter-chunk references or do they not occur?

And wrt to the cache: alternative on-disk cache methods might still be useful, but then I'm curious what they can add that TDB2 has not. I currently have no idea how and for what the cache is used by sparql anything

[justin2004]

And how do you deal with inter-chunk references or do they not occur?

that is handled by IRI alignment only (since the invocations of sparql anything are in parallel they don't share variable bindings). but usually this is just fine since it would usually be acceptable to run a construct query on 1 row of a csv at a time (or 1 json object in an array of objects).
what you can't do with this approach is whole file aggregations (e.g. max of some column) since a query execution only sees 1 slice at a time.

i think semantically this approach is the same as the current sparql anything slice capability BUT that implementation doesn't scale well. it is been a while since i looked but i believe it doesn't scale well because it is still building a large (indexed) datasetgraph that includes the output of each slice execution. and loading billions of triples into a TDB2 one transaction at a time starts to slow down BUT simply outputting quads to disk and concatenating at the end is super fast.

I currently have no idea how and for what the cache is used by sparql anything

i used to use the on-disk cache option to load the facade representation in a TDB2 on disk and then just iterate on my construct query. i got that behavior by setting the on-disk option to true (with a path to store the TDB2 at) and then specifying the location of my large input file. then the next query i deleted the reference to my large input file but since the cache was still on the facade RDF was still in the TDB2 but i didn't want to wait for it to get triplified again.

basically that is a shortcut for running a construct {?s ?p ?o} and loading it into fuseki and then running constructs against fuseki.

[enridaga]

BUT that implementation doesn't scale well. it is been a while since i looked but i believe it doesn't scale well because it is still building a large (indexed) datasetgraph that includes the output of each slice execution. and loading billions of triples into a TDB2 one transaction at a time starts to slow down BUT simply outputting quads to disk and concatenating at the end is super fast.

We should integrate this approach.