Research and implementation directions for Q4 2017

[szarnyasg]

Today I released v1.0 and would like to lay out future directions.

Fortunately, the benchmark is actively used by academic researchers. Graph processing is also a hot topic in academic research and by far the most trending database technology in the last 5 years with a wide palette of uses cases.

In the future, I'd like to shift the benchmark to graph processing engines, which has a couple of consequences:

1. This would be difficult while having to maintain 10+ tools, so I'll need to focus my efforts a bit, and that means removing some of the existing implementations. They will still be available in the history and the released 1.0 version, but will not be continuously developed along the changes.

2. I'll also remove the code responsible for measuring memory consumption as this is an aspect that's very difficult to get right even in a single JVM, and often meaningless with a disk-based database.

Meanwhile, I propose the following improvements:

1. We should add more implementations from the graph database/graph query engine landscape. Currently, there are two working implementations:

   • Neo4j - API and Cypher implementation
   • TinkerGraph - API implementation

   There are also work in progress implementations:

   • Graphflow
   • Dgraph

   And potential candidates:

   • TinkerGraph - Gremlin implementation
   • JanusGraph
   • OrientDB

   Both JanusGraph and OrientDB are compatible with TinkerPop v3, which allows us to create an initial implementation quite quickly.

2. We should add more queries. There are two particular aspects that are common in graph queries, but missing in our current workload:

   • aggregation (even something as simple as count)
   • transitive closure

3. As a consequence of the previous point, we should update the generator to allow the injection of errors for the new validation queries.