Refactor Query Module

[hanahmily]

We have to refactor the query module before introducing nested predicators( conditions ) and distributed topology later.

Logical Plan

We didn't get insights into the difference between logical and physical plans. A logical plan tends to convert relational calculus, whose classic implementation is SQL, to relational algebra(a set of operations). BanyanDB doesn't support any calculus, SQL, or its variants. The query APIs are algebra expressions. What the query module implemented are the physical plan and query execution.

We used a wrong denote to query components and should correct this misuse.

Reusable Plan

Each query would create a new physical plan instead of reusing an existing one. It is not the query plan's goal. Since most queries' behaviors are immutable, we should construct a reusable plan system to accelerate the query process.

Query process

The traditional query process consists of a logical plan and a physical plan. But the second layer, the physical plan, will play a more critical role in distributed querying.

In a distributed cluster, it could combine:

• Data localization. Find and restrict the data fragments.
• Disturbed optimization. Use environment parameters, cost function, and statistics to optimize the execution topology and ordering.
• Distributed execution. Use the iterator model to execute the local and remote queries.

The current query module would belong to the local execution. If this is the case, we should merge query/executor and query/logical into query/execution since the physical plan is a sub-task of the local execution.

^^ @lujiajing1126

[lujiajing1126]

Logical Plan

We didn't get insights into the difference between logical and physical plans. A logical plan tends to convert relational calculus, whose classic implementation is SQL, to relational algebra(a set of operations). BanyanDB doesn't support any calculus, SQL, or its variants. The query APIs are algebra expressions. What the query module implemented are the physical plan and query execution.

We used a wrong denote to query components and should correct this misuse.

I would like to complement this part a bit. The current implementation actually consists of two parts,

• UnresolvedPlan: Directly constructed from users' requests (i.e. they are equivalent). It is a form of unresolved logical plan since it simply contains some algebra and does not understand the underlying data types and schema.
• ResolvedPlan: the resolved plan is physical plan which can be executed directly with the help of ExecutionContext providing the storage module.

We currently just skip the generation of (resolved) Logical Plan (which resolves the schema and data types) generation and rule-based optimizer since we provide very limited API/semantics to clients.

Query process

The traditional query process consists of a logical plan and a physical plan. But the second layer, the physical plan, will play a more critical role in distributed querying.

In a distributed cluster, it could combine:

• Data localization. Find and restrict the data fragments.
• Disturbed optimization. Use environment parameters, cost function, and statistics to optimize the execution topology and ordering.
• Distributed execution. Use the iterator model to execute the local and remote queries.

The current query module would belong to the local execution. If this is the case, we should merge query/executor and query/logical into query/execution since the physical plan is a sub-task of the local execution.

Yes. I agree that the current implementation is only a subset of the whole physical plans. The ability of distributed query can be fulfilled by implementing distributed operators of physical plans.

Besides, we may need physical planner as a bridge to produce multiple physical plans and choose a best one based on the cost in the future.

[hanahmily]

The current plan structure is physical. Unlike the relational model, banyandb doesn't need the join operation, which is the core of the plan optimization. That means we could directly transform the request to the plan without much reduction or reordering.

The primary optimization process will take place in distributed phase. As I mentioned, we don't have the join. The cost function's parameters would be memory size, CPU usage, network through/latency, etc., instead of cardinality histogram, which helps relational query engine generate join topology.