-
Notifications
You must be signed in to change notification settings - Fork 5.5k
Description
Issue Details: In Presto, we have Arrow connector which uses Arrow Flight protocol to connect with remote datasources. Some IBM connectors like IBM GFS Connector extent Arrow connector to communicate with the Queryplex server—a backend service capable of connecting to multiple data sources and is responsible for federated query pushdown. This Queryplex server needs the remote database column definitions too along with the Presto generated sql to properly execute and pushdown queries.
Currently, ArrowColumnHandle contains the column name and the column type, where the type is converted from Queryplex metadata into a Presto-compatible type. These are used for holding query execution result in Presto. However, for Queryplex execution, it also need to propagate the remote column definition to the connector. To enable this, we need to introduce a new member variable in ArrowColumnHandle, to store additional informations like original metadata column definition.
Presto Component, Service, or Connector
Changes required in Arrow Connector service and the component name is presto-base-arrow-flight
Possible Implementation
Could add additionalProperties in ArrowColumnHandle, which holds all the additional properties in a map structure.