Querying s3 file data directly using Apache doris S3 Tabled Value Function

[Susmit07]

Hello Community Engineers,

We are looking to query S3 objects directly using the Apache Doris S3 Table-Valued Function (TVF) and would like to leverage Apache Arrow Flight SQL Server for rendering the data.

Is it possible to use the Apache Doris S3 Table-Valued Function with an Arrow Flight SQL Client code (can be in Python or Scala)?

Any examples or insights would be greatly appreciated!

Thanks in advance.

[xinyiZzz]

Hi @Susmit07

In theory, you can use ADBC to submit any query supported by Doris, including queries on S3 objects. look at the examples in the documentation (https://doris.apache.org/docs/dev/db-connect/arrow-flight-sql-connect?_highlight=arrow&_highlight=flight#python-usage) to use the Python ADBC client. other users have used the Scala client, but there are currently no examples, this is a TODO.

If you run into any issues, just add them to the comments.. Good luck.

[Susmit07]

Thanks with the response closing it for now