How to configure SparkSQL to access the Hive catalog in Gravitino？

[zhengsg]

I have already configured the Hive Thrift URI in the UI. Following the official documentation, starting SparkSQL and running show databasesand show tables returns results normally. However, select * from xxx and insert into xxx operations throw errors with the following information:
WARN ObjectStore：Failed to get database db, returning NoSuchObjectException
[SCHEMA_NOT_FOUND] The schema db can not found. Verify the spelling and correctness of the schema and catalog.

I have copied the HDFS configuration files to the SparkSQL conf directory.

The SparkSQL comand:
./bin/spark-sql
--conf spark.plugins="org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin"
--conf spark.sql.gravitino.uri=http://127.0.0.1:8090
--conf spark.sql.gravitino.metalake=test
--conf spark.sql.gravitino.client.socketTimeoutMs=60000
--conf spark.sql.gravitino.client.connectionTimeoutMs=60000
--conf spark.sql.warehouse.dir=hdfs://nameservice/hive

[FANNG1]

There are known limitations for using spark-sql client to query hive tables in Gravitino, you could try to set spark.bypass.spark.sql.hive.metastore.jars in Hive catalog properties, see: https://gravitino.apache.org/docs/1.0.0/spark-connector/spark-catalog-hive/#catalog-properties , or use spark-shell instead.

[FANNG1]

Have you tried this parameters? does it works?

[zhengsg]

By specifying spark.sql.hive.metastore.jars, the SELECT query can be executed to retrieve Hive data. However, when executing an INSERT INTO statement, the error NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadTable error occurs. This indicates an API mismatch caused by version incompatibility between Spark and Hive. Therefore, specifying spark.sql.hive.metastore.jarsis not a recommended approach? However, without using Gravitino, and by directly connecting to the Hive metastore using --conf spark.sql.hive.metastore.uris, both SELECT and INSERT statements can be executed normally

[FANNG1]

This indicates an API mismatch caused by version incompatibility between Spark and Hive

Please try to use the correct hive metastore version

[zhengsg]

This indicates an API mismatch caused by version incompatibility between Spark and Hive

Please try to use the correct hive metastore version

I believe the version of my Hive JAR files is correct. I copied the necessary JARs one by one from the Hive metastore service's runtime directory to a separate directory for SparkSQL to operate on Hive tables. Then, I assigned this directory path to spark.sql.hive.metastore.jars. So where exactly does the problem lie? I suggest that the Gravitino Spark plugin should focus on ensuring proper compatibility and adaptation, rather than relying on configuring spark.sql.hive.metastore.jars, as this approach is prone to compatibility conflicts with Spark.

[FANNG1]

There is a limitation for Spark sql client which couldn't overwrite hive metastore uris for build-in spark.sql.hive.metastore.jars configurations. please refer to apache/kyuubi#6362