HIVE-20189: Separate metastore client code into its own module #5924

deniskuzZ · 2025-07-03T09:19:47Z

What changes were proposed in this pull request?

move client classes into it's own module

Why are the changes needed?

improve the structure of standalone-metastore classes

Does this PR introduce any user-facing change?

No

How was this patch tested?

jenkins

dengzhhu653 · 2025-07-03T11:14:07Z

standalone-metastore/metastore-client/pom.xml

+  <dependencies>
+    <dependency>
+      <groupId>org.apache.hive</groupId>
+      <artifactId>hive-standalone-metastore-common</artifactId>


The new module depends on the metastore-common, which doesn't reduce the size or dependencies from metastore-common, not sure if it's ok, for the user what will benefit from the new module?

It's expected for a client module to depend on a common module. The client module delivers core client functionality and can be enhanced later with features like caching.
Offering a 'client' JAR aligns with common conventions and is expected to be user-friendly.
ATM we have 2 distinct cache wrappers in different Hive modules, just because there was no structure:

org.apache.hadoop.hive.metastore.HiveClientCache

org.apache.iceberg.hive.CachedClientPool

another point I don't understand, why ql and beeline modules depend on metastore-server? I think we should drop the dependency on server and move the classes into the metastore common or client.

I basically agree that we can potentially improve the structure in the future. We will have obvious guidelines about how to organize files.

metastore-client: Client-specific files, which Server doesn't need

metastore-server: Server-specific files, which Client doesn't need

metastore-common: Common modules

another point I don't understand, why ql and beeline modules depend on metastore-server? I think we should drop the dependency on server and move the classes into the metastore common or client.

I guess ql requires metastore-server to use an embedded HMS.

isn't embedded HMS only used in tests?

No, it isn't. It can be used when metastore.thrift.uris is empty. For example, our Hive docker image(not HMS docker image) can set up a HiveServer2 without HMS. It probably uses the embedded mode that runs HMS-equivalent threads in HS2.

hive/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java

Lines 2654 to 2663 in d7b3a5b

/**

* Check if metastore is being used in embedded mode.

* This utility function exists so that the logic for determining the mode is same

* in HiveConf and HiveMetaStoreClient

* @param msUri - metastore server uri

* @return true if the metastore is embedded

*/

public static boolean isEmbeddedMetaStore(String msUri) {

return (msUri == null) || msUri.trim().isEmpty();

}

Alternatively, we can say it is for testing purposes and separate the classes from hive-exec.

okumin

+1

okumin · 2025-07-14T14:05:48Z

hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java

+        Configuration conf, String userName) throws Exception {
+    Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier> t;
+    try (JobClient jcl = new JobClient(new JobConf(conf, HCatOutputFormat.class))) {
+      t = jcl.getDelegationToken(new Text(userName));


We may directly return it

ngsg

LGTM, +1

ngsg · 2025-07-14T23:17:04Z

...tore/metastore-client/src/main/java/org/apache/hadoop/hive/metastore/utils/TableFetcher.java

 import org.apache.hadoop.hive.metastore.TableType;
 import org.apache.hadoop.hive.metastore.Warehouse;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;


nit: I think the original import is sorted correctly, so we might not need this change.

sonarqubecloud · 2025-07-15T09:37:50Z

Quality Gate passed

Issues
1518 New issues
0 Accepted issues

Measures
1 Security Hotspot
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ requested a review from dengzhhu653 July 3, 2025 09:19

asf-ci-hive added the tests pending label Jul 3, 2025

deniskuzZ mentioned this pull request Jul 3, 2025

HIVE-29062: Add standalone module for packaging the Metastore #5923

Merged

asf-ci-hive added tests failed and removed tests pending labels Jul 3, 2025

dengzhhu653 reviewed Jul 3, 2025

View reviewed changes

deniskuzZ force-pushed the HIVE-20189 branch 2 times, most recently from 5461f07 to 24ca325 Compare July 3, 2025 13:10

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Jul 3, 2025

deniskuzZ force-pushed the HIVE-20189 branch 3 times, most recently from 959dfa7 to bc40344 Compare July 3, 2025 18:26

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Jul 3, 2025

deniskuzZ force-pushed the HIVE-20189 branch from 89b0d74 to 8374043 Compare July 5, 2025 18:20

asf-ci-hive added tests pending and removed tests failed labels Jul 5, 2025

deniskuzZ added 2 commits July 14, 2025 13:20

updated the compiler and javadoc plugin versions

b5314da

review comments iter#1

00db0ae

deniskuzZ force-pushed the HIVE-20189 branch from f3ea566 to ece6837 Compare July 14, 2025 10:23

asf-ci-hive added tests pending and removed tests passed labels Jul 14, 2025

deniskuzZ force-pushed the HIVE-20189 branch from ece6837 to 8f80df6 Compare July 14, 2025 10:24

deniskuzZ requested review from okumin and ngsg July 14, 2025 10:27

asf-ci-hive added tests failed and removed tests pending labels Jul 14, 2025

deniskuzZ force-pushed the HIVE-20189 branch from 8f80df6 to c1a5aab Compare July 14, 2025 12:19

asf-ci-hive added tests pending and removed tests failed labels Jul 14, 2025

okumin approved these changes Jul 14, 2025

View reviewed changes

deniskuzZ force-pushed the HIVE-20189 branch from c1a5aab to 7640a83 Compare July 14, 2025 14:38

asf-ci-hive added tests unstable tests pending tests passed and removed tests pending tests unstable labels Jul 14, 2025

ngsg approved these changes Jul 14, 2025

View reviewed changes

review comments iter#2

3c03a2d

deniskuzZ force-pushed the HIVE-20189 branch from 7640a83 to 3c03a2d Compare July 15, 2025 07:18

asf-ci-hive added tests pending and removed tests passed labels Jul 15, 2025

asf-ci-hive added tests passed and removed tests pending labels Jul 15, 2025

deniskuzZ merged commit ea27ed7 into apache:master Jul 15, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HIVE-20189: Separate metastore client code into its own module #5924

HIVE-20189: Separate metastore client code into its own module #5924

Uh oh!

deniskuzZ commented Jul 3, 2025 •

edited

Loading

Uh oh!

dengzhhu653 Jul 3, 2025

Uh oh!

deniskuzZ Jul 3, 2025 •

edited

Loading

Uh oh!

deniskuzZ Jul 3, 2025 •

edited

Loading

Uh oh!

okumin Jul 6, 2025

Uh oh!

deniskuzZ Jul 9, 2025

Uh oh!

okumin Jul 10, 2025

Uh oh!

okumin Jul 10, 2025

Uh oh!

okumin left a comment

Uh oh!

okumin Jul 14, 2025

Uh oh!

deniskuzZ Jul 14, 2025

Uh oh!

ngsg left a comment

Uh oh!

ngsg Jul 14, 2025

Uh oh!

deniskuzZ Jul 15, 2025

Uh oh!

sonarqubecloud bot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

	/**
	* Check if metastore is being used in embedded mode.
	* This utility function exists so that the logic for determining the mode is same
	* in HiveConf and HiveMetaStoreClient
	* @param msUri - metastore server uri
	* @return true if the metastore is embedded
	*/
	public static boolean isEmbeddedMetaStore(String msUri) {
	return (msUri == null) \|\| msUri.trim().isEmpty();
	}

HIVE-20189: Separate metastore client code into its own module #5924

HIVE-20189: Separate metastore client code into its own module #5924

Uh oh!

Conversation

deniskuzZ commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngsg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jul 15, 2025

Quality Gate passed

Uh oh!

Uh oh!

Uh oh!

deniskuzZ commented Jul 3, 2025 •

edited

Loading

deniskuzZ Jul 3, 2025 •

edited

Loading

deniskuzZ Jul 3, 2025 •

edited

Loading