HIVE-29130: Remove jline 2.x #6054

Aggarwal-Raghav · 2025-09-01T16:13:02Z

What changes were proposed in this pull request?

Although Hive has moved to Jline3, 2.x was also getting shipped in packaging.

Why are the changes needed?

To have single Jline jar. Check HIVE-29130 for more details

Does this PR introduce any user-facing change?

No

How was this patch tested?

On local setup

Aggarwal-Raghav · 2025-09-01T16:17:30Z

As Hadoop has jline 3.9.0**, in my setup** it was getting picked first compared to 3.25.0 from hive. Causing NoSuchMethodError org.jline.reader.EndOfFileException#getPartialLine() check here.

I would like the reviewer to confirm on this.

Aggarwal-Raghav · 2025-09-01T16:17:50Z

CC @ayushtkn @abstractdog

abstractdog · 2025-09-02T06:47:03Z

thanks @Aggarwal-Raghav for taking care of this, would you be so kind to add an enforcer item so ban this dependency even transitively in the future? that would be the perfect solution I believe

Aggarwal-Raghav · 2025-09-02T06:51:06Z

thanks @Aggarwal-Raghav for taking care of this, would you be so kind to add an enforcer item so ban this dependency even transitively in the future? that would be the perfect solution I believe

sure. Will update the PR.

Aggarwal-Raghav · 2025-09-02T14:44:28Z

packaging/src/main/assembly/beeline.xml

@@ -48,8 +48,7 @@
                <include>org.apache.hive:hive-service-rpc:jar</include>
                <include>commons-cli:commons-cli:jar</include>
                <include>commons-io:commons-io:jar</include>
-                <include>commons-logging:commons-logging:jar</include>


removing commons-logging as it was getting packaged in beeline tarball. Check HIVE-24691 and HIVE-20019

Aggarwal-Raghav · 2025-09-02T14:44:50Z

packaging/src/main/assembly/beeline.xml

@@ -48,8 +48,7 @@
                <include>org.apache.hive:hive-service-rpc:jar</include>
                <include>commons-cli:commons-cli:jar</include>
                <include>commons-io:commons-io:jar</include>
-                <include>commons-logging:commons-logging:jar</include>
-                <include>jline:jline:jar</include>


moved it to jline3.x groupId

Aggarwal-Raghav · 2025-09-02T14:45:54Z

standalone-metastore/metastore-common/pom.xml

-            <goals>
-              <goal>enforce</goal>
-            </goals>
-            <configuration>


removed the enforcer pluging from standalone-metastore/metastore-common/pom.xml and standalone-metastore/metastore-server/pom.xml and moved to standalone-metastore/pom.xml i.e. a common place and updated it with the same list of banned dependency as in parent pom.xml

- jline2.x has been added to enforcer plugin - commons-logging direct dependency has been removed from standalone-metastore module - enforcer plugin in standalone-metastore is now same as of parent pom.xml

Aggarwal-Raghav · 2025-09-03T06:42:06Z

The unit test failures in this PR are relevant. The reason for this is that Pig version 0.16.0 uses jline 1.x. The jline dependency is being excluded from Pig due to an enforcer rule, which is causing failures specifically in the hcatalog/hcatalog-pig-adapter module.
I believe the current unit test scenario in the cluster/runtime environment might also be failing because we are not shipping jline 1.x. Assumption: This would only work if jline 2.x, which is being shipped without this PR, is backward compatible with version 1.x.

My recommendation for the next step is to remove the jline enforcer rule from the parent pom.xml, keeping the exclusions everywhere else except hcatalog/hcatalog-pig-adapter. Is that the approach acceptable? Still with this approach [2] i.e. pig working in cluster is unknown.

possible_follow_up.patch

CC @abstractdog

ayushtkn · 2025-09-03T07:12:41Z

As Hadoop has jline 3.9.0**, in my setup** it was getting picked first compared to 3.25.0 from hive. Causing NoSuchMethodError org.jline.reader.EndOfFileException#getPartialLine() check here.

Hadoop's JLine 3.9.0 it was getting picked up first. So, we should make sure Hive Classpath is always before HADOOP jars.

I don't understand in that scope or for this problem why are we playing with Jline-2.x?

I am not very convinced on enforcing some places and not on some. We might not have a use case or expertise with Pig, so better not to touch it in that case. We had a thread around Pig to depreacate/remove some time back & we were shot down, so people tend to use it

Why not upgrade JLine in Hadoop?

abstractdog · 2025-09-03T07:17:22Z

The unit test failures in this PR are relevant. The reason for this is that Pig version 0.16.0 uses jline 1.x. The jline dependency is being excluded from Pig due to an enforcer rule, which is causing failures specifically in the hcatalog/hcatalog-pig-adapter module.

I believe the current unit test scenario in the cluster/runtime environment might also be failing because we are not shipping jline 1.x. Assumption: This would only work if jline 2.x, which is being shipped without this PR, is backward compatible with version 1.x.

My recommendation for the next step is to remove the jline enforcer rule from the parent pom.xml, keeping the exclusions everywhere else except hcatalog/hcatalog-pig-adapter. Is that the approach acceptable? Still with this approach [2] i.e. pig working in cluster is unknown.

possible_follow_up.patch

CC @abstractdog

it is what it is, thanks @Aggarwal-Raghav for checking, but before proceeding with this, can you please check if there is an include/exclude possibility in bannedDependencies to allow old jline transitively through pig only, does it make sense?

Aggarwal-Raghav · 2025-09-03T07:30:50Z

Thanks for the reply @ayushtkn.

I don't understand in that scope or for this problem why are we playing with Jline-2.x?

In hive packaging, both jline 2.11 and jline 3.25.0 are getting shipped. IMO, its wrong. The aim of this PR is to ship only jline3.x jar. If we are ok shipping both the version then I'll close this PR.

Why not upgrade JLine in Hadoop?

It's good to have same dependency version across components. In my setup, Jline 3.9.0 is getting picked up first which is causing NoSuchMethodError. -- I want to confirm if its due to my setup or not.

In conclusion,

if Hadoop Classpath first, then jline 3.9.0 is picked => hadoop has to upgrade jline to 3.25.0
if hive classpath is first, then we are all good.

ayushtkn · 2025-09-03T07:36:36Z

In conclusion,
if Hadoop Classpath first, then jline 3.9.0 is picked => hadoop has to upgrade jline to 3.25.0
if hive classpath is first, then we are all good.

So, the solution is like in ideal setup Hive Classpath should be first. In that case we are sorted. Else for the problem that you reported. Hadoop needs to upgrade JLine, correct?

And this PR is solving a different problem as reported in the ticket, this won't solve the problem that you reported?

Aggarwal-Raghav · 2025-09-03T07:37:44Z

In conclusion,
if Hadoop Classpath first, then jline 3.9.0 is picked => hadoop has to upgrade jline to 3.25.0
if hive classpath is first, then we are all good.

So, the solution is like in ideal setup Hive Classpath should be first. In that case we are sorted. Else for the problem that you reported. Hadoop needs to upgrade JLine, correct?

And this PR is solving a different problem as reported in the ticket, this won't solve the problem that you reported?

yes and yes.
In addition, I found 2 problems:

commons-logggins is also getting shipped which is big no-no based on HIVE-24691 and HIVE-20019
enforcer plugin defined in parent pom is not enforced on standalone-metastore module. IMO, the banned dependencies should be same.

Aggarwal-Raghav · 2025-09-03T10:22:38Z

can you please check if there is an include/exclude possibility in bannedDependencies to allow old jline transitively through pig only, does it make sense?

we can use include tag but there are 2 ways:
1. Do it on parent pom level: include-tag.patch

Benefit: Single place enforcer is defined. So, easy to maintain.
Drawback: For all the module jline 1.0 is enabled. But it won't be shipped in packaging. As we have excluded it from all problematic places flagged by enforcer plugin so it shouldn't be a problem.

2. Do it on hcatalog/hcatalog-pig-adapter/pom.xml level: include-tag-child-pom.patch

Benefit: Only for hcatalog/hcatalog-pig-adapter jline 1.0 is enabled, rest all modules are not impacted.
Drawback: Copy of enforcer plugin needs to be defined and maintainability of enforcer plugin for new bannedDependencies can be missed for this child pom.

@abstractdog , please let me know which is better.

abstractdog · 2025-09-03T10:46:47Z

can you please check if there is an include/exclude possibility in bannedDependencies to allow old jline transitively through pig only, does it make sense?

we can use include tag but there are 2 ways: 1. Do it on parent pom level: include-tag.patch

Benefit: Single place enforcer is defined. So, easy to maintain.

Drawback: For all the module jline 1.0 is enabled. But it won't be shipped in packaging. As we have excluded it from all problematic places flagged by enforcer plugin so it shouldn't be a problem.

2. Do it on hcatalog/hcatalog-pig-adapter/pom.xml level: include-tag-child-pom.patch

Benefit: Only for hcatalog/hcatalog-pig-adapter jline 1.0 is enabled, rest all modules are not impacted.

Drawback: Copy of enforcer plugin needs to be defined and maintainability of enforcer plugin for new bannedDependencies can be missed for this child pom.

@abstractdog , please let me know which is better.

I would go for 1), I don't like the whole copied enforcer plugin config in 2), thanks a lot for investigating all the possibilities!

Aggarwal-Raghav · 2025-09-03T11:16:44Z

UT passed in local.

cd hcatalog/hcatalog-pig-adapter ; mvn clean test -Dtest=TestTextFileHCatLoader,TestRCFileHCatLoader,TestAvroHCatLoader,TestSequenceFileHCatLoader,TestParquetHCatLoader,TestOrcHCatLoader -Drat.skip

Pushed the changes. Will see CI output.

abstractdog · 2025-09-03T11:19:02Z

pom.xml

+                    <exclude>jline:jline</exclude>
+                  </excludes>
+                  <includes>
+                    <include>jline:jline:1.0</include>


nit: please explain here with a brief comment why we still allow 1.0 to make it clear for the future if we can remove

abstractdog · 2025-09-03T12:37:29Z

thanks @Aggarwal-Raghav, I'm good now with the pom.xml changes, but haven't followed if you've reached consensus with @ayushtkn on the other aspect of this PR, please let me know guys

Aggarwal-Raghav · 2025-09-03T15:08:41Z

@ayushtkn, can you please take a look once!

sonarqubecloud · 2025-09-04T04:35:06Z

Quality Gate passed

Issues
112 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Aggarwal-Raghav · 2025-09-11T17:24:27Z

gentle ping for review @ayushtkn !

ayushtkn · 2025-09-11T17:41:23Z

Stuck with some stuff. If @abstractdog is happy. I am happy. feel free to move ahead :-)

asf-ci-hive added the tests pending label Sep 1, 2025

asf-ci-hive added tests unstable and removed tests pending labels Sep 1, 2025

Aggarwal-Raghav force-pushed the HIVE-29130 branch from 8c93883 to ca3d41d Compare September 1, 2025 20:28

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Sep 1, 2025

Aggarwal-Raghav force-pushed the HIVE-29130 branch from ca3d41d to 180afc3 Compare September 2, 2025 14:43

asf-ci-hive added tests pending and removed tests passed labels Sep 2, 2025

Aggarwal-Raghav commented Sep 2, 2025

View reviewed changes

HIVE-29130: Remove jline 2.x

c07490a

- jline2.x has been added to enforcer plugin - commons-logging direct dependency has been removed from standalone-metastore module - enforcer plugin in standalone-metastore is now same as of parent pom.xml

Aggarwal-Raghav force-pushed the HIVE-29130 branch from 180afc3 to c07490a Compare September 2, 2025 15:08

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Sep 2, 2025

Fix UT failures in hcatalog/pig-adapter module

2f0c89b

asf-ci-hive added tests pending and removed tests unstable labels Sep 3, 2025

abstractdog reviewed Sep 3, 2025

View reviewed changes

asf-ci-hive added tests passed and removed tests pending labels Sep 3, 2025

asf-ci-hive added tests pending and removed tests passed labels Sep 3, 2025

Address review comments

9f3d5d3

Aggarwal-Raghav force-pushed the HIVE-29130 branch from bf62265 to 9f3d5d3 Compare September 4, 2025 02:37

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Sep 4, 2025

asf-ci-hive added tests passed and removed tests pending labels Sep 4, 2025

HIVE-29130: Remove jline 2.x #6054

Are you sure you want to change the base?

HIVE-29130: Remove jline 2.x #6054

Uh oh!

Conversation

Aggarwal-Raghav commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Sep 1, 2025

Uh oh!

Aggarwal-Raghav commented Sep 1, 2025

Uh oh!

abstractdog commented Sep 2, 2025

Uh oh!

Aggarwal-Raghav commented Sep 2, 2025

Uh oh!

Aggarwal-Raghav Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn commented Sep 3, 2025

Uh oh!

abstractdog commented Sep 3, 2025

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025

Uh oh!

ayushtkn commented Sep 3, 2025

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025

Uh oh!

abstractdog commented Sep 3, 2025

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025

Uh oh!

abstractdog Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Sep 3, 2025

Uh oh!

Aggarwal-Raghav commented Sep 3, 2025

Uh oh!

sonarqubecloud bot commented Sep 4, 2025

Quality Gate passed

Uh oh!

Aggarwal-Raghav commented Sep 11, 2025

Uh oh!

ayushtkn commented Sep 11, 2025

Uh oh!

Uh oh!

Aggarwal-Raghav commented Sep 1, 2025 •

edited

Loading

Aggarwal-Raghav commented Sep 3, 2025 •

edited

Loading

Aggarwal-Raghav commented Sep 3, 2025 •

edited

Loading