[FLINK-30274] Bump commons-collections 3.x to commons-collections4

[chucheng92]

What is the purpose of the change

Apache commons-collections 3.x is a Java 1.3 compatible version, and it does not use Java 5 generics. Apache commons-collections4 4.4 is an upgraded version of commons-collections and it built by Java 8. So we bump commons-collections.

Brief change log

update pom

Verifying this change

This change is a trivial rework / code cleanup. Verifying by current total cases.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• f7c796e Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[MartijnVisser]

This is not a replacement, but just adding collections4 without using it. Where's the value in that?

I would expect that a replacement also shows the refactoring of the current usage of commons-collections to commons-collections4.

[chucheng92]

This is not a replacement, but just adding collections4 without using it. Where's the value in that?

I would expect that a replacement also shows the refactoring of the current usage of commons-collections to commons-collections4.

Add it in dependencyManagement to let the submodules match with this version in later using.

[MartijnVisser]

Again, why would we add a dependency without using it? There's no value in that. Please change current usage of commons-collections to commons-collections4. Else there's no value in the PR and I'll have to close it.

[chucheng92]

Again, why would we add a dependency without using it? There's no value in that. Please change current usage of commons-collections to commons-collections4. Else there's no value in the PR and I'll have to close it.

Yes. the best solution is replacing it. U can close this pr, but pls keep issue open, i will try to replace it truely.

[chucheng92]

@MartijnVisser PTAL, thanks.

[MartijnVisser]

@MartijnVisser PTAL, thanks.

@chucheng92 The PR can't pass the CI because it doesn't contain the necessary changes to the NOTICE files (where's there's still an outdated reference to the previous versions of commons-collections and none to the new one).

[chucheng92]

@MartijnVisser thanks. Is this caused by updating the commons-collections4? It add some transitive dependencies?

INFO org.apache.flink.tools.ci.licensecheck.NoticeFileChecker [] - Problems were detected for a NOTICE file.
flink-sql-connector-hbase-2.2:
These issues are assumed to be false-positives:
Dependency com.fasterxml.jackson.core:jackson-core:2.4.0 is not bundled, but listed.
Dependency com.google.protobuf:protobuf-java:3.7.1 is not bundled, but listed.
Dependency com.google.code.gson:gson:2.8.5 is not bundled, but listed.
Dependency com.fasterxml.jackson.core:jackson-databind:2.4.0 is not bundled, but listed.
Dependency com.fasterxml.jackson.core:jackson-annotations:2.4.0 is not bundled, but listed.
Dependency org.apache.commons:commons-collections4:4.3 is not bundled, but listed.
Dependency com.google.protobuf:protobuf-java-util:3.7.1 is not bundled, but listed.
Dependency commons-logging:commons-logging:1.1.1 is not bundled, but listed.
Dependency com.google.guava:guava:27.1-jre is not bundled, but listed.
Dependency com.google.guava:failureaccess:1.0.1 is not bundled, but listed.
Dependency io.netty:netty-all:4.1.34.Final is not bundled, but listed.
Dependency commons-cli:commons-cli:1.4 is not bundled, but listed.

Can we add a NOTICE file in flink-core to address this or temporarily we do not do this updating.

[MartijnVisser]

Is this caused by updating the commons-collections4?

Yes

It add some transitive dependencies?

No. Like those lines say These issues are assumed to be false-positives.

The issue lies here https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45877&view=logs&j=b59e5554-36c7-5512-ab1a-b80b74075fce&t=849d419c-1b8f-52b7-e455-d4bc36ec43ad&l=30273

	flink-dist:
		 These issue are legally problematic and MUST be fixed: 
			Dependency org.apache.commons:commons-collections4:4.4 is not listed.
		 These issues are mistakes that aren't legally problematic. They SHOULD be fixed at some point, but we don't have to: 
			Dependency commons-collections:commons-collections:3.2.2 is not bundled, but listed.

[chucheng92]

Is this caused by updating the commons-collections4?

Yes

It add some transitive dependencies?

No. Like those lines say These issues are assumed to be false-positives.

The issue lies here https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45877&view=logs&j=b59e5554-36c7-5512-ab1a-b80b74075fce&t=849d419c-1b8f-52b7-e455-d4bc36ec43ad&l=30273

	flink-dist:
		 These issue are legally problematic and MUST be fixed: 
			Dependency org.apache.commons:commons-collections4:4.4 is not listed.
		 These issues are mistakes that aren't legally problematic. They SHOULD be fixed at some point, but we don't have to: 
			Dependency commons-collections:commons-collections:3.2.2 is not bundled, but listed.

i think i already know how to do it. PR-21048 There is similar work done here. thanks.

[chucheng92]

@flinkbot run azure

[MartijnVisser]

@chucheng92 You need to rebase your PR on the latest master

[chucheng92]

@MartijnVisser thanks. i rebased it. ci passed. PTAL

[MartijnVisser]

@chucheng92 I still see some occurrences of org.apache.commons.collections, can we get completely get rid of everything that currently uses collections 3.x?

[chucheng92]

@MartijnVisser I think we can fix it completely. i will fix this and update pr ASAP. Sorry for my mistake. thanks.

[chucheng92]

[chucheng92]

hi. @MartijnVisser i have updated the pr. And i searched the source code didn't find any other occurs using org.apache.commons.collections. PTAL. thanks.

hi. @MartijnVisser please take a look ?

[MartijnVisser]

@chucheng92 I think in general this is looking good, but I would like to wait until https://issues.apache.org/jira/browse/FLINK-30859 is completed, to avoid another round of syncing towards the Flink Kafka connector repo.

[chucheng92]

@MartijnVisser thanks for explanations. make sense. we will be watching closely about kafka externalization.

[chucheng92]

@MartijnVisser hi, Martijn, https://issues.apache.org/jira/browse/FLINK-30859 has been resolved. Can you help me to check this pr?

[MartijnVisser]

@chucheng92 There are still multiple occurrences of org.apache.commons.collections in the codebase; can you double check please?

[chucheng92]

Hi @MartijnVisser thanks. it seems like some recent code introduced a old usage. I updated it.

please check.

[MartijnVisser]

please check.

The CI is failing for your PR. Please check

[chucheng92]

@MartijnVisser There is a circular reference.

ci failed becase flink-python kafka test case failed.

flink-python test case reference flink-sql-connector-kafka（not uber jar, without commons-collections class）, so search it in flink and cause ClassNotFoundException. So we need update flink-sql-connector-kafka first.

The pr is: apache/flink-connector-kafka#38, when it's fixed this pr can merge, and finally we may remove explicit dependency in flink-connector-kafka repo.

[AlvaroStream]

@MartijnVisser or @chucheng92 do you think this will enter in future releases?

Thanks.