Classpath is duplicated in native tests with incompatible versions, preventing upgrade to newer JUnit versions

[sbrannen]

Overview

When I run the java-application-with-tests locally I see that the classpath contains JUnit artifacts twice. In this particular case, they are the same versions.

[INFO] Executing: /opt/graalvm/graalvm-ce-java17-22.2.0/Contents/Home/bin/native-image -cp /Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/classes:/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/test-classes:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.8.1/junit-jupiter-api-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.8.1/junit-jupiter-params-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-launcher/1.8.1/junit-platform-launcher-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-engine/1.8.1/junit-platform-engine-1.8.1.jar:/Users/sbrannen/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-commons/1.8.1/junit-platform-commons-1.8.1.jar:/Users/sbrannen/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.8.1/junit-jupiter-engine-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter/5.8.1/junit-jupiter-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-reporting/1.8.1/junit-platform-reporting-1.8.1.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/native-maven-plugin/0.9.14-SNAPSHOT/native-maven-plugin-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-launcher/1.8.1/junit-platform-launcher-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-engine/1.8.1/junit-platform-engine-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-commons/1.8.1/junit-platform-commons-1.8.1.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.14-SNAPSHOT/junit-platform-native-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-console/1.8.1/junit-platform-console-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-reporting/1.8.1/junit-platform-reporting-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter/5.8.1/junit-jupiter-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.8.1/junit-jupiter-api-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.8.1/junit-jupiter-params-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.8.1/junit-jupiter-engine-5.8.1.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/utils/0.9.14-SNAPSHOT/utils-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/graalvm-reachability-metadata/0.9.14-SNAPSHOT/graalvm-reachability-metadata-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.14-SNAPSHOT/junit-platform-native-0.9.14-SNAPSHOT.jar --no-fallback -H:Path=/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target -H:Name=native-tests -Djunit.platform.listeners.uid.tracking.output.dir=/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/test-ids -H:Class=org.graalvm.junit.platform.NativeImageJUnitLauncher --features=org.graalvm.junit.platform.JUnitPlatformFeature

If I change the JUnit versions to 5.9.0/1.9.0, I then see this:

[INFO] Executing: /opt/graalvm/graalvm-ce-java17-22.2.0/Contents/Home/bin/native-image -cp /Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/classes:/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/test-classes:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.9.0/junit-jupiter-api-5.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.9.0/junit-jupiter-params-5.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-launcher/1.9.0/junit-platform-launcher-1.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-engine/1.9.0/junit-platform-engine-1.9.0.jar:/Users/sbrannen/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-commons/1.9.0/junit-platform-commons-1.9.0.jar:/Users/sbrannen/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.9.0/junit-jupiter-engine-5.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter/5.9.0/junit-jupiter-5.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-reporting/1.9.0/junit-platform-reporting-1.9.0.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/native-maven-plugin/0.9.14-SNAPSHOT/native-maven-plugin-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-launcher/1.9.0/junit-platform-launcher-1.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-engine/1.9.0/junit-platform-engine-1.9.0.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-commons/1.9.0/junit-platform-commons-1.9.0.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.14-SNAPSHOT/junit-platform-native-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-console/1.8.1/junit-platform-console-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/platform/junit-platform-reporting/1.8.1/junit-platform-reporting-1.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter/5.8.1/junit-jupiter-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.8.1/junit-jupiter-api-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.8.1/junit-jupiter-params-5.8.1.jar:/Users/sbrannen/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.8.1/junit-jupiter-engine-5.8.1.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/utils/0.9.14-SNAPSHOT/utils-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/graalvm-reachability-metadata/0.9.14-SNAPSHOT/graalvm-reachability-metadata-0.9.14-SNAPSHOT.jar:/Users/sbrannen/.m2/repository/org/graalvm/buildtools/junit-platform-native/0.9.14-SNAPSHOT/junit-platform-native-0.9.14-SNAPSHOT.jar --no-fallback -H:Path=/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target -H:Name=native-tests -Djunit.platform.listeners.uid.tracking.output.dir=/Users/sbrannen/source/native-build-tools/samples/java-application-with-tests/target/test-ids -H:Class=org.graalvm.junit.platform.NativeImageJUnitLauncher --features=org.graalvm.junit.platform.JUnitPlatformFeature

So, the 5.9.0/1.9.0 versions I declared actually get used, because they are first in the classpath, but we still should not include the 5.8.1/1.9.1 versions of JUnit artifacts in the classpath.

Note that this was tested with the Maven support. I did not verify if the same thing happens with the Gradle support.

Related Issues

• JUnitPlatformFeature unexpectedly failed with a JUnitException #541
• Incorrect / incomplete documentation regarding junit-platform-launcher incompatibilities #706

[lazar-mitrovic]

I'm almost certain that bug comes from this line:

native-build-tools/native-maven-plugin/src/main/java/org/graalvm/buildtools/maven/NativeTestMojo.java

Lines 112 to 115 in 3cf0497

	pluginArtifacts.stream()
	.filter(it -> it.getGroupId().startsWith(Utils.MAVEN_GROUP_ID) || it.getGroupId().startsWith("org.junit"))
	.map(it -> it.getFile().toPath())
	.forEach(imageClasspath::add);

Here we unconditionally add JUnit version that is used in our support feature to the image building classpath. This is obviously wrong, but from my testing I found that if I don't specify this explicitly some of our tests fail as our classpath building code misses the JUnit dependency for some reason.

@alvarosanchez can you take a look at this when you have time?

[sbrannen]

See also:

• Unnecessary warning emitted when using JUnit Jupiter 5.9 with Maven plugin #311

[snicoll]

So far it hasn't been much of an issue considering that the projects are working with somewhat the same junit version. You upgrade and the caller (for instance Spring Boot) upgrades as well. But the duplication looks strange, and the fact that we take the version of the plugin (and not the version of the project) is strange as well.

Is there any plan to fix this?

[sbrannen]

As mentioned in #541 (comment) and #706, users are currently forced to declare dependencies on all JUnit artifacts (or enough to pull in the rest transitively) if they want to use a newer version of JUnit 5 than the one hard coded into the NBT Maven plugin.

Please note that Maven Surefire has automatic support for figuring out which compatible versions of JUnit 5 artifacts are actually required at test runtime (such as the junit-jupiter-engine and junit-platform-launcher). That support came in Surefire 3.0 M4. See notes in the JUnit 5 User Guide for details.

On the other hand, Gradle now requires that users specify an explicit dependency on junit-platform-launcher. Note, however, that that was not always the case.

Similarly, I believe it's required (or at least recommended) to specify an explicit dependency on the junit-platform-launcher for proper IDE support.

For example, see:

https://github.com/spring-projects/spring-framework/blob/8ee09e57661d9ea95bcc64fc933b46cdf3b3feb9/build.gradle#L67-L69

As stated in the JUnit 5 User Guide:

Declaring a dependency on junit-platform-launcher

Even though pre-8.0 versions of Gradle don’t require declaring an explicit dependency on junit-platform-launcher, it is recommended to do so to ensure the versions of JUnit artifacts on the test runtime classpath are aligned.

Moreover, doing so is recommended and in some cases even required when importing the project into an IDE like Eclipse or IntelliJ IDEA.

[stevenschlansker]

So far it hasn't been much of an issue considering that the projects are working with somewhat the same junit version.

Maybe not for people 'in the know'. But it definitely harms Graal native adoption - for example we've been working on jdbi/jdbi#2476 since Aug 2023 to try to get Jdbi working with Graal native build pipeline, but keep running into issues like this that stop forward progress and still haven't completed the issue. (To be clear this issue is not the only one we ran into, but the 'death by multiple papercuts' effect where we keep running into rather inscrutable-to-the-end-user problems. At this point it seems like we know enough to apply a workaround, which I'll do next time I gather the energy to pick this PR up again...)

Having inconsistent classpaths can cause really weird behavior that takes a long time to track down if you don't know where to start looking. We'd love to see this fixed :)

[dnestoro]

Hey @stevenschlansker @snicoll @sbrannen, I made a pull request to fix this issue. It basically checks if there is already an artifact on the classpath with the same groupID and artifactID as the artifact we want to add (when adding JUnit artifacts).

What I am wondering is if we can break existing projects if we fix this issue.

Possible problem:

• If we update hardcoded JUnit version to 5.11.0 here - for example to support @FieldSource annotation (like I did in this PR)
• and user declares in its project earlier JUnit version (for example 5.10.0)
• we won't have 5.11.0 artifact on the classpath, and therefore we will get NoClassDefFoundError since FieldSource is not available before 5.11.0 version.

In that case we can catch and ignore NoClassDefFoundError when supporting annotations from newer JUnit versions (and that is fine I guess).

The question is: can this fix harm existing projects if we suddenly remove "duplicates" (same artifacts but with different versions)?

[stevenschlansker]

Thank you so much! At least from my perspective, the harm of having conflicting jars in the classpath far outweighs any harm caused by removing the duplicates - better to rip the bandage off and fix the root problem IMO.

I believe the current stock OpenJDK implementation (not sure if this is guaranteed by spec?) will simply drop missing annotations from reflection metadata without further harm, which might simplify the case you're concerned about.

[danthe1st]

I believe the current stock OpenJDK implementation (not sure if this is guaranteed by spec?) will simply drop missing annotations from reflection metadata without further harm, which might simplify the case you're concerned about.

According to this Stack Overflow answer by jarnbjo and section 13.5.8 of the current JLS:

Adding or deleting annotations has no effect on the correct linkage of the binary representations of programs in the Java programming language.