Replace surefire with a more native approach #5376
Description
We currently use an extension of Maven Surefire Plugin, but this often leads to some brittle bugs because of the classlaoding is not optimized to be used inside OSGi:
- Tycho 5.x: Context class loader set in Surefire Plugin leads to ClassNotFoundExceptions #5281
- JUnit 5 Test Suite not working - "No tests found" #2462
- Incomplete JUnit 5.9 support in Tycho 3.0.x #1943
- Add regression test for memory leak caused by surefire #1322
- tycho-surefire-plugin does not find junit5 suite #1097
- maven-surefire classpath order is different between 3.x and 4.x #4555
- JUnit 5 Test Suite @SelectPackages not finding tests #3709
- Rerun failed tests for dynamic tests leads to wrong xml reports generation #2432
- ...
Also there are some issues with we need to keep up with changes in that plugin as well:
Compared to that with modern JUnit, the benefits of surefire are not that big anymore and there are even native alternatives as well
Since a while we integrated even bnd-test framework that also controls the execution using native JUnit 5 inside an OSGi framework
One thing is that we support TestNG but I'm not aware of any bigger userbase for that:
While it seems good to reuse the maven "native" approach it more becomes a burden than a benefit and hinders further development. I therefore would suggest to simply drop / deprecate that part and streamline the process, either support our use-case with the bnd-testing or adopt what the junit-platform-maven-plugin does and directly integrate with the junit platform that is defacto standard for Eclipse test execution anyways. This would best fit into a new tycho-testing-plugin to also allow better naming of mojos e.g. the bnd-testing is not really related to sure-fire and the current "test" are more related to plugin-tests in PDE than regular JUnit tests (that usually execute without OSGi).
There is also: