TBD: Setup an continuously performance checking

[khmarbaise]

Goal

• Run a performance analysing on a regular basis to see performance degrading etc. happens
• It is inteded to identify problems during the development fixing of issue which might have influences on the performance of Maven itself.

Non-Goals

It is not a goal to

• make a comparison with any other build tool.
• ??

TODO

• Produce automatic graphical informations about performance results
  • Different Maven Versions
  • Different JDK Versions
  • Memory consumptions
  • Using cache extension

Requirements for measurements

• Select appropriate project(s) as foundations (some real life projects)
• Create a number of synthetic generated project(s)
• Define reference machine setup's
  • We need to have appropriate hardware to run those performance testing
  • My opinion is that it's can't be run on GitHub actions etc. because you will get all the time a different machine which means your measurements will never be reproducible.

Idea

• Use a library to show the result of the measurements (for example echarts)

Reference

• https://github.com/khmarbaise/maven-performance-testing
• https://echarts.apache.org/en/index.html (could be integrated into a spring boot app/quarkus app or alike)

TBD more details...

[delanym]

Would this include some comparison with Gradle?

[khmarbaise]

Would this include some comparison with Gradle?

No I would not do that...

[rmannibucau]

Guess we should limit ourselves to ensuring we don't have regression so likely a few project kinds (single plain jar module - commons like, multimodule etc) and have a way to see the evolution with versions (ideally snapshot) to prevent any regression like we got recently.
Indeed it should be done with a single jdk for multiple maven version.

Having tests with multiple JDK for the same maven version can be worth it but since at some point they all converge I consider it as minor compared to identifying regressions in our codebase.

Hope it makes sense.