4.x Allow user to influence JFR `RecordingStream` used for virtual thread meters

[tjquinno]

Environment Details

• Helidon Version: 4.x
• Helidon SE or Helidon MP
• JDK version:
• OS:
• Docker version (if applicable):

---

Problem Description

The new support for built-in meters related to virtual threads relies on Java Flight Recorder events. The RecordingStream Helidon uses for this currently uses the default JFR configuration settings.

This enhancement would allow users to use Helidon config or programmatically to choose which JFR config name or file to use for the RecordingStream.

Users would be able to choose the configuration by specifying:

• the JFR config name (for predefined JFR configurations--these .jfc files are always stored in JAVA_ROOT/lib/jfc), or
• the path to a custom .jfc file.

Helidon would interpret the setting as a name first and, if it could not find a JFR configuration by that name, then use it as as path to a .jfc file.

If Helidon could find neither, or the attempt to load a custom file fails, the server start-up would fail.

[vasanth-bhat]

Related comment from JIRA 9619

1. In JFR stream or JFR, the details as to which events are captured, threshold used for each event , stack capture etc depends on settings used to create the Recording (RecordingStream internally creates Recording) . JDK ships with 2 settings "default" and "profile".

For example , Below are settings for ”jdk.VirtualThreadPinned” event in JDK bundled settings "default" and "profile" . This would means when these settings are used, a Pinned JFR event would be recorded only for cases where the carrier thread was pinned for 20ms or longer.

 <event name="jdk.VirtualThreadPinned">
      <setting name="enabled">true</setting>
      <setting name="stackTrace">true</setting>
      <setting name="threshold">20 ms</setting>
 </event>

However many cases the system also run with custom settings via custom JFC file, and can can customise settings for both built-in JDK provided as well as custom defined JFR events.

2 ”jdk.VirtualThreadPinned” JFR events are not generated for all pinning cases. For example carrier thread also get pinned in Java-21 when the vthread mounted on them execute Object.wait(). But these do not generate ”jdk.VirtualThreadPinned”. Same is true for pinning due to blocking operation in class initialiser ( for ex : static blocks), or pining due to certain blocking operation in native code. So primarily, we get pinning events only for blocking operation from sync blocks.
The metric would primarily account only for pinning events due to blocking operation performed by vthread in a sync block in java-21. It would not account the other pinning scenarios.

[tjquinno]

This enhancement request would address your copied comment # 1 IIUC. Or are you requesting a change or addition to the proposed enhancement?

As for your copied # 2 , Helidon would expose what JFR reveals. The specifics of what in the JVM triggers specific events is outside Helidon's control. I will plan to revise the Helidon doc about the virtual thread metrics to emphasize that Helidon is reporting what JFR exposes.

I'm not aware of an alternative or additional supported way for Helidon to get further information about pinned threads to fill in the gap you described in the JFR events. If you know of one please share it.

[vasanth-bhat]

yes, agree with above comments.

one minor detail, the start up options event settings are typically be controlled by two parameters "settings" and "event-settings"

-XX:StartFlightRecording=parameter=value
one is settings=path, where path can be either a built-in config name or path to a custom jFC with custom settings.

The event settings can also be controlled by by parameter "event-settings"
event-setting=value , where value is a direct event setting and not a reference to custom settings file. It uses the form: "#=" . For New event setting, prefix the event name with '+'

Example :
-XX:StartFlightRecording:settings=default, event-setting=jdk.VirtualThreadPinned#threshold=1,+jdk.VirtualThreadStart#enabled=false,+jdk.VirtualThreadEnd#enabled=false

Ref :
https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html#advanced-runtime-options-for-java

So this would take the settings for event from built-in "default" and later would override few specific settings for certain events.

if we are using the settings config passed via command-line to create the RecordingStream it would eb good to consider values provided via both "settings" as well as "event-settings" parameter.

[vasanth-bhat]

I'm not aware of an alternative or additional supported way for Helidon to get further information about pinned threads to fill in the gap you described in the JFR events. If you know of one please share it.

Yes, there is no programmatic API.

Just for information
Currently one manual approach we use to detect these, is to have vthread thread dumps using jcmd , taken during the load tests , and have script parse those and look for specific patterns known to cause pinning in Java -21. ( Fro example , vthread stacks parked on Object.wait() or. ParkOnCarrierThread() etc). This jut for information only. There is no ask to use such techniques to detect carrier thread pinning

[tjquinno]

As explained in the issue description, Helidon will look up a Helidon config value--not use any JFR command-line values--to find out which JFR configuration name or file path to use for the RecordingStream to subscribe to JFR events.

Helidon's interpretation of the value as a predefined config name or a file path is consistent with the way the JFR command-line value is interpreted, but there is a key reason to use Helidon config and not the JFR command-line option for this purpose: the Helidon virtual threads metrics feature does not require that a JFR recording be in progress.

In contrast, on the JFR command line the settings value is on the StartFlightRecording option.

Of course, Helidon config supports config files and Java system properties and environment variables so a user could specify which config Helidon should use on the command line, just not with the StartFlightRecording JFR option.

Helidon does not support event-setting because there are no public supported JFR APIs that allow merging individual settings with settings in a config file.